Parallel Access to NetCDF Data

Parallel access to netCDF data files can speed read/write times from paralleled code - that is, code that is written with the MPI library to run on (many) multiple cores.

How Much Does it Help?

The amount of improvement you can get for parallel access depends heavily on your hardware.

Running tests on a high-powered, multi-core linux box, I can see improvements of 4X or even greater. On HPC systems with a parallel file system, you should be able to do better.

But parallel IO performance (IMHO), does not scale as well as processor performance. On systems with a large number of cores, you will max out the hardward channels to your storage before you are using all the cores for IO.

Using a subset of the cores for IO is a good solution, but annoying to program. This functionality is provided by the PIO library (see below).

Building NetCDF with Parallel Access

In order to use parallel access with netCDF, you must build netCDF correctly. The following must be true:

• All libraries (netcdf-c, hdf5, zlib, szlib, if used) must be built for parallel with MPICH or another MPI-based toolchain.
• HDF5 must be built with --enable-parallel option.
• You should use the --enable-parallel-tests option to netCDF's configure, to turn on tests for the parallel access features.

What Parallel Features are Provided?

When built with parallel enabled, netCDF offers the same parallel access as HDF5, for netCDF-4/HDF5 files only. 

Parallel access is not available for classic, 64-bit offset, or CDF5 files, unless specially enabled by also using parallel-netcdf (see below).

When parallel access is available, the user can control whether the access is collective or independent, on a per-variable basis. Normal netCDF put/get calls are used, but the will operate in parallel.

With this use of parallel IO, each processor keeps track of it's own space within the global array space, and calls the get/put functions using correctly calculated coordinates. 

What Is parallel-netcdf?

The parallel-netcdf library was developed by some folks at Northwestern University and Argonne National Labs. It provides parallel access for netCDF classic, 64-bit offset, and CDF5 files.

There are several different ways that the parallel-netcdf library can be used.

Using parallel-netcdf without netcdf-c Library

The parallel-netcdf library works as a standalone replacement for the netCDF (classic) library. It has a slightly different API that mirrors the netCDF API very closely.

For example, here are the prototypes for the create and open calls:

int ncmpi_create(MPI_Comm comm, const char *path, int cmode, MPI_Info info, int *ncidp);

int ncmpi_open(MPI_Comm comm, const char *path, int omode, MPI_Info info, int *ncidp);


Using parallel-netcdf this way makes for an easy install, but you must change your code to call the parallel-netcdf library instead of netCDF. In most cases, that just means changing nc_* into ncmpi_*.

Also, using the parallel-netcdf library this way, you do not have access to netcdf-4/HDF5 files. You can neither read nor write netcdf-4/HDF5 files with the parallel-netcdf library.

Using parallel-netcdf with the netcdf-c Library

Another way to use the parallel-netcdf library is within netcdf itself.

This is a slightly more complex install. When building netCDF, you must use the --enable-pnetcdf option, and ensure that the parallel-netcdf header and library can be found in your CPPFLAG/LDFLAG paths.

When netCDF is built with --enable-pnetcdf (and a parallel HDF5), then all netCDF files can be read/written with parallel access, through the normal netcdf API.

Using PIO - The Parallel IO Library

There is yet another option for parallel IO with netCDF, the PIO library. This is a library developed by Jim at NCAR, which I have been spending the last year or more adding features to.

This library wraps netCDF-4 and parallel-netcdf, and provides additional features, such as the ability to designate a subset of processors as IO processors.

I am also adding features like (limited) netcdf-4 features, asynchronous writes, and support for multiple computation components, all sharing one IO component.

The use of the PIO library is slightly different from the netCDF library, although the API closely mirrors the netCDF API. See the PIO documentation for more details about building and using PIO.