parallel netCDF question

[Russ Rew]

Hi Dan,

> I have a parallel code/machine which uses the standard netCDF library to 
> write data to a single file system NFS-mounted on the compute codes.  As 
> I got this code, it uses a fairly simple MPI send/recv mechanism 
> (similar to token passing) to allow output files to be written by 
> multiple processors in a serialized fashion (each opens, writes, then 
> closes the file).  This is slow, and bad!
> 
> As I don't know the netCDF library well, my first thought is simply to 
> have all the processors write at once, since they are going to write to 
> separate parts of the file.  However, depending on how the library does 
> buffering, this may not work.  (I know it wouldn't work with the cfitsio 
> package, for example.)  Does anyone know if this will work, will not 
> work, or will have uncertain behavior?

I know this will not work.  The way the netCDF library buffers its
output to lessen the number of disk writes, multiple writers will in
general write over each other's disk blocks, even if they each write
separate areas of the output file.

> Assuming that this will not always work, is parallel netCDF the next 
> choice, even when the file system is question is not a parallel file system?

I think you need a parallel file system to do what you have in mind.

> Any other thoughts or suggestions?

Some users have each process write a separate file, and subsequently
merge the many small files into one large netCDF file in a
post-processing step, but this can also be costly.

--Russ