parallel netCDF question

[Daniel S. Katz]

Thanks Russ (comments below)

Russ Rew wrote (on 12/22/05 1:08 PM):

>Hi Dan,
>
>  
>
>>I have a parallel code/machine which uses the standard netCDF library to 
>>write data to a single file system NFS-mounted on the compute codes.  As 
>>I got this code, it uses a fairly simple MPI send/recv mechanism 
>>(similar to token passing) to allow output files to be written by 
>>multiple processors in a serialized fashion (each opens, writes, then 
>>closes the file).  This is slow, and bad!
>>
>>As I don't know the netCDF library well, my first thought is simply to 
>>have all the processors write at once, since they are going to write to 
>>separate parts of the file.  However, depending on how the library does 
>>buffering, this may not work.  (I know it wouldn't work with the cfitsio 
>>package, for example.)  Does anyone know if this will work, will not 
>>work, or will have uncertain behavior?
>>    
>>
>
>I know this will not work.  The way the netCDF library buffers its
>output to lessen the number of disk writes, multiple writers will in
>general write over each other's disk blocks, even if they each write
>separate areas of the output file.
>
>  
>
>>Assuming that this will not always work, is parallel netCDF the next 
>>choice, even when the file system is question is not a parallel file system?
>>    
>>
>
>I think you need a parallel file system to do what you have in mind.
>  
>
This is not an option, unfortunately.  The 4 options are:
1.  Keep doing what I am doing, even though it is slow
2.  Accumulate data to node 0 in slabs, then have node 0 write out the 
output file
3.  Figure out a way to do this in parallel using the standard netcdf 
library that's faster than the current state.
4.  Use the parallel netcdf library, assuming that I can use it without 
a parallel file system.

Any other suggestions are welcome.

Dan

>  
>
>>Any other thoughts or suggestions?
>>    
>>
>
>Some users have each process write a separate file, and subsequently
>merge the many small files into one large netCDF file in a
>post-processing step, but this can also be costly.
>
>--Russ
>
>  
>

-- 
          Daniel S. Katz                Daniel.S.Katz at jpl.nasa.gov
    Jet Propulsion Laboratory               or d.katz at ieee.org
California Institute of Technology          or dsk at computer.org
        Mail Stop 126-104                 (818) 354-7359 (voice)
       4800 Oak Grove Drive                (818) 393-6141 (fax)
     Pasadena, CA  91109-8099       http://pat.jpl.nasa.gov/public/dsk/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20051222/bf1a9296/attachment.htm>