pnetCDF performance issues

[William Gropp]

Thanks, Rob.

This does raise several questions:

1) What should the defaults be so that users get good performance "out  
of the box"?
2) Can/should pnetCDF diagnose poor choices and inform the user?
3) Can/should MPI-IO "fix" this by exploiting the MPI-IO semantics to  
permit converting writes to be aligned (e.g., by caching)?

Of these, (1) is the most important for pnetCDF, particularly as users  
compare approaches.

Bill

On Mar 6, 2011, at 7:24 PM, Rob Latham wrote:

> On Sun, Mar 06, 2011 at 05:34:31PM -0700, Russ Rew wrote:
>> Hi Bill,
>>
>>> We've been running various benchmarks, including IOR (I know - there
>>> are lots of problems with this benchmark) and found that pnetCDF  
>>> tends
>>> to perform poorly on several of the systems we're using.  We tracked
>>> it down to the header  on the netCDF file - if this header is  
>>> written
>>> and isn't a good size (like 4MB), performance can become very poor.
>>> This may be an artifact of the MPI-IO implementation (which includes
>>> ones that use ROMIO), but it impacts uses of pnetCDF.  Have you seen
>>> this problem?  What is the best way to fix this (e.g., can/should
>>> pnetCDF expect unaligned writes to be poor in MPI-IO, and work to
>>> force aligned writes?).  Note that HDF-5 appears to work around this
>>> problem; in any event, in our tests, we didn't see a serious
>>> performance impact.
>>
>> A relatively straightforward fix for new files would be to force the
>> header to always be a good size, such as a multiple of 4MB.  When
>> writing a new file, all other variable offsets are computed  
>> relative to
>> where the first variable begins.
>
> in parallel-netcdf land we don't have an equivalent of nc__create
> (note the double underscore).  We have to align things through a
> different mechanism:
>
> http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/ 
> VariableAlignment
>
> ==rob
>
> -- 
> Rob Latham
> Mathematics and Computer Science Division
> Argonne National Lab, IL USA

William Gropp
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign