More on slow file output

[Robert Latham]

On Tue, Dec 05, 2006 at 10:42:32AM -0800, Katie Antypas wrote:
> I'm am doing some io benchmarking with the FLASH code on the NERSC 
> machine Bassi, (IBM Power 5) and I was also seeing some horrible 
> performance results.  I wanted to investigate this more before writing 
> the list, but it looks to me that that the slow down is caused when the 
> FLASH IO tries to 'reenter' define mode with the ncmpi_redef call.

Hi 

Bill and Katie, you certainly seem to be on the right track.  Katie,
thanks for the new information -- I wasn't really sure what the
problem might be, but the fact that FLASH reenters define mode gives
us a lot of suspects. 

Bill, regarding ncmpiio_move, The serial netcdf guys have a comment
that their equivalent move routine is only used by nc_enddef after
redefinition, and is also likely to be an expensive operation.

> (Anshu, FLASH defines all the pnetcdf dimensions and variables ahead of 
> time, then leaves define mode and writes the data.  However, we added a 
> some functionality early in the summer to write out the maximum and 
> minimum values of each unk variable and these are written out as 
> attributes of the given variable.  We reenter define mode in the routine 
> io_ncmpi_write_unknowns and io_ncmpi_write_unknowns_sp for plotfiles in 
> order to write out the attributes.)
> 
> Should the FLASH io should be doing something differently?  Should we 
> expect a performance hit from ncmpi_redef?  When I remove the calls to 
> reentering define mode and do not write out the attributes the 
> performance looks pretty good and in fact (on initial tests) appears to 
> be faster than the equivalent hdf5 tests.  I still have a few jobs 
> waiting in the queue here but as I get the results I will let you know.

We can make this more efficent in pnetcdf by increasing that buffer
size as Bill observered.  In general, though, yes, redefinition is
going to be a very expensive process, as most if not all of the header
(and then all the variable data too I suspect) are shifted in the
file.  

adding anything to the netcdf header is bad for performance in
general: serial netcdf would take a performance hit as well. 

The solution might be pretty simple:  I think if you were to create
those min and max attributes the first time you were in define mode
and give them some sort of known garbage value, you could then go back
and fill in those attributes with the real values. you could get all
the functionallity you want while still achieving good performance:
because no new space will be needed for these attributes, you'll avoid
the data shifting problem you are observing right now.

Would that be an easy change to test out?

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B