More on slow file output

[Katie Antypas]

I think it is a pretty easy fix to just write out the max and min 
attributes initially in define mode, although it will require a number 
of changes to the FLASH IO internal interfaces.  I didn't realize redef 
would cause such a performance hit.  Right now I'm running the tests 
without writing the max and min attributes which I don't think are all 
that important in any case.

The 128 proc case ran with no problems.  I'm waiting for some larger 
jobs to go through.  I'll let you guys know the results.

Katie


Robert Latham wrote:
> On Tue, Dec 05, 2006 at 10:42:32AM -0800, Katie Antypas wrote:
>   
>> I'm am doing some io benchmarking with the FLASH code on the NERSC 
>> machine Bassi, (IBM Power 5) and I was also seeing some horrible 
>> performance results.  I wanted to investigate this more before writing 
>> the list, but it looks to me that that the slow down is caused when the 
>> FLASH IO tries to 'reenter' define mode with the ncmpi_redef call.
>>     
>
> Hi 
>
> Bill and Katie, you certainly seem to be on the right track.  Katie,
> thanks for the new information -- I wasn't really sure what the
> problem might be, but the fact that FLASH reenters define mode gives
> us a lot of suspects. 
>
> Bill, regarding ncmpiio_move, The serial netcdf guys have a comment
> that their equivalent move routine is only used by nc_enddef after
> redefinition, and is also likely to be an expensive operation.
>
>   
>> (Anshu, FLASH defines all the pnetcdf dimensions and variables ahead of 
>> time, then leaves define mode and writes the data.  However, we added a 
>> some functionality early in the summer to write out the maximum and 
>> minimum values of each unk variable and these are written out as 
>> attributes of the given variable.  We reenter define mode in the routine 
>> io_ncmpi_write_unknowns and io_ncmpi_write_unknowns_sp for plotfiles in 
>> order to write out the attributes.)
>>
>> Should the FLASH io should be doing something differently?  Should we 
>> expect a performance hit from ncmpi_redef?  When I remove the calls to 
>> reentering define mode and do not write out the attributes the 
>> performance looks pretty good and in fact (on initial tests) appears to 
>> be faster than the equivalent hdf5 tests.  I still have a few jobs 
>> waiting in the queue here but as I get the results I will let you know.
>>     
>
> We can make this more efficent in pnetcdf by increasing that buffer
> size as Bill observered.  In general, though, yes, redefinition is
> going to be a very expensive process, as most if not all of the header
> (and then all the variable data too I suspect) are shifted in the
> file.  
>
> adding anything to the netcdf header is bad for performance in
> general: serial netcdf would take a performance hit as well. 
>
> The solution might be pretty simple:  I think if you were to create
> those min and max attributes the first time you were in define mode
> and give them some sort of known garbage value, you could then go back
> and fill in those attributes with the real values. you could get all
> the functionallity you want while still achieving good performance:
> because no new space will be needed for these attributes, you'll avoid
> the data shifting problem you are observing right now.
>
> Would that be an easy change to test out?
>
> ==rob
>
>