More on slow file output

[Katie Antypas]

I have some more results to report with parallel-netcdf on the IBM Power 
5 at NERSC.  I have removed all calls to reentering define mode and this 
allowed me improve performance up to a point.  I'm now seeing errors 
leaving just the regular define mode.  The crashes don't seem to happen 
at the same concurrency each time.  Last week, I was able to run a 128 
proc case through and the 256 proc case crashed.  Today, a 64 proc case 
is crashing.  Specifically, this is the routine and error I'm catching 
before aborting.


 status = ncmpi_enddef(ncid);
 if (status < 0){
    Driver_abortFlashC("Error io_ncmpi_write_header: enddef\n");
 }

Any ideas or do you have any more tests for me to do?

Katie


Katie Antypas wrote:
> I think it is a pretty easy fix to just write out the max and min 
> attributes initially in define mode, although it will require a number 
> of changes to the FLASH IO internal interfaces.  I didn't realize 
> redef would cause such a performance hit.  Right now I'm running the 
> tests without writing the max and min attributes which I don't think 
> are all that important in any case.
>
> The 128 proc case ran with no problems.  I'm waiting for some larger 
> jobs to go through.  I'll let you guys know the results.
>
> Katie
>
>
> Robert Latham wrote:
>> On Tue, Dec 05, 2006 at 10:42:32AM -0800, Katie Antypas wrote:
>>  
>>> I'm am doing some io benchmarking with the FLASH code on the NERSC 
>>> machine Bassi, (IBM Power 5) and I was also seeing some horrible 
>>> performance results.  I wanted to investigate this more before 
>>> writing the list, but it looks to me that that the slow down is 
>>> caused when the FLASH IO tries to 'reenter' define mode with the 
>>> ncmpi_redef call.
>>>     
>>
>> Hi
>> Bill and Katie, you certainly seem to be on the right track.  Katie,
>> thanks for the new information -- I wasn't really sure what the
>> problem might be, but the fact that FLASH reenters define mode gives
>> us a lot of suspects.
>> Bill, regarding ncmpiio_move, The serial netcdf guys have a comment
>> that their equivalent move routine is only used by nc_enddef after
>> redefinition, and is also likely to be an expensive operation.
>>
>>  
>>> (Anshu, FLASH defines all the pnetcdf dimensions and variables ahead 
>>> of time, then leaves define mode and writes the data.  However, we 
>>> added a some functionality early in the summer to write out the 
>>> maximum and minimum values of each unk variable and these are 
>>> written out as attributes of the given variable.  We reenter define 
>>> mode in the routine io_ncmpi_write_unknowns and 
>>> io_ncmpi_write_unknowns_sp for plotfiles in order to write out the 
>>> attributes.)
>>>
>>> Should the FLASH io should be doing something differently?  Should 
>>> we expect a performance hit from ncmpi_redef?  When I remove the 
>>> calls to reentering define mode and do not write out the attributes 
>>> the performance looks pretty good and in fact (on initial tests) 
>>> appears to be faster than the equivalent hdf5 tests.  I still have a 
>>> few jobs waiting in the queue here but as I get the results I will 
>>> let you know.
>>>     
>>
>> We can make this more efficent in pnetcdf by increasing that buffer
>> size as Bill observered.  In general, though, yes, redefinition is
>> going to be a very expensive process, as most if not all of the header
>> (and then all the variable data too I suspect) are shifted in the
>> file. 
>> adding anything to the netcdf header is bad for performance in
>> general: serial netcdf would take a performance hit as well.
>> The solution might be pretty simple:  I think if you were to create
>> those min and max attributes the first time you were in define mode
>> and give them some sort of known garbage value, you could then go back
>> and fill in those attributes with the real values. you could get all
>> the functionallity you want while still achieving good performance:
>> because no new space will be needed for these attributes, you'll avoid
>> the data shifting problem you are observing right now.
>>
>> Would that be an easy change to test out?
>>
>> ==rob
>>
>>