bad performance

[Rob Latham]

On Tue, Nov 24, 2009 at 04:45:34PM +0100, Joeckel, Patrick wrote:
> Rob Latham wrote:
> >On Mon, Nov 23, 2009 at 03:03:20PM +0100, Joeckel, Patrick wrote:
> >>Dear Rob,
> >>
> >>setting
> >>IBM_io_buffer_size to 2949120
> >>and
> >>IBM_sparse_access to true
> >>reduced the factor to approx. 2 (compared to serial netCDF),
> >>which is already more than a factor of 2 improvement.
> >
> >OK, great! did you have to set both of these to see a benefit?
> >
> Yes, I did. But in the meantime I checked without setting
> IBM_io_buffer_size. This seems to be even 15% faster !!!???
> Doeas that sound reasonable.
> Is it worth to try only  IBM_io_buffer_size without
> IBM_sparse_access ?

Hi Patrick.

I don't know the internals of IBM's MPI-IO library, but I can imagine
algorithms that they might use with IBM_sparse_access which would
improve your performance.   If you're willing to try
IBM_io_buffer_size without IBM_sparse_access, that would be great: it
will help us understand which hints are safe for the library to set.

It sounds like we need to not only look at the size of a record
variable, but come up with some threshold value.  In your case, you
have many record variables, and so have a very sparse access pattern.
I don't know if we can automatically guess when to tweak the
intermediate buffer size and when to use 'sparse access'.    Perhaps
we can introduce pnetcdf-specific hints?  

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA