parallel-netcdf buffered I/O interface

[Jim Edwards]

Hi Wei-keng,

The  NC_EINSUFFBUF error code is returned from the bput call?   If you get
that error will you need to make that same bput call again after flushing?
But the other tasks involved in the same bput call who didn't have full
buffers would do what?

I could use a query function and to avoid the independent write calls would
do an mpi_allreduce on the max memory used before calling the mpi_waitall.
If the max is approaching the buffer size I would flush all io tasks. This
is basically what I have implemented in pio with iput - I have a user
determined limit on the size of the buffer and grow the buffer with each
iput call, when the buffer meets (or exceeds) the limit on any task I call
waitall on all tasks.


On Tue, Aug 14, 2012 at 10:07 PM, Wei-keng Liao <wkliao at ece.northwestern.edu
> wrote:

> Hi, Jim,
>
> The usage of bput APIs is very similar to iput, except the followings.
> 1. users must tell pnetcdf the size of buffer to be used by pnetcdf
> internally (attach and detach calls).
> 2. once a bput API returns, user's buffer can be reused or freed (because
> the write
>   data has been copied to the internal buffer.)
>
> The internal buffer is per file (as the attach API requires an ncid
> argument.) It can be used to aggregate
> requests to multiple variables defined in the file.
>
> I did not implement a query API to check the current usage of the buffer.
> If this query is useful, we
> can implement it. Let me know. But please note this query will be an
> independent call, so you
> will have to call independent wait (nfmpi_wait). Independent wait uses MPI
> independent I/O, causing
> poor performance, not recommended. Otherwise, you need an MPI reduce to
> ensure all processes know
> when to call the collective wait_all.
>
> You are right about flushing. The buffer will not be flushed automatically
> and all file I/O happens in wait_all.
> If the attached buffer ran out of space, NC_EINSUFFBUF error code
> (non-fatal) will return. It can be
> used to determine to call wait API, as described above. However, an
> automatic flushing would require an MPI
> independent I/O, again meaning a poor performance. So, I recommend to make
> sure the buffer size is
> sufficient large. In addition, if you let pnetcdf do type conversion
> between two types of different size
> (e.g. short to int), you must calculate the size of attach buffer using
> the larger type.
>
> If automatic flushing is highly desired, we can add it later.
>
> Once the call to wait/wait_all returns, the internal buffer is marked
> empty.
>
> Let me know if the above answers your questions.
>
> Wei-keng
>
> On Aug 14, 2012, at 2:04 PM, Jim Edwards wrote:
>
> > No, the flush must happen in the nfmpi_wait_all.
> > But does that call mark the buffer as empty?  I'll wait and bug
> > Wei-keng.
> >
> > On Tue, Aug 14, 2012 at 12:56 PM, Rob Latham <robl at mcs.anl.gov> wrote:
> > On Tue, Aug 14, 2012 at 12:52:46PM -0600, Jim Edwards wrote:
> >> Hi Rob,
> >>
> >> I assume that the same buffer can be used for multiple variables (as
> long
> >> as they are associated with the same file).    Is there a query
> function so
> >> that you know when you've used the entire buffer and it's time to flush?
> >
> > It does not appear to be so.  The only non-data-movement routines in
> > the API are these:
> >
> > int ncmpi_buffer_attach(int ncid, MPI_Offset bufsize);
> > int ncmpi_buffer_detach(int ncid);
> >
> > The end-user doesn't flush, I don't think.  I had the impression that
> once the
> > buffer filled up, the library did the flush, then started filling up the
> buffer
> > again.  This one I'll need Wei-keng to confirm.
> >
> > ==rob
> >
> >> Jim
> >>
> >> On Tue, Aug 14, 2012 at 11:41 AM, Rob Latham <robl at mcs.anl.gov> wrote:
> >>
> >>> On Tue, Aug 14, 2012 at 10:50:15AM -0600, Jim Edwards wrote:
> >>>> No, I'm using iput and blocking get.   I'm doing my own buffereing
> layer
> >>> in
> >>>> pio.   I might consider using the bput functions - can you point me to
> >>> some
> >>>> documentation/examples?
> >>>
> >>> Sure.  It's too bad Wei-keng is on vacation this month, as he's the
> >>> one who designed and implemented this new feature for pnetcdf 1.3.0.
> >>> Wei-keng: i'm not expecting you to reply while on vacation.  I'm just
> >>> CCing you so you know I'm talking about your work :>
> >>>
> >>> I think this might be the entire contents of our documentation:
> >>>
> >>> "A new set of buffered put APIs (eg. ncmpi_bput_vara_float) is added.
> >>> They make a copy of the user's buffer internally, so the user's buffer
> >>> can be reused when the call returns. Their usage are similar to the
> >>> iput APIs. "
> >>>
> >>> Hey, check that out: Wei-keng wrote up a fortran example:
> >>>
> >>>
> >>>
> http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/trunk/examples/tutorial/pnetcdf-write-bufferedf.F
> >>>
> >>> There's also the C version:
> >>>
> >>>
> >>>
> http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/trunk/examples/tutorial/pnetcdf-write-buffered.c
> >>>
> >>>
> >>> ==rob
> >>>
> >>>> On Tue, Aug 14, 2012 at 10:16 AM, Rob Latham <robl at mcs.anl.gov>
> wrote:
> >>>>
> >>>>> Hi Jim
> >>>>>
> >>>>> You've been using the new 'bput/bget' routines, right?  Can you tell
> >>>>> me a bit about what you are using them for, and what -- if any --
> >>>>> benefit they've provided?
> >>>>>
> >>>>> (Rationale: our program management likes to see papers and
> >>>>> presentations, but the most valued contribution is 'science impact').
> >>>>>
> >>>>> Thanks
> >>>>> ==rob
> >>>>>
> >>>>> --
> >>>>> Rob Latham
> >>>>> Mathematics and Computer Science Division
> >>>>> Argonne National Lab, IL USA
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Rob Latham
> >>> Mathematics and Computer Science Division
> >>> Argonne National Lab, IL USA
> >>>
> >>
> >>
> >>
> >
> > --
> > Rob Latham
> > Mathematics and Computer Science Division
> > Argonne National Lab, IL USA
> >
> >
> >
> > --
> > Jim Edwards
> >
> > CESM Software Engineering Group
> > National Center for Atmospheric Research
> > Boulder, CO
> > 303-497-1842
> >
>
>
>


-- 

Jim Edwards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20120815/65f01ab6/attachment.html>