switch between indep and collective modes

[Rob Ross]

Hi Wei-keng,

Right now we implicitly follow the MPI-IO specifications for default 
mode. This means that changes to the same regions of the file (i.e. the 
same variable) by different processes will result in undefined state of 
those regions.

Writing in parallel to *different* regions has the expected outcome -- 
everything is good.

Part of the use of a collective "mode" as the default was also to make 
it clear that the define mode was a collective thing.

RobL is right that we were also thinking about this consistency issue, 
however. We know that there can be value in uncoordinated caching, and 
requiring this switch enables us to perform uncoordinated caching in 
PnetCDF, knowing that if the user wants to perform independent 
operations that they will *have* to make a call to do so. And we could 
then maintain the same semantics as the MPI-IO default I/O mode (as we 
do now).

Right now we don't have this caching implemented, but I think you 
understand the value of such a thing from your research.

Make sense?

As an aside, we should document the semantics.

Rob

Wei-keng Liao wrote:
> 
> If pnetcdf does not specifically define its parallel access consistency 
> semantics, I wonder if we could define it to follow the MPI-IO 
> semantics, because pnetcdf is built on top of MPI-IO. The same 
> inconsistent buffering situation would happen on MPI-IO as well as on 
> pnetcdf if the mode switching requirement is absent.
> 
> Does adding this mode switching requirement put pnetcdf in a more relax 
> consistency semantics, so it enables pnetcdf implementation to do 
> optimizations like local data caching? Otherwise, if pnetcdf completely 
> follows the MPI-IO semantics, I don't see the need for this requirement.
> 
> Maybe pnetcdf application users can comment on this, so we can see whether
> following MPI-IO consistency semantics is proper for pnetcdf.
> 
> Wei-keng
> 
> 
> On Tue, 3 Jul 2007, Robert Latham wrote:
> 
>> On Fri, Jun 22, 2007 at 12:39:58AM -0500, Wei-keng Liao wrote:
>>> According to the API document, "the separation of the two modes is
>>> necessary to provide consistent views of file data when moving between
>>> MPI-IO collective and independent operations". In what situation the
>>> inconsistency will occur if we don't enforce this mode switching?
>>
>> Hi Wei-keng
>>
>> I think we wanted to avoid a situation where an MPI-IO implementation
>> optimizes collective communication in such a way that indepedent
>> accesses might get unexpected data.
>>
>> ROMIO will use two-phase I/O for a collective noncontiguous write.  If
>> that is followed by an independent read, the results could be
>> undefined.   Explicitly entering independent mode means we can flush
>> any buffers the MPI-IO implementation might be using for collective
>> I/O.
>>
>>> Is pnetcdf's file consistency policy the same as MPI-IO?
>>
>> I don't know what the consistency semantics for netcdf would be in a
>> parallel environment.  If two processes write different data to the
>> same indexes of a variable, it's not clear who would 'win'.  I don't
>> know if applications regularly read and write variables, or if they
>> usually just all-write or all-read.
>>
>> ==rob
>>
>> -- 
>> Rob Latham
>> Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
>> Argonne National Lab, IL USA                 B29D F333 664A 4280 315B
>>
>