metadata consistency

Wei-keng Liao wkliao at ece.northwestern.edu
Thu Jul 11 12:03:35 CDT 2013


Dear PnetCDF users,

I am working on strengthening the PnetCDF's metadata consistency and
would like to change/limit the usage of APIs that modified the
metadata (file header) of a netCDF file. These APIs are:
    ncmpi_rename_dim(),
    ncmpi_rename_var(),
    ncmpi_copy_att(),
    ncmpi_rename_att(), and
    ncmpii_put_att().

(The consistency here is referring to the consistency of file header
data stored in memory across all MPI processes.)

In netCDF, the above APIs are allowed in data mode if the space
required to store the new metadata (attributes, names, etc.) is
less than the old one. Otherwise, they must be called in the define
mode.

In PnetCDF, I would like to change that to allow these APIs only
in define mode. If your applications require the above APIs to
be called in data mode, please do let me know.

Here is my reason for the above change. In data mode, if metadata
is changed on one process's memory (or even the change is written
to the file by that process because NC_SHARE is set), there is no
way to propagate the change from this process to other processes,
until ncmpi_close() or ncmpi_sync() is called. If allowing these
APIs in define mode only, we can rely on ncmpi_endef() to
ensure/check the consistency.

Please let me know if your applications will have a problem with
such change.

(My plan is to make NC_SHARE the default mode for PnetCDF as
PnetCDF IS developed to handle parallel access to shared files.
The above suggested API changes is the first step of my plan.)


Wei-keng



More information about the parallel-netcdf mailing list