Operation not allowed in collective data mode

[Robert Latham]

On Wed, Jul 18, 2007 at 07:12:35AM -0700, Koontz, Annette S wrote:
> I'm a long-time netcdf user, but new to pnetcdf. I am converting a code
> that runs in parallel, writing small netcdf files into one that now uses
> pnetcdf and writes just one netcdf file.

Excellent.  

> I've managed to create netcdf files using pnetcdf, but when the code
> gets to the point where it writes data to the file, it is failing with
> the "Operation not allowed in collective data mode" error.
> 
> I'm having some trouble deciphering what this message means. 

pnetcdf has two sets of access routines.  Those ending in _all are
'collective' (e.g. ncmpi_put_vara_double_all), while those without the
_all are 'independent' (e.g. ncmpi_put_vara_double).  

Sometimes a code really does want just one process to write a
variable.  In such a case, you need to call ncmpi_begin_indep_data()
before the independent access and call ncmpi_end_indep_data after.

> All the processors are writing data to the same file, but at different
> locations (i.e. the "start" parameter is different for each processor).

It sounds like it is the case that instead of switching to independent
data mode you can use the _all version of your access routine.
Changing the 'start' parameter is definitely the right thing to do.

Let us know how this works, and if you run into any problems witn your
conversion to pnetcdf.  You might find docs/porting_notes.txt helpful,
or you might find some points we should add to that file. 

You can read the porting_notes.txt online as well:
http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/trunk/doc/porting_notes.txt

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B