Pnetcdf New File Format

[Wei-keng Liao]

Hi, Katie,

Thank you for reporting the ncmpidump and the second array size problems. 
The additional arrays should not have the 2GB limintation. We are working 
on them right now. We will let you know once the fix is uploaded.

regards,

Wei-keng

On Tue, 12 Aug 2008, Katie Antypas wrote:

> Thanks for the responses.  I think I was off by an exponent.  I knew 
> there was the subarray limit, but had it at 2^32 rather than 2^31.
> 
> I have a few other things to report too.
> 
> First, I believe I found a bug in the ncmpidump tool.  For a 
> multidimensional variable of size (x,y,z), ncmpidump reports the 
> variable dimensions as (x,x,y) and which leads the data output to be 
> incorrect as well.
> 
> Secondly, I have successfully created a variable size of > 4GB using a 
> 3d-multidimensional variable, without using the UNLIMITED dimension, so 
> this is good news.  I tested creating a variable up to 32 GB without 
> problems. Next I tried to put additional variables into the file, but it 
> appears that any variables after the first one are still cut off at 4GB.  
> Is this a known issue?
> 
> I am working with a group who needs to write out about ~50 16GB 
> variables for a production run.  There are a number of different work 
> arounds but right now we have to focus on performance.  The current 
> implementation puts each variable in a separate file, but we are finding 
> the overhead of opening and closing files to be high which is why I am 
> trying to test putting all or at least some of the variables in the same 
> file.
> 
> Katie
> 
> Wei-keng Liao wrote:
> > Hi, Katie,
> >
> > The development of the new pnetcdf 64bit integer format is still in its
> > very early stage. Dr. Kui Gao here at Northwestern is working on it.
> >
> > The reason of why 1D array with more than 2^31 elements is not work is
> > because the current pnetcdf's implementation calls
> > MPI_Type_create_subarray() and its arguments are defined as int (32bit
> > integers). Rob R., Rob L. and we are all aware of this problem. Kui is
> > implementing an alternative approach now. Once this is done, we will
> > update the source asap.
> >
> > As for the integer type, my suggestion is to use MPI_Offset, rather than
> > int64_t. int64_t is for internal implementation. Using MPI_Offset in
> > application programs is more portable and consistent.
> >
> > Wei-keng
> >
> >
> > On Mon, 11 Aug 2008, Katie Antypas wrote:
> >
> >    
> > > Hi Rob,
> > >
> > > I am testing out the new pnetcdf file format which is supposed to allow
> > > variable sizes greater than 4GB (as long as they have fewer than 2^31
> > > elements).  I am running on the Franklin Cray XT4 system at NERSC with
> > > the PGI compilers and have not successfully created a 1d variable size
> > > larger than 4GB.  (I'll try multi-dimensions soon.)
> > >
> > > I pass "NC_CLOBBER | NC_64BIT_DATA" to ncmpi_create now.  Also, I was
> > > getting type warnings and noticed a new type int64_t in places where
> > > type int and MPI_Offset were used before in a number of calls.  What is
> > > your recommendation to users regarding the int64_t type?  Should they
> > > replace MPI_Offset with int64_t?
> > >
> > > In either case though, if I do or don't make any changes to the ncmpi
> > > calls (namely ncmpi_create, ncmpi_def_var, ncmpi_put_vara_double_all)
> > > then I can only create a variable up to 2GB variable size which is under
> > > 2^31 NC_DOUBLE elements in a 1 dimensional array.  (If we were counting
> > > bytes, rather than doubles my test would be at the 2^31 elements limit.)
> > > I didn't think this was the way elements were counted though?
> > >
> > > Katie
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >      
> >
> >    
>