Pnetcdf New File Format

[Katie Antypas]

Thanks for the responses.  I think I was off by an exponent.  I knew 
there was the subarray limit, but had it at 2^32 rather than 2^31.

I have a few other things to report too.

First, I believe I found a bug in the ncmpidump tool.  For a 
multidimensional variable of size (x,y,z), ncmpidump reports the 
variable dimensions as (x,x,y) and which leads the data output to be 
incorrect as well.

Secondly, I have successfully created a variable size of > 4GB using a 
3d-multidimensional variable, without using the UNLIMITED dimension, so 
this is good news.  I tested creating a variable up to 32 GB without 
problems.  Next I tried to put additional variables into the file, but 
it appears that any variables after the first one are still cut off at 
4GB.  Is this a known issue?

I am working with a group who needs to write out about ~50 16GB 
variables for a production run.  There are a number of different work 
arounds but right now we have to focus on performance.  The current 
implementation puts each variable in a separate file, but we are finding 
the overhead of opening and closing files to be high which is why I am 
trying to test putting all or at least some of the variables in the same 
file.

Katie

Wei-keng Liao wrote:
> Hi, Katie,
>
> The development of the new pnetcdf 64bit integer format is still in its
> very early stage. Dr. Kui Gao here at Northwestern is working on it.
>
> The reason of why 1D array with more than 2^31 elements is not work is
> because the current pnetcdf's implementation calls
> MPI_Type_create_subarray() and its arguments are defined as int (32bit
> integers). Rob R., Rob L. and we are all aware of this problem. Kui is
> implementing an alternative approach now. Once this is done, we will
> update the source asap.
>
> As for the integer type, my suggestion is to use MPI_Offset, rather than
> int64_t. int64_t is for internal implementation. Using MPI_Offset in
> application programs is more portable and consistent.
>
> Wei-keng
>
>
> On Mon, 11 Aug 2008, Katie Antypas wrote:
>
>    
>> Hi Rob,
>>
>> I am testing out the new pnetcdf file format which is supposed to allow
>> variable sizes greater than 4GB (as long as they have fewer than 2^31
>> elements).  I am running on the Franklin Cray XT4 system at NERSC with
>> the PGI compilers and have not successfully created a 1d variable size
>> larger than 4GB.  (I'll try multi-dimensions soon.)
>>
>> I pass "NC_CLOBBER | NC_64BIT_DATA" to ncmpi_create now.  Also, I was
>> getting type warnings and noticed a new type int64_t in places where
>> type int and MPI_Offset were used before in a number of calls.  What is
>> your recommendation to users regarding the int64_t type?  Should they
>> replace MPI_Offset with int64_t?
>>
>> In either case though, if I do or don't make any changes to the ncmpi
>> calls (namely ncmpi_create, ncmpi_def_var, ncmpi_put_vara_double_all)
>> then I can only create a variable up to 2GB variable size which is under
>> 2^31 NC_DOUBLE elements in a 1 dimensional array.  (If we were counting
>> bytes, rather than doubles my test would be at the 2^31 elements limit.)
>> I didn't think this was the way elements were counted though?
>>
>> Katie
>>
>>
>>
>>
>>
>>
>>
>>
>>      
>
>    
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20080812/fa07d928/attachment.htm>