writing large variables

[Wei-keng Liao]

Hi, John,

The mode NC_64BIT_DATA (CDF-5 format) allows you to define an array variable
that has more than 2^31 elements. Note this is about the number of "elements"
not the size of an array.

If your array has less elements but the size is more than 4GB, then
NC_64BIT_OFFSET can still be used. For example, double foo[Z][Y][X] has
Z*Y*X elements. If Z*Y*X < 2^31 and Z*Y*X*sizeof(double) > 2^31, then you
can still use NC_64BIT_OFFSET.

Is this your case?

Wei-keng

On Jan 16, 2013, at 9:58 AM, John Clyne wrote:

> Thanks for the quick response, Rob. I've poked the Unidata folks as well to see if they have any updates on their CDF-5 support plans. One followup question: Is it possible to output large variables from pnetcdf without using CDF-5? netCDF seems to support this in a CDF-2 format, albeit with restrictions. For our application we can live with those restrictions.'
> 
> Thanks again for your help.
> 
> Best,
> 
> jc
> 
> On Jan 16, 2013, at 7:54 AM, Rob Latham wrote:
> 
>> On Tue, Jan 15, 2013 at 05:09:05PM -0700, John Clyne wrote:
>>> Is it possible to write a large variable (>4GB) to a file with pnetcdf and read back the variable from the resulting file with netCDF? Outputting a large variable with pnetcdf appears to require passing the NC_64BIT_DATA flag (not NC_64BIT_OFFSET) to nc_create_par() - without this flag an error is generated. The file is written successfully, but when NC_64BIT_DATA is used the file is unrecognized by netcdf. For example:
>>> 
>>> yslogin2[43] ncdump -h vx.0000.nc0
>>> ncdump: vx.0000.nc0: NetCDF: Unknown file format
>>> 
>>> From what I can gather from the web the NC_64BIT_DATA results in the generation of a CDF-5 formatted file. Is there support for CDF-5 in netCDF, or any other options for mixing pnetcdf and netCDF?
>> 
>> Hi John:  the short answer is there is no "unidata netCDF" support for
>> pnetcdf's CDF-5 (giant variables) file format.  
>> 
>> I've been working with Unidata  on and off over the last few years to
>> find a way that we could use NetCDF-4's "netcdf on pnetcdf" feature to
>> support CDF-5, but that support right now only exists as a series of
>> patches yet to be incorporated into Unidata's tree.
>> 
>> ==rob
>> 
>> -- 
>> Rob Latham
>> Mathematics and Computer Science Division
>> Argonne National Lab, IL USA
> 
> John Clyne
> National Center for Atmospheric Research
> 303.497.1236 (w), 303.809.1922 (c)
> clyne at ucar.edu
> 
> 
>