writing large variables

[John Clyne]

Hi Wei-Keng,

I should have been more clear. The array has more than 2^31 elements. Our test case presently has on the order of 2^33 elements, and soon we'll need to support 2^36 elements or more.

It sounds like the NC_64BIT_DATA flag is required in our case?

thanks - jc

On Jan 16, 2013, at 9:27 AM, Wei-keng Liao wrote:

> Hi, John,
> 
> The mode NC_64BIT_DATA (CDF-5 format) allows you to define an array variable
> that has more than 2^31 elements. Note this is about the number of "elements"
> not the size of an array.
> 
> If your array has less elements but the size is more than 4GB, then
> NC_64BIT_OFFSET can still be used. For example, double foo[Z][Y][X] has
> Z*Y*X elements. If Z*Y*X < 2^31 and Z*Y*X*sizeof(double) > 2^31, then you
> can still use NC_64BIT_OFFSET.
> 
> Is this your case?
> 
> Wei-keng
> 
> On Jan 16, 2013, at 9:58 AM, John Clyne wrote:
> 
>> Thanks for the quick response, Rob. I've poked the Unidata folks as well to see if they have any updates on their CDF-5 support plans. One followup question: Is it possible to output large variables from pnetcdf without using CDF-5? netCDF seems to support this in a CDF-2 format, albeit with restrictions. For our application we can live with those restrictions.'
>> 
>> Thanks again for your help.
>> 
>> Best,
>> 
>> jc
>> 
>> On Jan 16, 2013, at 7:54 AM, Rob Latham wrote:
>> 
>>> On Tue, Jan 15, 2013 at 05:09:05PM -0700, John Clyne wrote:
>>>> Is it possible to write a large variable (>4GB) to a file with pnetcdf and read back the variable from the resulting file with netCDF? Outputting a large variable with pnetcdf appears to require passing the NC_64BIT_DATA flag (not NC_64BIT_OFFSET) to nc_create_par() - without this flag an error is generated. The file is written successfully, but when NC_64BIT_DATA is used the file is unrecognized by netcdf. For example:
>>>> 
>>>> yslogin2[43] ncdump -h vx.0000.nc0
>>>> ncdump: vx.0000.nc0: NetCDF: Unknown file format
>>>> 
>>>> From what I can gather from the web the NC_64BIT_DATA results in the generation of a CDF-5 formatted file. Is there support for CDF-5 in netCDF, or any other options for mixing pnetcdf and netCDF?
>>> 
>>> Hi John:  the short answer is there is no "unidata netCDF" support for
>>> pnetcdf's CDF-5 (giant variables) file format.  
>>> 
>>> I've been working with Unidata  on and off over the last few years to
>>> find a way that we could use NetCDF-4's "netcdf on pnetcdf" feature to
>>> support CDF-5, but that support right now only exists as a series of
>>> patches yet to be incorporated into Unidata's tree.
>>> 
>>> ==rob
>>> 
>>> -- 
>>> Rob Latham
>>> Mathematics and Computer Science Division
>>> Argonne National Lab, IL USA
>> 
>> John Clyne
>> National Center for Atmospheric Research
>> 303.497.1236 (w), 303.809.1922 (c)
>> clyne at ucar.edu
>> 
>> 
>> 
> 

John Clyne
National Center for Atmospheric Research
303.497.1236 (w), 303.809.1922 (c)
clyne at ucar.edu