[EXTERNAL] Re: writing large variables

[Gregory Sjaardema]

I don't think that this is correct. The NC_64BIT_OFFSET enforces the 
4GiB limit on the data, not the number of elements.

For your example, you would could only use NC_64BIT_OFFSET if 
"z*x*y*sizeof(double) < 2^32"

--Greg

On 1/16/13 9:27 AM, Wei-keng Liao wrote:
> Hi, John,
>
> The mode NC_64BIT_DATA (CDF-5 format) allows you to define an array variable
> that has more than 2^31 elements. Note this is about the number of "elements"
> not the size of an array.
>
> If your array has less elements but the size is more than 4GB, then
> NC_64BIT_OFFSET can still be used. For example, double foo[Z][Y][X] has
> Z*Y*X elements. If Z*Y*X < 2^31 and Z*Y*X*sizeof(double) > 2^31, then you
> can still use NC_64BIT_OFFSET.
>
> Is this your case?
>
> Wei-keng
>
> On Jan 16, 2013, at 9:58 AM, John Clyne wrote:
>
>> Thanks for the quick response, Rob. I've poked the Unidata folks as well to see if they have any updates on their CDF-5 support plans. One followup question: Is it possible to output large variables from pnetcdf without using CDF-5? netCDF seems to support this in a CDF-2 format, albeit with restrictions. For our application we can live with those restrictions.'
>>
>> Thanks again for your help.
>>
>> Best,
>>
>> jc
>>
>> On Jan 16, 2013, at 7:54 AM, Rob Latham wrote:
>>
>>> On Tue, Jan 15, 2013 at 05:09:05PM -0700, John Clyne wrote:
>>>> Is it possible to write a large variable (>4GB) to a file with pnetcdf and read back the variable from the resulting file with netCDF? Outputting a large variable with pnetcdf appears to require passing the NC_64BIT_DATA flag (not NC_64BIT_OFFSET) to nc_create_par() - without this flag an error is generated. The file is written successfully, but when NC_64BIT_DATA is used the file is unrecognized by netcdf. For example:
>>>>
>>>> yslogin2[43] ncdump -h vx.0000.nc0
>>>> ncdump: vx.0000.nc0: NetCDF: Unknown file format
>>>>
>>>>  From what I can gather from the web the NC_64BIT_DATA results in the generation of a CDF-5 formatted file. Is there support for CDF-5 in netCDF, or any other options for mixing pnetcdf and netCDF?
>>> Hi John:  the short answer is there is no "unidata netCDF" support for
>>> pnetcdf's CDF-5 (giant variables) file format.
>>>
>>> I've been working with Unidata  on and off over the last few years to
>>> find a way that we could use NetCDF-4's "netcdf on pnetcdf" feature to
>>> support CDF-5, but that support right now only exists as a series of
>>> patches yet to be incorporated into Unidata's tree.
>>>
>>> ==rob
>>>
>>> -- 
>>> Rob Latham
>>> Mathematics and Computer Science Division
>>> Argonne National Lab, IL USA
>> John Clyne
>> National Center for Atmospheric Research
>> 303.497.1236 (w), 303.809.1922 (c)
>> clyne at ucar.edu
>>
>>
>>
>