> 4GiB variables

[Robert Latham]

On Wed, Apr 04, 2007 at 11:33:52AM -0700, Katie Antypas wrote:
> Hi All,
> 
> We are seeing the 4GiB limitation on variable sizes on the Power 5 
> machine Bassi at NERSC.  The last fixed-size variable in a file can be 
> larger.  John Shalf's group is also seeing the same limitation, although 
> he though he recalled speaking with Rob Ross about how this limitation 
> was lifted in recent versions.  Is this the case?

Hi Katie

This appears to be a limitation of both serial NetCDF and
Parallel-NetCDF, even with the CDF-2 format: 

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html

    No fixed-size variable can require more than 2^32 - 4 bytes (i.e.
    4GiB - 4 bytes, or 4,294,967,292 bytes) of storage for its data,
    unless it is the last fixed-size variable and there are no record
    variables.  When there are no record variables, the last
    fixed-size variable can be any size supported by the file system,
    e.g. terabytes.

http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10
    Have all netCDF size limits been eliminated?

    No, there are still some limits on sizes of netCDF objects, even
    with the new 64-bit offset format. Each fixed-size variable and
    the data for one record's worth of a record variable are limited
    in size to a little less that 4 GiB, which is twice the size limit
    in versions earlier than netCDF 3.6.

    The maximum number of records remains 232-1.

Is there a straightforward way for you to work with variables less
than 4GB?  If not, we can talk with the serial netcdf guys about ways
to work around this limit.   Just let us know.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B