> 4GiB variables

[Katie Antypas]

After speaking with the folks here at LBL the consensus seems to be that 
if pnetcdf is to be used with petascale level applications then the 4 
GiB variable size limit is a pretty serious restriction and one that we 
would like to push to be removed even if this meant the files would be 
incompatible with 32-bit systems.  Although you can chop up datasets 
manually, this eliminates the elegance of pnetcdf, and already a number 
of our existing benchmarks will require chunking variables in order to 
use pnetcdf.

One researcher here has already hacked the Unidata NetCDF-4 code to 
remove this limitation without negative consequences (so far), but we 
haven't figured out how to do it with pNetCDF yet.

Katie


Robert Latham wrote:
> On Wed, Apr 04, 2007 at 11:33:52AM -0700, Katie Antypas wrote:
>   
>> Hi All,
>>
>> We are seeing the 4GiB limitation on variable sizes on the Power 5 
>> machine Bassi at NERSC.  The last fixed-size variable in a file can be 
>> larger.  John Shalf's group is also seeing the same limitation, although 
>> he though he recalled speaking with Rob Ross about how this limitation 
>> was lifted in recent versions.  Is this the case?
>>     
>
> Hi Katie
>
> This appears to be a limitation of both serial NetCDF and
> Parallel-NetCDF, even with the CDF-2 format: 
>
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html
>
>     No fixed-size variable can require more than 2^32 - 4 bytes (i.e.
>     4GiB - 4 bytes, or 4,294,967,292 bytes) of storage for its data,
>     unless it is the last fixed-size variable and there are no record
>     variables.  When there are no record variables, the last
>     fixed-size variable can be any size supported by the file system,
>     e.g. terabytes.
>
> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10
>     Have all netCDF size limits been eliminated?
>
>     No, there are still some limits on sizes of netCDF objects, even
>     with the new 64-bit offset format. Each fixed-size variable and
>     the data for one record's worth of a record variable are limited
>     in size to a little less that 4 GiB, which is twice the size limit
>     in versions earlier than netCDF 3.6.
>
>     The maximum number of records remains 232-1.
>
> Is there a straightforward way for you to work with variables less
> than 4GB?  If not, we can talk with the serial netcdf guys about ways
> to work around this limit.   Just let us know.
>
> ==rob
>
>