[pnetcdf-devel] [parallel-netcdf] #29: cdf-2 max array size

[Wei-keng Liao]

Hi, Luke

There are some limits for CDF-2 file format. One of them is "No fixed-size
variable can require more than 2^32 - 4 bytes (i.e. 4GiB - 4 bytes, or
4,294,967,292 bytes) of storage for its data, unless it is the last
fixed-size variable and there are no record variables. When there are no
record variables, the last fixed-size variable can be any size supported by
the file system, e.g. terabytes.)". See this URL.
http://www.unidata.ucar.edu/software/netcdf/docs/file_structure_and_performance.html#offset_format_limitations

The size referred there is for the whole variable in bytes, so the check
must consider variable's element size, xsz.)

It was a bug in PnetCDF 1.5.0 that fails to honor the above limitation.
The bug has been fixed in 1.6.0.


Wei-keng

On Mar 24, 2017, at 4:08 PM, parallel-netcdf wrote:

> #29: cdf-2 max array size
> --------------------------------------+-------------------------------------
> Reporter:  luke.vanroekel@…          |       Owner:  robl 
>     Type:  defect/bug                |      Status:  new  
> Priority:  major                     |   Milestone:       
> Component:  parallel-netcdf           |     Version:  1.8.1
> Keywords:  CDF2 max size             |  
> --------------------------------------+-------------------------------------
> Hello,
>   I've been using pnetcdf to write very large data sets (max array 1.2E9
> elements), if I use pnetcdf/1.5.0, everything works fine with the datatype
> NC_64BIT_OFFSET.  But if I use a newer version (same data, same datatype),
> I receive the NC_EVARSIZE error and the file is not written.
> 
> I spent a bit of time looking through the code in versions 1.5.0 and 1.8.1
> and it seems the way you enforce maxsize on CDF2 files has changed.  In
> 1.5.0 you simply check if the product of the dimensions of the biggest
> array is > 2^32-4, but in 1.8.1, this check seems to be agains
> (2^32-4)/(array element size (xsz in check_vlens)).  I'm curious why this
> was changed?  The output from version 1.5.0 looks good, so it doesn't seem
> as though we are violating any CDF2 file size constraints.
> 
> Let me know if you need more information.
> 
> I'm not sure if this should be tagged as defect/bug type or clarification.
> So I'll leave as is, my apologies if the type is inappropriate.
> 
> Regards,
> Luke
> 
> -- 
> Ticket URL: <http://trac.mcs.anl.gov/projects/parallel-netcdf/ticket/29>
> parallel-netcdf <https://trac.mcs.anl.gov/projects/parallel-netcdf>
> 
> _______________________________________________
> pnetcdf-devel mailing list
> pnetcdf-devel at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/pnetcdf-devel