parallel NetCDF: One or more variable sizes violate format constraints

Tue Nov 10 00:07:06 CST 2015

Hi Wei-keng,

Thanks for your help.

Felicity

On 10/11/2015 4:57 pm, "Wei-keng Liao" <wkliao at eecs.northwestern.edu>
wrote:

>Hi, Felicity
>
>I believe you found a bug in PnetCDF.
>I will fix this in the next release of PnetCDF.
>
>The statement "the offset to the beginning of this variable must be less
>than about 2 GiB"
>means the sum of file header and all variables other than the last one
>cannot be
>more than 2 GiB.
>
>In your case, the sum of variable x and y are 2 * 66661 * 4 = 533288 bytes
>and the file header should be small, which makes the starting file offset
>of
>variable "hf_topog" less than 2 GiB. Hence, your file is legal and valid.
>
>Thanks for your example.
>Regards.
>
>Wei-keng
>
>On Nov 9, 2015, at 11:18 PM, Felicity Graham wrote:
>
>> Hi Wei-keng,
>>
>> Here is the output:
>>
>> $ ncdump -h hf_topog.nc
>> netcdf hf_topog {
>> dimensions:
>> x = 66661 ;
>> y = 66661 ;
>> variables:
>> float x(x) ;
>> x:long_name = "easting" ;
>> x:units = "m" ;
>> float y(y) ;
>> y:long_name = "northing" ;
>> y:units = "m" ;
>> float hf_topog(y, x) ;
>> hf_topog:long_name = "high frequency topography" ;
>> }
>>
>>
>> I’m not sure what is meant by that statement “the offset to the
>>beginning
>> of this variable must be less than about 2 GiB”?
>>
>> Cheers,
>> Felicity
>>
>>
>> On 10/11/2015 4:15 pm, "Wei-keng Liao" <wkliao at eecs.northwestern.edu>
>> wrote:
>>
>>> Hi, Felicity
>>>
>>> According to "Netcdf classic format limitations", there is one
>>> possibility your file is still valid.
>>> "If you don't use the unlimited dimension, only one variable can
>>>exceed 2
>>> GiB in size, but it can be as large as the underlying file system
>>> permits. It must be the last variable in the dataset, and the offset to
>>> the beginning of this variable must be less than about 2 GiB."
>>>
>>> Maybe your file is exactly the same described above.
>>>
>>> Can you run command "ncdump -h file.nc" and send me the output?
>>>
>>>
>>> Wei-keng
>>>
>>> On Nov 9, 2015, at 10:58 PM, Felicity Graham wrote:
>>>
>>>> Hi Wei-keng and Jim,
>>>>
>>>> Thanks very much for your replies.
>>>>
>>>> I think the file may be corrupted, as I can¹t read even a small
>>>>segment
>>>> in
>>>> serial.
>>>>
>>>> I was wondering about that note in the ³Netcdf classic format
>>>> limitations²
>>>> documentation. But, ncdump -k on the file returns Œclassic¹, so it
>>>> obviously hadn¹t been enabled at creation.
>>>>
>>>> Thanks again,
>>>> Felicity
>>>>
>>>>
>>>> On 10/11/2015 3:51 pm, "Wei-keng Liao" <wkliao at eecs.northwestern.edu>
>>>> wrote:
>>>>
>>>>> Hi, Felicity and Hi, Jim
>>>>>
>>>>> From netCDF 3.6.3 C Interface Guide, NC_64BIT_OFFSET is supported.
>>>>>See
>>>>> URL
>>>>>
>>>>>
>>>>>http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netcdf
>>>>>-c
>>>>> /n
>>>>> c_005fcreate.html#nc_005fcreate
>>>>> This flag, when used at file creation, allows creating a single file
>>>>>of
>>>>> size larger than 4 GB,
>>>>> but the number of elements in each array variable is still limited
>>>>>by 2
>>>>> billions.
>>>>>
>>>>> You can verify whether or not your file is created with this feature
>>>>> enabled by running command
>>>>> 'ncdump -k file.nc' and the output should be '64-bit offset'. If you
>>>>> got
>>>>> 'classic', then
>>>>> this feature is not enabled and your file may be corrupt, as Jim
>>>>> guessed.
>>>>>
>>>>> In PnetCDF, command 'ncmpidump -k' can also be used in the similar
>>>>>way.
>>>>>
>>>>>
>>>>> Wei-keng
>>>>>
>>>>> On Nov 9, 2015, at 9:24 PM, Jim Edwards wrote:
>>>>>
>>>>>> The netcdf-3 classic file is incorrect and probably corrupt -  16GB
>>>>>> exceeds the variable size constraints of that format.  You can use
>>>>>> CDF5
>>>>>> as output by pnetcdf to write the file and then read it again with
>>>>>> pnetcdf but netcdf doesn't support the CDF5 standard until version
>>>>>> 4.4.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 9, 2015 at 7:47 PM, Felicity Graham
>>>>>> <f.s.graham at utas.edu.au> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I¹d like to use parallel netcdf in C to read the only variable from
>>>>>>a
>>>>>> 16GB netcdf file and perform some operations.
>>>>>>
>>>>>> The file format is classic, and was created with netcdf 3.6.3. I¹m
>>>>>> using openmpi 1.6.3 and pnetcdf 1.4.1.
>>>>>>
>>>>>> I¹ve tested on a separate netcdf-4 formatted file, and the reading
>>>>>> works fine, but on the classic file, I get the error  NetCDF: One or
>>>>>> more variable sizes violate format constraints when I try to open
>>>>>>the
>>>>>> file (ncmpi_open).
>>>>>>
>>>>>> A stripped down version of the C code is below.
>>>>>>
>>>>>> Your help is much appreciated.
>>>>>>
>>>>>> Cheers,
>>>>>> Felicity
>>>>>>
>>>>>>
>>>>>> #include <stdio.h>
>>>>>> #include <mpi.h>
>>>>>> #include <pnetcdf.h>
>>>>>>
>>>>>> #define FILE_NAME ³<path_to_file>/hf_topog.nc"
>>>>>> #define VAR_NAME "hf_topog"
>>>>>>
>>>>>> #define CHECK_ERR {\
>>>>>>   if (err!=NC_NOERR) {\
>>>>>>       printf("Error at line=%d: %s\n", __LINE__,
>>>>>> ncmpi_strerror(err));\
>>>>>>       goto fn_exit;\
>>>>>>   }}
>>>>>>
>>>>>> int main(int argc, char **argv)
>>>>>> {
>>>>>> int err, varid, ncid;
>>>>>> float * topog_in;
>>>>>>
>>>>>> MPI_Init(&argc,&argv);
>>>>>>
>>>>>> err = ncmpi_open(MPI_COMM_WORLD, FILE_NAME, NC_NOWRITE,
>>>>>> MPI_INFO_NULL, &ncid);
>>>>>> CHECK_ERR;
>>>>>>
>>>>>> err = ncmpi_close(ncid);
>>>>>> CHECK_ERR;
>>>>>>
>>>>>> fn_exit:
>>>>>> MPI_Finalize();
>>>>>>
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> University of Tasmania Electronic Communications Policy (December,
>>>>>> 2014).
>>>>>> This email is confidential, and is for the intended recipient only.
>>>>>> Access, disclosure, copying, distribution, or reliance on any of it
>>>>>>by
>>>>>> anyone outside the intended recipient organisation is prohibited and
>>>>>> may
>>>>>> be a criminal offence. Please delete if obtained in error and email
>>>>>> confirmation to the sender. The views expressed in this email are
>>>>>>not
>>>>>> necessarily the views of the University of Tasmania, unless clearly
>>>>>> intended otherwise.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jim Edwards
>>>>>>
>>>>>> CESM Software Engineer
>>>>>> National Center for Atmospheric Research
>>>>>> Boulder, CO
>>>>>
>>>>
>>>>
>>>>
>>>> University of Tasmania Electronic Communications Policy (December,
>>>> 2014).
>>>> This email is confidential, and is for the intended recipient only.
>>>> Access, disclosure, copying, distribution, or reliance on any of it by
>>>> anyone outside the intended recipient organisation is prohibited and
>>>>may
>>>> be a criminal offence. Please delete if obtained in error and email
>>>> confirmation to the sender. The views expressed in this email are not
>>>> necessarily the views of the University of Tasmania, unless clearly
>>>> intended otherwise.
>>>
>>
>>
>>
>> University of Tasmania Electronic Communications Policy (December,
>>2014).
>> This email is confidential, and is for the intended recipient only.
>>Access, disclosure, copying, distribution, or reliance on any of it by
>>anyone outside the intended recipient organisation is prohibited and may
>>be a criminal offence. Please delete if obtained in error and email
>>confirmation to the sender. The views expressed in this email are not
>>necessarily the views of the University of Tasmania, unless clearly
>>intended otherwise.
>

University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.