parallel NetCDF: One or more variable sizes violate format constraints

[Wei-keng Liao]

Hi, Felicity

According to "Netcdf classic format limitations", there is one possibility your file is still valid.
"If you don't use the unlimited dimension, only one variable can exceed 2 GiB in size, but it can be as large as the underlying file system permits. It must be the last variable in the dataset, and the offset to the beginning of this variable must be less than about 2 GiB."

Maybe your file is exactly the same described above.

Can you run command "ncdump -h file.nc" and send me the output?


Wei-keng

On Nov 9, 2015, at 10:58 PM, Felicity Graham wrote:

> Hi Wei-keng and Jim,
> 
> Thanks very much for your replies.
> 
> I think the file may be corrupted, as I can¹t read even a small segment in
> serial.
> 
> I was wondering about that note in the ³Netcdf classic format limitations²
> documentation. But, ncdump -k on the file returns Œclassic¹, so it
> obviously hadn¹t been enabled at creation.
> 
> Thanks again,
> Felicity
> 
> 
> On 10/11/2015 3:51 pm, "Wei-keng Liao" <wkliao at eecs.northwestern.edu>
> wrote:
> 
>> Hi, Felicity and Hi, Jim
>> 
>> From netCDF 3.6.3 C Interface Guide, NC_64BIT_OFFSET is supported. See URL
>> http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netcdf-c/n
>> c_005fcreate.html#nc_005fcreate
>> This flag, when used at file creation, allows creating a single file of
>> size larger than 4 GB,
>> but the number of elements in each array variable is still limited by 2
>> billions.
>> 
>> You can verify whether or not your file is created with this feature
>> enabled by running command
>> 'ncdump -k file.nc' and the output should be '64-bit offset'. If you got
>> 'classic', then
>> this feature is not enabled and your file may be corrupt, as Jim guessed.
>> 
>> In PnetCDF, command 'ncmpidump -k' can also be used in the similar way.
>> 
>> 
>> Wei-keng
>> 
>> On Nov 9, 2015, at 9:24 PM, Jim Edwards wrote:
>> 
>>> The netcdf-3 classic file is incorrect and probably corrupt -  16GB
>>> exceeds the variable size constraints of that format.  You can use CDF5
>>> as output by pnetcdf to write the file and then read it again with
>>> pnetcdf but netcdf doesn't support the CDF5 standard until version 4.4.
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Nov 9, 2015 at 7:47 PM, Felicity Graham
>>> <f.s.graham at utas.edu.au> wrote:
>>> Hello,
>>> 
>>> I¹d like to use parallel netcdf in C to read the only variable from a
>>> 16GB netcdf file and perform some operations.
>>> 
>>> The file format is classic, and was created with netcdf 3.6.3. I¹m
>>> using openmpi 1.6.3 and pnetcdf 1.4.1.
>>> 
>>> I¹ve tested on a separate netcdf-4 formatted file, and the reading
>>> works fine, but on the classic file, I get the error  NetCDF: One or
>>> more variable sizes violate format constraints when I try to open the
>>> file (ncmpi_open).
>>> 
>>> A stripped down version of the C code is below.
>>> 
>>> Your help is much appreciated.
>>> 
>>> Cheers,
>>> Felicity
>>> 
>>> 
>>> #include <stdio.h>
>>> #include <mpi.h>
>>> #include <pnetcdf.h>
>>> 
>>> #define FILE_NAME ³<path_to_file>/hf_topog.nc"
>>> #define VAR_NAME "hf_topog"
>>> 
>>> #define CHECK_ERR {\
>>>    if (err!=NC_NOERR) {\
>>>        printf("Error at line=%d: %s\n", __LINE__,
>>> ncmpi_strerror(err));\
>>>        goto fn_exit;\
>>>    }}
>>> 
>>> int main(int argc, char **argv)
>>> {
>>>  int err, varid, ncid;
>>>  float * topog_in;
>>> 
>>>  MPI_Init(&argc,&argv);
>>> 
>>>  err = ncmpi_open(MPI_COMM_WORLD, FILE_NAME, NC_NOWRITE,
>>> MPI_INFO_NULL, &ncid);
>>>  CHECK_ERR;
>>> 
>>>  err = ncmpi_close(ncid);
>>>  CHECK_ERR;
>>> 
>>> fn_exit:
>>>  MPI_Finalize();
>>> 
>>>  return 0;
>>> }
>>> 
>>> 
>>> 
>>> 
>>> University of Tasmania Electronic Communications Policy (December,
>>> 2014).
>>> This email is confidential, and is for the intended recipient only.
>>> Access, disclosure, copying, distribution, or reliance on any of it by
>>> anyone outside the intended recipient organisation is prohibited and may
>>> be a criminal offence. Please delete if obtained in error and email
>>> confirmation to the sender. The views expressed in this email are not
>>> necessarily the views of the University of Tasmania, unless clearly
>>> intended otherwise.
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jim Edwards
>>> 
>>> CESM Software Engineer
>>> National Center for Atmospheric Research
>>> Boulder, CO
>> 
> 
> 
> 
> University of Tasmania Electronic Communications Policy (December, 2014).
> This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.