parallel NetCDF: One or more variable sizes violate format constraints

[Wei-keng Liao]

Hi, Felicity and Hi, Jim

From netCDF 3.6.3 C Interface Guide, NC_64BIT_OFFSET is supported. See URL
http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netcdf-c/nc_005fcreate.html#nc_005fcreate
This flag, when used at file creation, allows creating a single file of size larger than 4 GB,
but the number of elements in each array variable is still limited by 2 billions.

You can verify whether or not your file is created with this feature enabled by running command
'ncdump -k file.nc' and the output should be '64-bit offset'. If you got 'classic', then
this feature is not enabled and your file may be corrupt, as Jim guessed.

In PnetCDF, command 'ncmpidump -k' can also be used in the similar way.


Wei-keng

On Nov 9, 2015, at 9:24 PM, Jim Edwards wrote:

> The netcdf-3 classic file is incorrect and probably corrupt -  16GB exceeds the variable size constraints of that format.  You can use CDF5 as output by pnetcdf to write the file and then read it again with pnetcdf but netcdf doesn't support the CDF5 standard until version 4.4.      
> 
> 
> 
> On Mon, Nov 9, 2015 at 7:47 PM, Felicity Graham <f.s.graham at utas.edu.au> wrote:
> Hello, 
> 
> I’d like to use parallel netcdf in C to read the only variable from a 16GB netcdf file and perform some operations. 
> 
> The file format is classic, and was created with netcdf 3.6.3. I’m using openmpi 1.6.3 and pnetcdf 1.4.1.
> 
> I’ve tested on a separate netcdf-4 formatted file, and the reading works fine, but on the classic file, I get the error  NetCDF: One or more variable sizes violate format constraints when I try to open the file (ncmpi_open).
> 
> A stripped down version of the C code is below.
> 
> Your help is much appreciated.
> 
> Cheers,
> Felicity
> 
> 
> #include <stdio.h>
> #include <mpi.h>
> #include <pnetcdf.h>
> 
> #define FILE_NAME “<path_to_file>/hf_topog.nc"
> #define VAR_NAME "hf_topog"
> 
> #define CHECK_ERR {\
>     if (err!=NC_NOERR) {\
>         printf("Error at line=%d: %s\n", __LINE__, ncmpi_strerror(err));\
>         goto fn_exit;\
>     }}
> 
> int main(int argc, char **argv)
> {
>   int err, varid, ncid;
>   float * topog_in;
> 
>   MPI_Init(&argc,&argv);
> 
>   err = ncmpi_open(MPI_COMM_WORLD, FILE_NAME, NC_NOWRITE, MPI_INFO_NULL, &ncid);
>   CHECK_ERR;
> 
>   err = ncmpi_close(ncid);
>   CHECK_ERR;
> 
> fn_exit:
>   MPI_Finalize();
> 
>   return 0;
> }
> 
> 
> 
> 
> University of Tasmania Electronic Communications Policy (December, 2014). 
> This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
> 
> 
> 
> 
> -- 
> Jim Edwards
> 
> CESM Software Engineer
> National Center for Atmospheric Research
> Boulder, CO