CDF-2 Bug fix for netcdf 3.6.0

[Robert Latham]

The serial netcdf folks found a bug in the way they handle CDF-2
formatted datasets (datasets larger than 2GB).  I've fixed this in
CVS and it will be in the next release.   In the meantime, take note
of the situation under which this bug is triggered.  

If you think you will be affected by this bug, you can apply this
patch to parallel-netcdf:

http://www.mcs.anl.gov/~robl//pnetcdf/cdf2-redefine.patch

==rob

----- Forwarded message from Russ Rew <russ at unidata.ucar.edu> -----

Sender: owner-netcdfgroup at unidata.ucar.edu
From: Russ Rew <russ at unidata.ucar.edu>
Reply-To: Russ Rew <russ at unidata.ucar.edu>
Subject: Bug fix for netcdf 3.6.0
Organization: UCAR Unidata Program
Date: Thu, 17 Feb 2005 16:15:17 -0700
Message-Id: <200502172315.j1HNFMv2022896 at unidata.ucar.edu>
To: netcdfgroup at unidata.ucar.edu

Hi,

A potentially serious bug has been discovered in large file support in
netCDF-3.6.0:

  http://www.unidata.ucar.edu/packages/netcdf/docs/known_problems.html#cdf2-bug

Until you can upgrade, avoid rewriting in place any large (> 2 GiByte)
netCDF files that use the new 64-bit offset format under the
conditions described below.

You can upgrade the 3.6.0 library with this patch that adds one line
to the file libsrc/v1hpg.c:

  http://www.unidata.ucar.edu/packages/netcdf/patches/patch-3.6.0-cdf2

or get the patched version 3.6.0-p1 from one of 

  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.Z
  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.gz

or install the current beta release (3.6.1-beta3) from one of

  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-beta.tar.Z
  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-beta.tar.gz

The bug occurs following this sequence of steps:

   1. A large netCDF file is created using the new 64-bit offset
      format variant (also known as the version 2 format) and the file
      is closed.
   2. Later the file is opened for writing, followed by either of the
      following operations: 
      
         - enter define mode (calling nc_redef(), nf_redef(), or
           nf90_redef(), in C, Fortran-77, or Fortran-90 interface, for
           example) to add a new dimension, variable, or attribute; or 
      
         - write a new value for an existing attribute (either a
           global or a variable-specific attribute). 

Under these conditions, after you leave define mode or close the file,
the file header is written out in the "classic" (version 1) netCDF
format, chopping the leading bits off any variable offsets that were
large enough to require more than 32 bits. If there were no such huge
variable offsets, the file is undamaged and remains readable as a
classic netCDF file. If there were any huge variable offsets (> 2
GiB), data for the first such variable and all subsequent variables
will not be accessed correctly. 

It is possible to restore the header for such a file to the correct
64-bit offset form so that the data can subsequently be accessed
correctly, if no data values have been overwritten since the file
header was changed to classic format. Feel free to contact us for help
restoring the file headers if this applies to you. If you have any
large 64-bit offset format netCDF files that might have mistakenly
been rewritten with classic format headers, please be careful not to
write any more data into them, as it could overwrite data that could
not subsequently be recovered.

To determine if a 64-bit offset file has been converted by this bug
into a classic format file, see the FAQ "How can I tell if a netCDF
file uses the classic format or new 64-bit offset format?" from

  http://www.unidata.ucar.edu/packages/netcdf/faq.html

We're very sorry for any problems and inconvenience caused by this
bug.  Our testing just missed it, but we've added more extensive
testing in version 3.6.1.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ at unidata.ucar.edu          http://www.unidata.ucar.edu/staff/russ






       




----- End forwarded message -----

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B