[netCDF #KLB-596506]: apparent bug in netcdf-4.2
Unidata netCDF Support
support-netcdf at unidata.ucar.edu
Mon Mar 4 22:02:23 CST 2013
Jim,
> That turns out to have been the problem. The original file was created
> with pnetcdf.
So now I'm a little confused about the relationship between pnetcdf and netCDF.
Except for pnetcdf's CDF-5 format (64-bit everything), I thought their software could
read and write both netCDF-3 formats, "classic" and 64-bit offset. But your comment
seems to imply that files created with pnetcdf might not be modifiable by our netCDF
software. I didn't see any warnings about this limitation in the parallel-netCDF
documentation, and it seems like a problem that our software can open and try to
write to a pnetcdf file in a way that seems to silently corrupt the variable offsets in the
file header.
Is this all due to a difference in interpretation of what the v_align parameter means in
the nc__enddef(ncid, h_minfree, v_align, v_minfree, r_align) function?
--Russ
> On Mon, Mar 4, 2013 at 3:12 PM, Jim Edwards <jedwards at ucar.edu> wrote:
>
> > Russ,
> >
> > We think that the original file may have been written with pnetcdf. We
> > are going to try to recreate the file with netcdf and again with pnetcdf
> > and see if that explains the issue.
> >
> > Jim
> >
> >
> > On Mon, Mar 4, 2013 at 2:31 PM, Samuel Levis <slevis at ucar.edu> wrote:
> >
> >> Not exactly. I tried 2-degree to 2-degree, 2-degree to 0.5, 2-degree to
> >> 0.25, and others. All cases worked except the ones with the 0.5-degree file
> >> as output.
> >>
> >> I also tried 0.5-degree to 0.5-degree (mapping the file into itself) and
> >> that failed. When I say failed, I mean that the output file ends up with
> >> junk in it.
> >>
> >> Sam
> >>
> >>
> >> On 03/04/2013 02:26 PM, Jim Edwards wrote:
> >>
> >> Hi Russ,
> >>
> >> Another piece of information. This program interpolates data from a
> >> file of one resolution (2 degree in this case) to another. When the output
> >> file is low resolution, 1/2 degree or lower, the output file looks fine, no
> >> corruption that we can detect. It's only when the output file is higher
> >> resolution (1/4 degree) that this problem comes about.
> >>
> >> Jim
> >>
> >> On Mon, Mar 4, 2013 at 2:04 PM, Jim Edwards <jedwards at ucar.edu> wrote:
> >>
> >>> Hi Russ,
> >>>
> >>> It looks like that file was originally created on bluefire on 11/21/11,
> >>> I don't have any information about which netcdf library was used, but I
> >>> think that some adjustment may have been made inside netcdf for performance
> >>> on gpfs filesystems.
> >>>
> >>> But doesn't your own
> >>>
> >>> int nc__enddef(int ncid, size_t h_minfree, size_t v_align,
> >>> size_t v_minfree, size_t r_align);
> >>>
> >>>
> >>> allow for changing this alignment? I don't know that that was done for
> >>> this file, but it would seem to suggest that there is no assumption being
> >>> violated about these alignments. Or that one part of netcdf is assuming
> >>> something which another part is not.
> >>>
> >>>
> >>>
> >>> On Mon, Mar 4, 2013 at 12:53 PM, Unidata netCDF Support <
> >>> support-netcdf at unidata.ucar.edu> wrote:
> >>>
> >>>> Hi Jim,
> >>>>
> >>>> I'm curious how the original file you provided was created and perhaps
> >>>> modified. It has a peculiar alignment characteristic that I haven't
> >>>> seen before, and if there are more netCDF files being created the same
> >>>> way, we may nned to adapt.
> >>>>
> >>>> Could you tell me the history of the file, what file system it was
> >>>> written on, and whether the netCDF library with which it was written
> >>>> was modified in any way?
> >>>>
> >>>> The file has this characteristic, which would indicate a non-Posix
> >>>> file system: it is using 512-byte alignment of data values rather than
> >>>> the 4-byte alignment assumed by netCDF. So, for example, the data
> >>>> block for fixed-size variables begins with 9 scalar integers that
> >>>> should take 4 bytes each. The offsets computed for these values from
> >>>> the beginning of the fixed-size data block are 0, 4, 8, 12, 16, 20,
> >>>> 24, 28, 32, so there is no padding or wasted space. The offsets from
> >>>> the beginning of the fixed-size data block that are actually stored in
> >>>> the
> >>>> header for these variables are 0, 512, 1024, ... , 4096. If the file
> >>>> system used to write the data originally could not write data on
> >>>> 4-byte boundaries, I think that violates the assumption of netCDF and
> >>>> POSIX I/O. Nevertheless, if the nc_endef() call pays attention to the
> >>>> file offsets for each variable that are stored in the header (as the
> >>>> netCDF library does when reading the file), rather than computing them
> >>>> from assuming 4-byte alignment, perhaps this file can be modified
> >>>> correctly.
> >>>>
> >>>> The function where we might be able to adapt to this is
> >>>> nc3internal.c:NC_begins(), which is called from
> >>>> nc3internal.c:NC_enddef(). In any case it's a netCDF bug to write
> >>>> something that can't be later read correctly, so if our unmodified
> >>>> library wrote that file and we can't adapt to it, then it was a bug
> >>>> to not emit an error message for trying to create a file on the original
> >>>> non-POSIX file system. Also, the data seems to all be there in the
> >>>> "corrupted" file, which can be fixed by just restoring the variable
> >>>> offsets in the file header to the peculiar values in the original ...
> >>>>
> >>>> --Russ
> >>>>
> >>>> Russ Rew UCAR Unidata Program
> >>>> russ at unidata.ucar.edu http://www.unidata.ucar.edu
> >>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: KLB-596506
> >>>> Department: Support netCDF
> >>>> Priority: Normal
> >>>> Status: Closed
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Jim Edwards
> >>>
> >>> CESM Software Engineering Group
> >>> National Center for Atmospheric Research
> >>> Boulder, CO
> >>> 303-497-1842
> >>>
> >>
> >>
> >>
> >> --
> >> Jim Edwards
> >>
> >> CESM Software Engineering Group
> >> National Center for Atmospheric Research
> >> Boulder, CO
> >> 303-497-1842
> >>
> >>
> >> --
> >> Samuel Levis - slevis at ucar.edu
> >> National Center for Atmospheric Research
> >> PO Box 3000, Boulder CO 80307-3000 <- use for mail
> >> 3090 Center Green Dr., Boulder CO 80301 <- vs. shipping
> >>
> >> tel 303 497-1627; fax -1348; skype: samuellevis2http://www.cgd.ucar.edu/tss
> >>
> >> Terrestrial Sciences Section in the
> >> Climate & Global Dynamics Division
> >>
> >>
> >
> >
> > --
> > Jim Edwards
> >
> > CESM Software Engineering Group
> > National Center for Atmospheric Research
> > Boulder, CO
> > 303-497-1842
> >
>
>
>
> --
> Jim Edwards
>
> CESM Software Engineering Group
> National Center for Atmospheric Research
> Boulder, CO
> 303-497-1842
>
>
Russ Rew UCAR Unidata Program
russ at unidata.ucar.edu http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: KLB-596506
Department: Support netCDF
Priority: Normal
Status: Closed
More information about the parallel-netcdf
mailing list