Hi Russ,<br><br>That turns out to have been the problem. The original file was created with pnetcdf. <br><br>Jim<br><br><br><br><div class="gmail_quote">On Mon, Mar 4, 2013 at 3:12 PM, Jim Edwards <span dir="ltr"><<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Russ,<br><br>We think that the original file may have been written with pnetcdf. We are going to try to recreate the file with netcdf and again with pnetcdf and see if that explains the issue. <br>
<span class="HOEnZb"><font color="#888888"><br>Jim</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">
On Mon, Mar 4, 2013 at 2:31 PM, Samuel Levis <span dir="ltr"><<a href="mailto:slevis@ucar.edu" target="_blank">slevis@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Not exactly. I tried 2-degree to 2-degree, 2-degree to 0.5, 2-degree
to 0.25, and others. All cases worked except the ones with the
0.5-degree file as output.<br>
<br>
I also tried 0.5-degree to 0.5-degree (mapping the file into itself)
and that failed. When I say failed, I mean that the output file ends
up with junk in it.<br>
<br>
Sam<div><div><br>
<br>
<div>On 03/04/2013 02:26 PM, Jim Edwards
wrote:<br>
</div>
<blockquote type="cite">Hi Russ,<br>
<br>
Another piece of information. This program interpolates data
from a file of one resolution (2 degree in this case) to another.
When the output file is low resolution, 1/2 degree or lower, the
output file looks fine, no corruption that we can detect. It's
only when the output file is higher resolution (1/4 degree) that
this problem comes about. <br>
<br>
Jim<br>
<br>
<div class="gmail_quote">On Mon, Mar 4, 2013 at 2:04 PM, Jim
Edwards <span dir="ltr"><<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Russ,<br>
<br>
It looks like that file was originally created on bluefire on
11/21/11, I don't have any information about which netcdf
library was used, but I think that some adjustment may have
been made inside netcdf for performance on gpfs
filesystems. <br>
<br>
But doesn't your own <br>
<pre>int nc__enddef(int ncid, size_t h_minfree, size_t v_align,
size_t v_minfree, size_t r_align);
</pre>
allow for changing this alignment? I don't know that that
was done for this file, but it would seem to suggest that
there is no assumption being violated about these alignments.
Or that one part of netcdf is assuming something which another
part is not. <br>
<div>
<div>
<br>
<br>
<br>
<div class="gmail_quote">On Mon, Mar 4, 2013 at 12:53 PM,
Unidata netCDF Support <span dir="ltr"><<a href="mailto:support-netcdf@unidata.ucar.edu" target="_blank">support-netcdf@unidata.ucar.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
Jim,<br>
<br>
I'm curious how the original file you provided was
created and perhaps<br>
modified. It has a peculiar alignment characteristic
that I haven't<br>
seen before, and if there are more netCDF files being
created the same<br>
way, we may nned to adapt.<br>
<br>
Could you tell me the history of the file, what file
system it was<br>
written on, and whether the netCDF library with which
it was written<br>
was modified in any way?<br>
<br>
The file has this characteristic, which would indicate
a non-Posix<br>
file system: it is using 512-byte alignment of data
values rather than<br>
the 4-byte alignment assumed by netCDF. So, for
example, the data<br>
block for fixed-size variables begins with 9 scalar
integers that<br>
should take 4 bytes each. The offsets computed for
these values from<br>
the beginning of the fixed-size data block are 0, 4,
8, 12, 16, 20,<br>
24, 28, 32, so there is no padding or wasted space.
The offsets from<br>
the beginning of the fixed-size data block that are
actually stored in the<br>
header for these variables are 0, 512, 1024, ... ,
4096. If the file<br>
system used to write the data originally could not
write data on<br>
4-byte boundaries, I think that violates the
assumption of netCDF and<br>
POSIX I/O. Nevertheless, if the nc_endef() call pays
attention to the<br>
file offsets for each variable that are stored in the
header (as the<br>
netCDF library does when reading the file), rather
than computing them<br>
from assuming 4-byte alignment, perhaps this file can
be modified<br>
correctly.<br>
<br>
The function where we might be able to adapt to this
is<br>
nc3internal.c:NC_begins(), which is called from<br>
nc3internal.c:NC_enddef(). In any case it's a netCDF
bug to write<br>
something that can't be later read correctly, so if
our unmodified<br>
library wrote that file and we can't adapt to it, then
it was a bug<br>
to not emit an error message for trying to create a
file on the original<br>
non-POSIX file system. Also, the data seems to all be
there in the<br>
"corrupted" file, which can be fixed by just restoring
the variable<br>
offsets in the file header to the peculiar values in
the original ...<br>
<br>
--Russ<br>
<br>
Russ Rew UCAR
Unidata Program<br>
<a href="mailto:russ@unidata.ucar.edu" target="_blank">russ@unidata.ucar.edu</a>
<a href="http://www.unidata.ucar.edu" target="_blank">http://www.unidata.ucar.edu</a><br>
<br>
<br>
<br>
Ticket Details<br>
===================<br>
Ticket ID: KLB-596506<br>
Department: Support netCDF<br>
Priority: Normal<br>
Status: Closed<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
<span><font color="#888888">-- <br>
Jim Edwards<br>
<br>
<font>CESM Software Engineering Group<br>
National Center for Atmospheric Research<br>
Boulder, CO <br>
<a href="tel:303-497-1842" value="+13034971842" target="_blank">303-497-1842</a><br>
</font>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Jim Edwards<br>
<br>
<font>CESM Software Engineering Group<br>
National Center for Atmospheric Research<br>
Boulder, CO <br>
<a href="tel:303-497-1842" value="+13034971842" target="_blank">303-497-1842</a><br>
</font>
</blockquote>
<br>
</div></div><span><font color="#888888"><pre cols="72">--
Samuel Levis - <a href="mailto:slevis@ucar.edu" target="_blank">slevis@ucar.edu</a>
National Center for Atmospheric Research
PO Box 3000, Boulder CO 80307-3000 <- use for mail
3090 Center Green Dr., Boulder CO 80301 <- vs. shipping
tel <a href="tel:303%20497-1627" value="+13034971627" target="_blank">303 497-1627</a>; fax -1348; skype: samuellevis2
<a href="http://www.cgd.ucar.edu/tss" target="_blank">http://www.cgd.ucar.edu/tss</a>
Terrestrial Sciences Section in the
Climate & Global Dynamics Division
</pre>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>Jim Edwards<br><br><font>CESM Software Engineering Group<br>National Center for Atmospheric Research<br>Boulder, CO <br><a href="tel:303-497-1842" value="+13034971842" target="_blank">303-497-1842</a><br>
</font>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Jim Edwards<br><br><font>CESM Software Engineering Group<br>National Center for Atmospheric Research<br>Boulder, CO <br>303-497-1842<br></font>