<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">CESM doesn't add new variables to a file. But in some cases we will create a new initial file </div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">by copying an existing initial file and adding or modifying some of the fields. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 28, 2015 at 11:04 AM, Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@eecs.northwestern.edu" target="_blank">wkliao@eecs.northwestern.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, Bill<br>
<br>
The bug happens when the offset alignment is enabled (i.e. most files created by<br>
PnetCDF library) and new variables are added to the file (by using netCDF library<br>
when re-entering define mode after opening an existing file). I agree with your<br>
suggestion to CESM users be caution if they used netCDF older than 4.4.0.<br>
<br>
Because netCDF library does not do alignment at all, one solution is to disable<br>
alignment in PnetCDF to produce non-aligned files. This can be done by passing<br>
an MPI hint or setting a run-time environment variable.<br>
MPI_Info_set(info, "nc_var_align_size", 1);<br>
setenv PNETCDF_HINTS "nc_var_align_size=1"<br>
See<br>
<a href="http://cucis.ece.northwestern.edu/projects/PnetCDF/faq.html#align" rel="noreferrer" target="_blank">http://cucis.ece.northwestern.edu/projects/PnetCDF/faq.html#align</a><br>
<a href="http://cucis.ece.northwestern.edu/projects/PnetCDF/doc/pnetcdf-c/PNETCDF_005fHINTS.html#PNETCDF_005fHINTS" rel="noreferrer" target="_blank">http://cucis.ece.northwestern.edu/projects/PnetCDF/doc/pnetcdf-c/PNETCDF_005fHINTS.html#PNETCDF_005fHINTS</a><br>
<br>
Please note disabling alignment may have an impact to the I/O performance.<br>
However, the impact is less if you use PnetCDF nonblocking APIs to aggregate<br>
multiple requests into a single one.<br>
<br>
I thought adding new variables to an existing file happens rarely in netCDF applications<br>
because of the high penalty to move (shift) the record variables down.<br>
Is CESM doing this?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Wei-keng<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Oct 28, 2015, at 7:29 AM, Bill Sacks wrote:<br>
<br>
> Hi Wei-keng,<br>
><br>
> Do you have any sense of when this bug would apply? I am telling people to use caution when doing any manipulations of files written by pnetcdf, using tools built on top of the vanilla netcdf library (i.e., not pnetcdf-based tools). Would you agree?<br>
><br>
> Thanks,<br>
> Bill<br>
><br>
>> On Oct 27, 2015, at 4:29 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
>><br>
>> Hi, Bill<br>
>><br>
>> I confirm this is a bug in netCDF. Please go ahead submit a bug to the netCDF group.<br>
>><br>
>> Below is the patch to fix this bug.<br>
>><br>
>> % diff wkliao/libsrc/nc3internal.c ../netcdf-4.3.3.1/libsrc/nc3internal.c<br>
>> 213c213<br>
>> < if ((*vpp)->begin < ncp->old->vars.value[j]->begin) {<br>
>> ---<br>
>>> if ((*vpp)->begin < ncp->old->vars.value[j]->begin)<br>
>> 218,219d217<br>
>> < index = (*vpp)->begin;<br>
>> < }<br>
>><br>
>><br>
>> I also wrote a short program (attached) that adds 2 new variables and tested<br>
>> it on your file created by PnetCDF method. I have to add a printf statement in<br>
>> netCDF library to print the variable offsets. See comments inside the test<br>
>> program. You can also send the codes to netCDF support.<br>
>><br>
>> If you decide to apply the patch to your netCDF library, please let me know<br>
>> if it works for you.<br>
>><br>
>> Wei-keng<br>
>><br>
>> <add_var.c><br>
>> On Oct 27, 2015, at 3:19 PM, Bill Sacks wrote:<br>
>><br>
>>> Hi Wei-keng,<br>
>>><br>
>>> Thanks very much for looking into this. I'm happy to submit a bug to the netCDF group if you think that's the best next step.<br>
>>><br>
>>> Superficially, this sure sounds similar to <a href="https://bugtracking.unidata.ucar.edu/browse/NCF-234" rel="noreferrer" target="_blank">https://bugtracking.unidata.ucar.edu/browse/NCF-234</a> – but maybe there are details that make it differ.<br>
>>><br>
>>> Thanks,<br>
>>> Bill<br>
>>><br>
>>>> On Oct 27, 2015, at 1:11 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
>>>><br>
>>>> Hi, Bill<br>
>>>><br>
>>>> I checked the file starting offsets for the two newly added variables.<br>
>>>> It appears that ncks (netCDF underneath) does not respect the offset<br>
>>>> alignment used in the files created by PnetCDF.<br>
>>>><br>
>>>> Your file created by netCDF has no alignment in between two adjacent variables.<br>
>>>> The other file created by PnetCDF has an alignment of 512 bytes.<br>
>>>> So, when ncks adds 2 new variables, I found the file offsets of the<br>
>>>> two new variables overlap with the last variable of the existing file.<br>
>>>> This indicates a bug in netCDF library, as ncks does not use PnetCDF library.<br>
>>>><br>
>>>> I will dig into netCDF library to see what happens internally.<br>
>>>><br>
>>>> Wei-keng<br>
>>>><br>
>>>> On Oct 27, 2015, at 1:41 PM, Bill Sacks wrote:<br>
>>>><br>
>>>>> Looking back at my notes, it seems that this problem sometimes appears in differences in actual values – i.e., it doesn't appear to just be a difference in where there are fill values.<br>
>>>>><br>
>>>>> Thank you,<br>
>>>>> Bill<br>
>>>>><br>
>>>>>> On Oct 27, 2015, at 12:30 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
>>>>>><br>
>>>>>> Hi, Bill<br>
>>>>>><br>
>>>>>> I can reproduce what you are seeing.<br>
>>>>>><br>
>>>>>> If the differences happen only to those missing array elements (fill values),<br>
>>>>>> then this is because PnetCDF supports the fill mode only in 1.6.1.<br>
>>>>>> Please note the way fill mode is used differs from netCDF. See the release note<br>
>>>>>> and example codes in<br>
>>>>>> <a href="http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/ReleaseNotes-1.6.1" rel="noreferrer" target="_blank">http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/ReleaseNotes-1.6.1</a><br>
>>>>>><br>
>>>>>> Please let me know if this is the case.<br>
>>>>>><br>
>>>>>> Wei-keng<br>
>>>>>><br>
>>>>>> On Oct 27, 2015, at 12:41 PM, Bill Sacks wrote:<br>
>>>>>><br>
>>>>>>> I have put the attachment on a public ftp server:<br>
>>>>>>><br>
>>>>>>> ftp <a href="http://ftp.cgd.ucar.edu" rel="noreferrer" target="_blank">ftp.cgd.ucar.edu</a><br>
>>>>>>><br>
>>>>>>> user name: anonymous<br>
>>>>>>> password: (your email address)<br>
>>>>>>><br>
>>>>>>> cd pub/sacks<br>
>>>>>>> get pnetcdf_bug.tar.gz<br>
>>>>>>><br>
>>>>>>> Thanks,<br>
>>>>>>> Bill<br>
>>>>>>><br>
>>>>>>>> On Oct 27, 2015, at 11:11 AM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>> Hi, Bill<br>
>>>>>>>><br>
>>>>>>>> Bug NCF-234 should not be the cause, as you are using netCDF 4.3.3.1.<br>
>>>>>>>> The fix has been applied to 4.3.0. I will take a look and get back to you.<br>
>>>>>>>><br>
>>>>>>>> Somehow your attachment did not come through my mail system.<br>
>>>>>>>> I check PnetCDF mail archive and it does not appear there either.<br>
>>>>>>>> <a href="http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2015-October/001746.html" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2015-October/001746.html</a><br>
>>>>>>>><br>
>>>>>>>> Maybe the file is too big? If that is the case, please send it to me directly.<br>
>>>>>>>> Thanks<br>
>>>>>>>><br>
>>>>>>>> Wei-keng<br>
>>>>>>>><br>
>>>>>>>> On Oct 27, 2015, at 10:36 AM, Bill Sacks wrote:<br>
>>>>>>>><br>
>>>>>>>>> I wonder if this could be related to this (fixed) bug:<br>
>>>>>>>>><br>
>>>>>>>>> <a href="https://bugtracking.unidata.ucar.edu/browse/NCF-234" rel="noreferrer" target="_blank">https://bugtracking.unidata.ucar.edu/browse/NCF-234</a><br>
>>>>>>>>><br>
>>>>>>>>> As with that one, it's possible that the problem is actually in netCDF and not in pnetcdf. Does anyone have an idea for how to determine if this is a pnetcdf problem or a netcdf problem? Or should I go ahead and post this to the netcdf bug list as well?<br>
>>>>>>>>><br>
>>>>>>>>> Charlie: I'm feeling more and more that NCO is probably off the hook here: sorry for dragging you into this initially :-)<br>
>>>>>>>>><br>
>>>>>>>>> Bill<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>>> On Oct 27, 2015, at 9:21 AM, Bill Sacks <<a href="mailto:wsacks@gmail.com">wsacks@gmail.com</a>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> Hi,<br>
>>>>>>>>>><br>
>>>>>>>>>> I have run into what appears to be a bug in pnetcdf: I have a file written by pnetcdf (via CESM). When I try to append a variable onto it using ncks -A, the new variable gets written properly, but a different variable on the file gets garbage values put into it. If the original file is written with standard netcdf rather than pnetcdf, the problem does not occur.<br>
>>>>>>>>>><br>
>>>>>>>>>> I am attaching a tar file that contains files needed to see the problem. It contains two restart files written by CESM (file names beginning check_ncks...): one written with pnetcdf and one with standard netcdf (the latter has "netcdf" in its name). It also contains a third file from which I was trying to copy variables onto this file.<br>
>>>>>>>>>><br>
>>>>>>>>>> To reproduce:<br>
>>>>>>>>>><br>
>>>>>>>>>> cp <a href="http://check_ncks_problem_noInterp_1027.clm2.r.0001-01-01-01800.nc" rel="noreferrer" target="_blank">check_ncks_problem_noInterp_1027.clm2.r.0001-01-01-01800.nc</a> <a href="http://test.nc" rel="noreferrer" target="_blank">test.nc</a><br>
>>>>>>>>>> ncks -A -v COL_Z_p,LEVGRND_CLASS_p <a href="http://finidat_interp_dest.nc" rel="noreferrer" target="_blank">finidat_interp_dest.nc</a> <a href="http://test.nc" rel="noreferrer" target="_blank">test.nc</a><br>
>>>>>>>>>> ncdump -v plant_nalloc <a href="http://check_ncks_problem_noInterp_1027.clm2.r.0001-01-01-01800.nc" rel="noreferrer" target="_blank">check_ncks_problem_noInterp_1027.clm2.r.0001-01-01-01800.nc</a> > dump1<br>
>>>>>>>>>> ncdump -v plant_nalloc <a href="http://test.nc" rel="noreferrer" target="_blank">test.nc</a> > dump2<br>
>>>>>>>>>> diff dump1 dump2 | less<br>
>>>>>>>>>><br>
>>>>>>>>>> Notice that many points that were FillValue have been replaced by garbage.<br>
>>>>>>>>>><br>
>>>>>>>>>> If you do the same thing, but using <a href="http://check_ncks_problem_noInterp_netcdf_1027.clm2.r.0001-01-01-01800.nc" rel="noreferrer" target="_blank">check_ncks_problem_noInterp_netcdf_1027.clm2.r.0001-01-01-01800.nc</a>, then the dumps are identical.<br>
>>>>>>>>>><br>
>>>>>>>>>> I originally filed a bug report with NCO <<a href="https://sourceforge.net/p/nco/bugs/84/" rel="noreferrer" target="_blank">https://sourceforge.net/p/nco/bugs/84/</a>>, but Charlie Zender and Jim Edwards both feel that this is most likely a problem in the writing of the original file, which points to a possible pnetcdf problem.<br>
>>>>>>>>>><br>
>>>>>>>>>> CESM was built with<br>
>>>>>>>>>><br>
>>>>>>>>>> module load netcdf-mpi/<a href="http://4.3.3.1" rel="noreferrer" target="_blank">4.3.3.1</a><br>
>>>>>>>>>> module load pnetcdf/1.6.0<br>
>>>>>>>>>><br>
>>>>>>>>>> (on NCAR's yellowstone machine).<br>
>>>>>>>>>><br>
>>>>>>>>>> Thank you,<br>
>>>>>>>>>> Bill<br>
>>>>>>>>>><br>
>>>>>>>>>> --<br>
>>>>>>>>>> Bill Sacks<br>
>>>>>>>>>> CESM Software Engineering Group<br>
>>>>>>>>>> National Center for Atmospheric Research<br>
>>>>>>>>>> <a href="tel:%28303%29%20497-1762" value="+13034971762">(303) 497-1762</a><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>
</div>