pnetcdf bug?

[Wei-keng Liao]

Hi, Charlie

File offsets are hidden from users when using netCDF library.
So, the script option or even a C program may take some time
to write one, as one must dissect the netCDF file header to
find the offsets of two adjacent variables. I am not a script
expert, but I can try write a C program for this. 

If you want to write a script, I suggest to first check the
netCDF file header format in the URL below.

http://cucis.ece.northwestern.edu/projects/PnetCDF/doc/pnetcdf-c/CDF_002d2-file-format-specification.html#CDF_002d2-file-format-specification

Each variable "var" contains its starting file offset (begin) in the header.
    var = name nelems [dimid ...] vatt_list nc_type vsize begin

You will also need to do Endianness byte swap for the integer stored in "begin".

Sound crazy? But it will be very easy for netCDF/PnetCDF to report this
info. Maybe you want to make a request to netCDF team (something
like a new command-line option in ncdump.)

Wei-keng

On Oct 28, 2015, at 12:35 PM, Charlie Zender wrote:

> Hi Wei-keng,
> People do this all the time in custom post-processing,
> where speed/penalties are not such a big deal, and
> where combining datasets is necessary.
> 
> Thanks for you quick work identifying and solving the problem.
> 
> It would be helpful to have a script that tests a
> (p)netCDF file to see if the offset alignment is enabled,
> because I expect to triage more bug reports submitted
> to NCO where the actual cause is the alignment in the
> underlying file. Any guidance on creating such a script?
> cz
> 
> On 10/28/15 10:04, Wei-keng Liao wrote:
>> I thought adding new variables to an existing file happens rarely in netCDF applications
>> because of the high penalty to move (shift) the record variables down.
>> Is CESM doing this?
> 
> -- 
> Charlie Zender, Earth System Sci. & Computer Sci.
> University of California, Irvine 949-891-2429 )'(