values sporadically set to zero in output arrays

[Wei-keng Liao]

Hi, Andreas

PnetCDF has a set of programs for testing, in both sequential and parallel runs.
Once you finished the make command, you can run "make check" followed by "make ptest".
Detailed instructions can be found in file INSTALL.

If your system is a cross-compile environment, then you will have to run the above
make commands from a batch script. You need to set a few environment variables.
See file INSTALL for more info. If this is the case, I suggest you first build
all the executables of testing programs by running the commands below, so you
don't spend compile time on the batch job.
    cd test; make; cd ../examples; make; cd ..

Could you tell us a little about your I/O? Such as the array size, the PnetCDF APIs
(var, vara, vars ...) in used, blocking or nonblocking one, whether the problem
happens to record variables only, and your configure command.

There is a possibility that it can be the system problem, if your earlier
statement below is still true.
>>> On a totally different system I did not encounter this problem (so far).



Wei-keng

On Aug 21, 2015, at 10:18 AM, Pfeiffer, Andreas wrote:

> Hi Wei-keng,
> 
> I checked the file as suggested. However, the zero values show up in the
> ncdump output where I had to expect them from my plots.
> 
> I will extract the relevant output routines from my model to prepare a reasonable small test program to reproduce this issue.
> On the other hand, could you recommend some of the test programs shipped with pnetcdf as a starting point for some more tests?
> 
> Thanks,
> Andreas
> 
> 
> Wei-keng Liao wrote:
>> Hi, Andreas
>> 
>> PnetCDF does not do buffering and neither does MPI-IO in OpenMPI.
>> 
>> There is a possibility that the file is written correctly (without
>> zeros) but the read application failed to read the data. If the
>> variable is not too big, could you try command "ncdump" to check for
>> zero values? If no zeros were found, then it is either the read
>> application or the file system. If ncdump did show zero values, then
>> could you provide a short program that can reproduce the problem?
>> 
>> Wei-keng
>> 
>> On Aug 17, 2015, at 11:07 AM, Pfeiffer, Andreas wrote:
>> 
>>> Hi,
>>> 
>>> I am currently investigating an issue with pnetcdf (1.6.1) writing
>>> out arrays with values wrongly set to zero in some part of my
>>> overall domain of a global climate model (see examples in
>>> attachment). This is happening sporadically only for a very small
>>> part of my overall dataset and only for very few points in time of
>>> my simulation. It is reproducible in a sense that it can be
>>> observed for every single simulation. However, the zeros show up
>>> for different variables at different dates for a simulation that is
>>> run several times with the identical setup.
>>> 
>>> My first guess would be that this is due to some unfortunate
>>> settings for buffering/flushing e.g. in my pnetcdf and/or mpi-io
>>> implementation. But this might well be the wrong starting point for
>>> my investigation.
>>> 
>>> The examples have been simulated on our cluster (gpfs filesystem
>>> version 3.4.0-9) based on openmpi 1.6.5 and compiler intel 14.0.4.
>>> Tests with GNU compilers 4.8.1 did show the same effect.
>>> 
>>> On a totally different system I did not encounter this problem (so
>>> far).
>>> 
>>> Any hints would be highly appreciated.
>>> 
>>> Thanks, Andreas
>>> 
>>> -- -------------------------------------------------------- Dr.
>>> Andreas Pfeiffer Institut fuer Physik der Atmosphaere DLR
>>> Oberpfaffenhofen Muenchner Str. 20 D-82234 Wessling Germany Tel.:
>>> +49-8153-28-3958 Fax.: +49-8153-28-1841 Email: a.pfeiffer at dlr.de
>>> www.dlr.de/ipa
>>> --------------------------------------------------------
>>> <sigmadot.png><o3.png>
>> 
>> .
>> 
> 
> -- 
> --------------------------------------------------------
> Dr. Andreas Pfeiffer
> Institut fuer Physik der Atmosphaere
> DLR Oberpfaffenhofen
> Muenchner Str. 20
> D-82234 Wessling
> Germany
> Tel.: +49-8153-28-3958
> Fax.: +49-8153-28-1841
> Email: a.pfeiffer at dlr.de
> www.dlr.de/ipa
> --------------------------------------------------------