values sporadically set to zero in output arrays

[Pfeiffer, Andreas]

Hi Wei-keng,

I checked the file as suggested. However, the zero values show up in the
ncdump output where I had to expect them from my plots.

I will extract the relevant output routines from my model to prepare a 
reasonable small test program to reproduce this issue.
On the other hand, could you recommend some of the test programs shipped 
with pnetcdf as a starting point for some more tests?

Thanks,
Andreas


Wei-keng Liao wrote:
> Hi, Andreas
>
> PnetCDF does not do buffering and neither does MPI-IO in OpenMPI.
>
> There is a possibility that the file is written correctly (without
> zeros) but the read application failed to read the data. If the
> variable is not too big, could you try command "ncdump" to check for
> zero values? If no zeros were found, then it is either the read
> application or the file system. If ncdump did show zero values, then
> could you provide a short program that can reproduce the problem?
>
> Wei-keng
>
> On Aug 17, 2015, at 11:07 AM, Pfeiffer, Andreas wrote:
>
>> Hi,
>>
>> I am currently investigating an issue with pnetcdf (1.6.1) writing
>> out arrays with values wrongly set to zero in some part of my
>> overall domain of a global climate model (see examples in
>> attachment). This is happening sporadically only for a very small
>> part of my overall dataset and only for very few points in time of
>> my simulation. It is reproducible in a sense that it can be
>> observed for every single simulation. However, the zeros show up
>> for different variables at different dates for a simulation that is
>> run several times with the identical setup.
>>
>> My first guess would be that this is due to some unfortunate
>> settings for buffering/flushing e.g. in my pnetcdf and/or mpi-io
>> implementation. But this might well be the wrong starting point for
>> my investigation.
>>
>> The examples have been simulated on our cluster (gpfs filesystem
>> version 3.4.0-9) based on openmpi 1.6.5 and compiler intel 14.0.4.
>> Tests with GNU compilers 4.8.1 did show the same effect.
>>
>> On a totally different system I did not encounter this problem (so
>> far).
>>
>> Any hints would be highly appreciated.
>>
>> Thanks, Andreas
>>
>> -- -------------------------------------------------------- Dr.
>> Andreas Pfeiffer Institut fuer Physik der Atmosphaere DLR
>> Oberpfaffenhofen Muenchner Str. 20 D-82234 Wessling Germany Tel.:
>> +49-8153-28-3958 Fax.: +49-8153-28-1841 Email: a.pfeiffer at dlr.de
>> www.dlr.de/ipa
>> --------------------------------------------------------
>> <sigmadot.png><o3.png>
>
> .
>

-- 
--------------------------------------------------------
Dr. Andreas Pfeiffer
Institut fuer Physik der Atmosphaere
DLR Oberpfaffenhofen
Muenchner Str. 20
D-82234 Wessling
Germany
Tel.: +49-8153-28-3958
Fax.: +49-8153-28-1841
Email: a.pfeiffer at dlr.de
www.dlr.de/ipa
--------------------------------------------------------