values sporadically set to zero in output arrays

[Pfeiffer, Andreas]

Hi,

I am currently investigating an issue with pnetcdf (1.6.1) writing out 
arrays with values wrongly set to zero in some part of my overall domain 
of a global climate model (see examples in attachment). This is 
happening sporadically only for a very small part of my overall dataset 
and only for very few points in time of my simulation. It is 
reproducible in a sense that it can be observed for every single 
simulation. However, the zeros show up for different variables at 
different dates for a simulation that is run several times with the 
identical setup.

My first guess would be that this is due to some unfortunate settings 
for buffering/flushing e.g. in my pnetcdf and/or mpi-io implementation. 
But this might well be the wrong starting point for my investigation.

The examples have been simulated on our cluster (gpfs filesystem version 
3.4.0-9) based on openmpi 1.6.5 and compiler intel 14.0.4. Tests with 
GNU compilers 4.8.1 did show the same effect.

On a totally different system I did not encounter this problem (so far).

Any hints would be highly appreciated.

Thanks,
Andreas

-- 
--------------------------------------------------------
Dr. Andreas Pfeiffer
Institut fuer Physik der Atmosphaere
DLR Oberpfaffenhofen
Muenchner Str. 20
D-82234 Wessling
Germany
Tel.: +49-8153-28-3958
Fax.: +49-8153-28-1841
Email: a.pfeiffer at dlr.de
www.dlr.de/ipa
--------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sigmadot.png
Type: image/png
Size: 93201 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20150817/8a93e77a/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: o3.png
Type: image/png
Size: 76332 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20150817/8a93e77a/attachment-0003.png>