values sporadically set to zero in output arrays

[Wei-keng Liao]

Hi, Andreas

PnetCDF does not do buffering and neither does MPI-IO in OpenMPI.

There is a possibility that the file is written correctly (without zeros)
but the read application failed to read the data. If the variable is not too big,
could you try command "ncdump" to check for zero values? If no zeros were
found, then it is either the read application or the file system.
If ncdump did show zero values, then could you provide a short program that
can reproduce the problem?

Wei-keng

On Aug 17, 2015, at 11:07 AM, Pfeiffer, Andreas wrote:

> Hi,
> 
> I am currently investigating an issue with pnetcdf (1.6.1) writing out arrays with values wrongly set to zero in some part of my overall domain of a global climate model (see examples in attachment). This is happening sporadically only for a very small part of my overall dataset and only for very few points in time of my simulation. It is reproducible in a sense that it can be observed for every single simulation. However, the zeros show up for different variables at different dates for a simulation that is run several times with the identical setup.
> 
> My first guess would be that this is due to some unfortunate settings for buffering/flushing e.g. in my pnetcdf and/or mpi-io implementation. But this might well be the wrong starting point for my investigation.
> 
> The examples have been simulated on our cluster (gpfs filesystem version 3.4.0-9) based on openmpi 1.6.5 and compiler intel 14.0.4. Tests with GNU compilers 4.8.1 did show the same effect.
> 
> On a totally different system I did not encounter this problem (so far).
> 
> Any hints would be highly appreciated.
> 
> Thanks,
> Andreas
> 
> -- 
> --------------------------------------------------------
> Dr. Andreas Pfeiffer
> Institut fuer Physik der Atmosphaere
> DLR Oberpfaffenhofen
> Muenchner Str. 20
> D-82234 Wessling
> Germany
> Tel.: +49-8153-28-3958
> Fax.: +49-8153-28-1841
> Email: a.pfeiffer at dlr.de
> www.dlr.de/ipa
> --------------------------------------------------------
> <sigmadot.png><o3.png>