values sporadically set to zero in output arrays

[Ross, Robert B.]

Is there any NFS in the loop? -- Rob

On Aug 17, 2015, at 11:49 AM, Wei-keng Liao <wkliao at eecs.northwestern.edu> wrote:

> Hi, Andreas
> 
> PnetCDF does not do buffering and neither does MPI-IO in OpenMPI.
> 
> There is a possibility that the file is written correctly (without zeros)
> but the read application failed to read the data. If the variable is not too big,
> could you try command "ncdump" to check for zero values? If no zeros were
> found, then it is either the read application or the file system.
> If ncdump did show zero values, then could you provide a short program that
> can reproduce the problem?
> 
> Wei-keng
> 
> On Aug 17, 2015, at 11:07 AM, Pfeiffer, Andreas wrote:
> 
>> Hi,
>> 
>> I am currently investigating an issue with pnetcdf (1.6.1) writing out arrays with values wrongly set to zero in some part of my overall domain of a global climate model (see examples in attachment). This is happening sporadically only for a very small part of my overall dataset and only for very few points in time of my simulation. It is reproducible in a sense that it can be observed for every single simulation. However, the zeros show up for different variables at different dates for a simulation that is run several times with the identical setup.
>> 
>> My first guess would be that this is due to some unfortunate settings for buffering/flushing e.g. in my pnetcdf and/or mpi-io implementation. But this might well be the wrong starting point for my investigation.
>> 
>> The examples have been simulated on our cluster (gpfs filesystem version 3.4.0-9) based on openmpi 1.6.5 and compiler intel 14.0.4. Tests with GNU compilers 4.8.1 did show the same effect.
>> 
>> On a totally different system I did not encounter this problem (so far).
>> 
>> Any hints would be highly appreciated.
>> 
>> Thanks,
>> Andreas
>> 
>> -- 
>> --------------------------------------------------------
>> Dr. Andreas Pfeiffer
>> Institut fuer Physik der Atmosphaere
>> DLR Oberpfaffenhofen
>> Muenchner Str. 20
>> D-82234 Wessling
>> Germany
>> Tel.: +49-8153-28-3958
>> Fax.: +49-8153-28-1841
>> Email: a.pfeiffer at dlr.de
>> www.dlr.de/ipa
>> --------------------------------------------------------
>> <sigmadot.png><o3.png>
>