values sporadically set to zero in output arrays

[Pfeiffer, Andreas]

Hi Rob,

I checked with the guys maintaining the cluster - there is no NFS involved.

Thanks,
Andreas


Ross, Robert B. wrote:
> Is there any NFS in the loop? -- Rob
>
> On Aug 17, 2015, at 11:49 AM, Wei-keng Liao
> <wkliao at eecs.northwestern.edu> wrote:
>
>> Hi, Andreas
>>
>> PnetCDF does not do buffering and neither does MPI-IO in OpenMPI.
>>
>> There is a possibility that the file is written correctly (without
>> zeros) but the read application failed to read the data. If the
>> variable is not too big, could you try command "ncdump" to check
>> for zero values? If no zeros were found, then it is either the read
>> application or the file system. If ncdump did show zero values,
>> then could you provide a short program that can reproduce the
>> problem?
>>
>> Wei-keng
>>
>> On Aug 17, 2015, at 11:07 AM, Pfeiffer, Andreas wrote:
>>
>>> Hi,
>>>
>>> I am currently investigating an issue with pnetcdf (1.6.1)
>>> writing out arrays with values wrongly set to zero in some part
>>> of my overall domain of a global climate model (see examples in
>>> attachment). This is happening sporadically only for a very small
>>> part of my overall dataset and only for very few points in time
>>> of my simulation. It is reproducible in a sense that it can be
>>> observed for every single simulation. However, the zeros show up
>>> for different variables at different dates for a simulation that
>>> is run several times with the identical setup.
>>>
>>> My first guess would be that this is due to some unfortunate
>>> settings for buffering/flushing e.g. in my pnetcdf and/or mpi-io
>>> implementation. But this might well be the wrong starting point
>>> for my investigation.
>>>
>>> The examples have been simulated on our cluster (gpfs filesystem
>>> version 3.4.0-9) based on openmpi 1.6.5 and compiler intel
>>> 14.0.4. Tests with GNU compilers 4.8.1 did show the same effect.
>>>
>>> On a totally different system I did not encounter this problem
>>> (so far).
>>>
>>> Any hints would be highly appreciated.
>>>
>>> Thanks, Andreas
>>>
>>> -- -------------------------------------------------------- Dr.
>>> Andreas Pfeiffer Institut fuer Physik der Atmosphaere DLR
>>> Oberpfaffenhofen Muenchner Str. 20 D-82234 Wessling Germany Tel.:
>>> +49-8153-28-3958 Fax.: +49-8153-28-1841 Email: a.pfeiffer at dlr.de
>>> www.dlr.de/ipa
>>> --------------------------------------------------------
>>> <sigmadot.png><o3.png>
>>
>
> .
>

-- 
--------------------------------------------------------
Dr. Andreas Pfeiffer
Institut fuer Physik der Atmosphaere
DLR Oberpfaffenhofen
Muenchner Str. 20
D-82234 Wessling
Germany
Tel.: +49-8153-28-3958
Fax.: +49-8153-28-1841
Email: a.pfeiffer at dlr.de
www.dlr.de/ipa
--------------------------------------------------------