Hi everybody,<br><br>I finally manage to remove this bug, which was of course coming from my source code!<br>The guilty: a "time" global attribute coming from the "ctime" function which of course is different across a large number of processors... I know this is silly but actually I was not expecting a check on the global attributes, especially with an error message "NC definitions mismatch".<br>
So I have to apologize for such a stupid mistake, but at the same time, it reinforces the idea to rethink this check function.<br>Thanks again.<br><br>Julien<br><br><div class="gmail_quote">2009/6/16 Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@ece.northwestern.edu" target="_blank">wkliao@ece.northwestern.edu</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
I agree this suggestion. So, there are three options.<br>
<br>
1. When pnetcdf is built with debug mode (at configure time), we will enable<br>
the consistency checking across all processes. In this case, the error<br>
is considered fatal. For debugging purpose, this should be fine. Note that<br>
the consistency checking may become costly, especially when the number of<br>
processes is large. We do not expect a production pnetcdf to be built with<br>
the debug mode.<br>
<br>
2. When built without debug mode, pnetcdf will only take process 0's inputs and<br>
ignore all others. Also, consistency checking is disabled.<br>
<br>
3. A middle ground: enable consistency checking but only process 0's inputs<br>
are used to define variables, attributes, etc. if inconsistency is detected.<br>
The error is not fatal, but only gives a warning message.<br>
<br>
If these are fine to the pnetcdf community, we will start to implement them.<br><font color="#888888">
<br>
Wei-keng</font><div><div></div><div><br>
<br>
On Jun 15, 2009, at 8:21 PM, Yu-hengTseng wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks Rob and Wei-keng,<br>
It will be a good idea to make this check as a warming only. In most realistic applications (including Community Atmospheric Model, within CCSM development), it's almost impossible to have the same dimension for the same array variable within different processes. Usually, 1 or 2 shift. Cheers,<br>
Yu-heng Tseng<br>
Department of Atmospheric Sciences<br>
National Taiwan University<br>
No. 1, Sec. 4, Roosevelt Rd, Taipei 106, Taiwan<br>
tel: 886-2-33663918<br>
email: <a href="mailto:yhtseng@as.ntu.edu.tw" target="_blank">yhtseng@as.ntu.edu.tw</a><br>
<br>
----- Original Message -----<br>
From: <a href="mailto:wkliao@ece.northwestern.edu" target="_blank">wkliao@ece.northwestern.edu</a><br>
To: <a href="mailto:wkliao@ece.northwestern.edu" target="_blank">wkliao@ece.northwestern.edu</a>;parallel-netcdf<<a href="mailto:parallel-netcdf@lists.mcs.anl.gov" target="_blank">parallel-netcdf@lists.mcs.anl.gov</a>><br>
Sent: 2009-06-16 03:00:01<br>
Subject: Re: Problem on Blue Gene/P<br>
<br>
For example, when defining a new 2D array variable and the number of<br>
processes is 2, P0 and P1.<br>
The metadata (array dimensions, attributes, etc. in define mode) must<br>
be the same between P0<br>
and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this<br>
error message will<br>
appear.<br>
<br>
Wei-keng<br>
<br>
On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:<br>
<br>
> Thanks everybody for your help.<br>
><br>
> I am afraid I don't get the point "your code is defining netcdf<br>
> variables and attributes in<br>
> a slightly different way on some MPI processes than others"...<br>
> depending on what?<br>
><br>
> Another test I could try is to unable the check made by ncpmi_enddef<br>
> if it is possible, and see which kind of output file I get.<br>
> I don't know if it is possible to do it easily without recompiling<br>
> the library.<br>
><br>
> I will try anyway the binary debugging.<br>
><br>
><br>
> 2009/6/15 Rob Latham<br>
> On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:<br>
> > While it does not create any problems on small cases, bigger cases<br>
> stop at<br>
> > the ncmpi_enddef call on some files (randomly, even with<br>
> synchronisation in<br>
> > between), saying that there is a mismatch between dimensions.<br>
> After many<br>
> > check it does not seems that there is something wrong with the<br>
> dimensions. I<br>
> > have no idea of how to solve the problem. Did anyone had similar<br>
> problem?<br>
> > Thanks for your help.<br>
><br>
> Hi Julien. Wei-keng is right: I know you've checked carefully, but<br>
> some part of your code is defining netcdf variables and attributes in<br>
> a slightly different way on some MPI processes than others.<br>
><br>
> The main way people debug this is through binary search: comment out<br>
> half of the define-mode portion; if the problem persists, comment out<br>
> half of the remainder, else, try with the other half.<br>
><br>
> You're not the first to encounter this problem. Maybe this could be a<br>
> warning and not an error, and maybe we should just have the define<br>
> mode view as rank 0 sees it be the one that wins if there's a<br>
> discrepancy. I don't know how many people (if any) rely on the<br>
> current behavior to find problems.<br>
><br>
> ==rob<br>
><br>
> --<br>
> Rob Latham<br>
> Mathematics and Computer Science Division<br>
> Argonne National Lab, IL USA<br>
><br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div><br>