Hi,<br><br>Sorry for missing the "reply all" sometimes. As I said to Jim, we are really talking about source mistakes then.<br>I was wondering if some other parameters (like position of the process on the nodes) could change the way each Mpi process will work.<br>
In my case there is only 1 process-depending size, which is the array size, and after printing it is everywhere the same.<br><br>The reasons why I think it comes from somewhere else are:<br>-Smaller cases work fine<br>-The error message appears "randomly". i.e. if I write 64 files (on 4096 cores, means that 64 process write on the same file), and putting some Barrier between each file writing, it will randomly sometimes crash on the 3rd, 6th or n..th file. <br>
<br>That's why I am looking for some other reasons I can't expect, even if I know that the problem might probably comes from my source code.<br><br>Thanks again for helping me.<br><br>Julien<br><br><div class="gmail_quote">
2009/6/15 Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@ece.northwestern.edu">wkliao@ece.northwestern.edu</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
For example, when defining a new 2D array variable and the number of processes is 2, P0 and P1.<br>
The metadata (array dimensions, attributes, etc. in define mode) must be the same between P0<br>
and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this error message will<br>
appear.<br><font color="#888888">
<br>
Wei-keng</font><div><div></div><div class="h5"><br>
<br>
On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks everybody for your help.<br>
<br>
I am afraid I don't get the point "your code is defining netcdf variables and attributes in<br>
a slightly different way on some MPI processes than others"... depending on what?<br>
<br>
Another test I could try is to unable the check made by ncpmi_enddef if it is possible, and see which kind of output file I get.<br>
I don't know if it is possible to do it easily without recompiling the library.<br>
<br>
I will try anyway the binary debugging.<br>
<br>
<br>
2009/6/15 Rob Latham <<a href="mailto:robl@mcs.anl.gov" target="_blank">robl@mcs.anl.gov</a>><br>
On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:<br>
> While it does not create any problems on small cases, bigger cases stop at<br>
> the ncmpi_enddef call on some files (randomly, even with synchronisation in<br>
> between), saying that there is a mismatch between dimensions. After many<br>
> check it does not seems that there is something wrong with the dimensions. I<br>
> have no idea of how to solve the problem. Did anyone had similar problem?<br>
> Thanks for your help.<br>
<br>
Hi Julien. Wei-keng is right: I know you've checked carefully, but<br>
some part of your code is defining netcdf variables and attributes in<br>
a slightly different way on some MPI processes than others.<br>
<br>
The main way people debug this is through binary search: comment out<br>
half of the define-mode portion; if the problem persists, comment out<br>
half of the remainder, else, try with the other half.<br>
<br>
You're not the first to encounter this problem. Maybe this could be a<br>
warning and not an error, and maybe we should just have the define<br>
mode view as rank 0 sees it be the one that wins if there's a<br>
discrepancy. I don't know how many people (if any) rely on the<br>
current behavior to find problems.<br>
<br>
==rob<br>
<br>
--<br>
Rob Latham<br>
Mathematics and Computer Science Division<br>
Argonne National Lab, IL USA<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div><br>