Problem on Blue Gene/P

[Julien Bodart]

Hi,

Sorry for missing the "reply all" sometimes. As I said to Jim, we are really
talking about source mistakes then.
I was wondering if some other parameters (like position of the process on
the nodes) could change the way each Mpi process will work.
In my case there is only 1 process-depending size, which is the array size,
and after printing it is everywhere the same.

The reasons why I think it comes from somewhere else are:
-Smaller cases work fine
-The error message appears "randomly". i.e. if I write 64 files (on 4096
cores, means that 64 process write on the same file), and putting some
Barrier between each file writing, it will randomly sometimes crash on the
3rd, 6th or n..th file.

That's why I am looking for some other reasons I can't expect, even if I
know that the problem might probably comes from my source code.

Thanks again for helping me.

Julien

2009/6/15 Wei-keng Liao <wkliao at ece.northwestern.edu>

>
> For example, when defining a new 2D array variable and the number of
> processes is 2, P0 and P1.
> The metadata (array dimensions, attributes, etc. in define mode) must be
> the same between P0
> and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this
> error message will
> appear.
>
> Wei-keng
>
>
> On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:
>
>  Thanks everybody for your help.
>>
>> I am afraid I don't get the point "your code is defining netcdf variables
>> and attributes in
>> a slightly different way on some MPI processes than others"... depending
>> on what?
>>
>> Another test I could try is to unable the check made by ncpmi_enddef if it
>> is possible, and see which kind of output file I get.
>> I don't know if it is possible to do it easily without recompiling the
>> library.
>>
>> I will try anyway the binary debugging.
>>
>>
>> 2009/6/15 Rob Latham <robl at mcs.anl.gov>
>> On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:
>> > While it does not create any problems on small cases, bigger cases stop
>> at
>> > the ncmpi_enddef call on some files (randomly, even with synchronisation
>> in
>> > between), saying that there is a mismatch between dimensions. After many
>> > check it does not seems that there is something wrong with the
>> dimensions. I
>> > have no idea of how to solve the problem. Did anyone had similar
>> problem?
>> > Thanks for your help.
>>
>> Hi Julien. Wei-keng is right: I know you've checked carefully, but
>> some part of your code is defining netcdf variables and attributes in
>> a slightly different way on some MPI processes than others.
>>
>> The main way people debug this is through binary search: comment out
>> half of the define-mode portion; if the problem persists, comment out
>> half of the remainder, else, try with the other half.
>>
>> You're not the first to encounter this problem.  Maybe this could be a
>> warning and not an error, and maybe we should just have the define
>> mode view as rank 0 sees it be the one that wins if there's a
>> discrepancy.   I don't know how many people (if any) rely on the
>> current behavior to find problems.
>>
>> ==rob
>>
>> --
>> Rob Latham
>> Mathematics and Computer Science Division
>> Argonne National Lab, IL USA
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20090615/c4316999/attachment.htm>