Problem on Blue Gene/P

[Rob Ross]

We'll need to do some initial communication to ensure that all  
processes set the value the same way... -- Rob

On Jun 16, 2009, at 7:49 AM, Jim Edwards wrote:

> Can we make this controllable at runtime possibly through an  
> environment variable so that one can toggle between modes without  
> recompiling?
>
> Thanks,
> Jim
>
> On Mon, Jun 15, 2009 at 9:53 PM, Wei-keng Liao <wkliao at ece.northwestern.edu 
> > wrote:
>
>
> I agree this suggestion. So, there are three options.
>
> 1. When pnetcdf is built with debug mode (at configure time), we  
> will enable
>   the consistency checking across all processes. In this case, the  
> error
>   is considered fatal. For debugging purpose, this should be fine.  
> Note that
>   the consistency checking may become costly, especially when the  
> number of
>   processes is large. We do not expect a production pnetcdf to be  
> built with
>   the debug mode.
>
> 2. When built without debug mode, pnetcdf will only take process 0's  
> inputs and
>   ignore all others. Also, consistency checking is disabled.
>
> 3. A middle ground: enable consistency checking but only process 0's  
> inputs
>   are used to define variables, attributes, etc. if inconsistency is  
> detected.
>   The error is not fatal, but only gives a warning message.
>
> If these are fine to the pnetcdf community, we will start to  
> implement them.
>
> Wei-keng
>
>
> On Jun 15, 2009, at 8:21 PM, Yu-hengTseng wrote:
>
> Thanks Rob and Wei-keng,
> It will be a good idea to make this check as a warming only. In most  
> realistic applications (including Community Atmospheric Model,  
> within CCSM development), it's almost impossible to have the same  
> dimension for the same array variable within different processes.  
> Usually, 1 or 2 shift. Cheers,
> Yu-heng Tseng
> Department of Atmospheric Sciences
> National Taiwan University
> No. 1, Sec. 4, Roosevelt Rd, Taipei 106, Taiwan
> tel: 886-2-33663918
> email: yhtseng at as.ntu.edu.tw
>
> ----- Original Message -----
> From: wkliao at ece.northwestern.edu
> To: wkliao at ece.northwestern.edu;parallel-netcdf<parallel-netcdf at lists.mcs.anl.gov 
> >
> Sent: 2009-06-16 03:00:01
> Subject: Re: Problem on Blue Gene/P
>
> For example, when defining a new 2D array variable and the number of
> processes is 2, P0 and P1.
> The metadata (array dimensions, attributes, etc. in define mode) must
> be the same between P0
> and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this
> error message will
> appear.
>
> Wei-keng
>
> On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:
>
> > Thanks everybody for your help.
> >
> > I am afraid I don't get the point "your code is defining netcdf
> > variables and attributes in
> > a slightly different way on some MPI processes than others"...
> > depending on what?
> >
> > Another test I could try is to unable the check made by ncpmi_enddef
> > if it is possible, and see which kind of output file I get.
> > I don't know if it is possible to do it easily without recompiling
> > the library.
> >
> > I will try anyway the binary debugging.
> >
> >
> > 2009/6/15 Rob Latham
> > On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:
> > > While it does not create any problems on small cases, bigger cases
> > stop at
> > > the ncmpi_enddef call on some files (randomly, even with
> > synchronisation in
> > > between), saying that there is a mismatch between dimensions.
> > After many
> > > check it does not seems that there is something wrong with the
> > dimensions. I
> > > have no idea of how to solve the problem. Did anyone had similar
> > problem?
> > > Thanks for your help.
> >
> > Hi Julien. Wei-keng is right: I know you've checked carefully, but
> > some part of your code is defining netcdf variables and attributes  
> in
> > a slightly different way on some MPI processes than others.
> >
> > The main way people debug this is through binary search: comment out
> > half of the define-mode portion; if the problem persists, comment  
> out
> > half of the remainder, else, try with the other half.
> >
> > You're not the first to encounter this problem.  Maybe this could  
> be a
> > warning and not an error, and maybe we should just have the define
> > mode view as rank 0 sees it be the one that wins if there's a
> > discrepancy.   I don't know how many people (if any) rely on the
> > current behavior to find problems.
> >
> > ==rob
> >
> > --
> > Rob Latham
> > Mathematics and Computer Science Division
> > Argonne National Lab, IL USA
> >
>
>
>
>
>
>