Problem on Blue Gene/P

[Rob Ross]

Hi Julien,

Huh. I wouldn't have expected that. Perhaps we should adjust the  
PnetCDF semantics for the *values* such that you get a value from some  
process, not defined which one if they are different?

Rob

On Jun 16, 2009, at 10:27 AM, Julien Bodart wrote:

> Hi Rob,
>
> This was in the value actually:
>
> current_time = time(NULL) ;
> GSTRING_ASS(my_string, ctime(&current_time) ) ;
> status =ncmpi_put_att_text ( nc_id, NC_GLOBAL, "time_stamp"      ,  
> (MPI_Offset) (my_string->len-1) , my_string->str ) ;
>
> GSTRING_ASS being a macro that return a string, where a string being  
> a char(str) and the number of element in the char (len)
>
> Regards,
>
> Julien
>
> 2009/6/16 Rob Ross <rross at mcs.anl.gov>
> Hi Julien,
>
> So you were using the time in the name of the attribute, not in the  
> value?
>
> Thanks,
>
> Rob
>
>
> On Jun 16, 2009, at 7:22 AM, Julien Bodart wrote:
>
> Hi everybody,
>
> I finally manage to remove this bug, which was of course coming from  
> my source code!
> The guilty: a "time" global attribute coming from the "ctime"  
> function which of course is different across a large number of  
> processors... I know this is silly but actually I was not expecting  
> a check on the global attributes, especially with an error message  
> "NC definitions mismatch".
> So I have to apologize for such a stupid mistake, but at the same  
> time, it reinforces the idea to rethink this check function.
> Thanks again.
>
> Julien
>
> 2009/6/16 Wei-keng Liao <wkliao at ece.northwestern.edu>
>
>
> I agree this suggestion. So, there are three options.
>
> 1. When pnetcdf is built with debug mode (at configure time), we  
> will enable
>  the consistency checking across all processes. In this case, the  
> error
>  is considered fatal. For debugging purpose, this should be fine.  
> Note that
>  the consistency checking may become costly, especially when the  
> number of
>  processes is large. We do not expect a production pnetcdf to be  
> built with
>  the debug mode.
>
> 2. When built without debug mode, pnetcdf will only take process 0's  
> inputs and
>  ignore all others. Also, consistency checking is disabled.
>
> 3. A middle ground: enable consistency checking but only process 0's  
> inputs
>  are used to define variables, attributes, etc. if inconsistency is  
> detected.
>  The error is not fatal, but only gives a warning message.
>
> If these are fine to the pnetcdf community, we will start to  
> implement them.
>
> Wei-keng
>
>
> On Jun 15, 2009, at 8:21 PM, Yu-hengTseng wrote:
>
> Thanks Rob and Wei-keng,
> It will be a good idea to make this check as a warming only. In most  
> realistic applications (including Community Atmospheric Model,  
> within CCSM development), it's almost impossible to have the same  
> dimension for the same array variable within different processes.  
> Usually, 1 or 2 shift. Cheers,
> Yu-heng Tseng
> Department of Atmospheric Sciences
> National Taiwan University
> No. 1, Sec. 4, Roosevelt Rd, Taipei 106, Taiwan
> tel: 886-2-33663918
> email: yhtseng at as.ntu.edu.tw
>
> ----- Original Message -----
> From: wkliao at ece.northwestern.edu
> To: wkliao at ece.northwestern.edu;parallel-netcdf<parallel-netcdf at lists.mcs.anl.gov 
> >
> Sent: 2009-06-16 03:00:01
> Subject: Re: Problem on Blue Gene/P
>
> For example, when defining a new 2D array variable and the number of
> processes is 2, P0 and P1.
> The metadata (array dimensions, attributes, etc. in define mode) must
> be the same between P0
> and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this
> error message will
> appear.
>
> Wei-keng
>
> On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:
>
> > Thanks everybody for your help.
> >
> > I am afraid I don't get the point "your code is defining netcdf
> > variables and attributes in
> > a slightly different way on some MPI processes than others"...
> > depending on what?
> >
> > Another test I could try is to unable the check made by ncpmi_enddef
> > if it is possible, and see which kind of output file I get.
> > I don't know if it is possible to do it easily without recompiling
> > the library.
> >
> > I will try anyway the binary debugging.
> >
> >
> > 2009/6/15 Rob Latham
> > On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:
> > > While it does not create any problems on small cases, bigger cases
> > stop at
> > > the ncmpi_enddef call on some files (randomly, even with
> > synchronisation in
> > > between), saying that there is a mismatch between dimensions.
> > After many
> > > check it does not seems that there is something wrong with the
> > dimensions. I
> > > have no idea of how to solve the problem. Did anyone had similar
> > problem?
> > > Thanks for your help.
> >
> > Hi Julien. Wei-keng is right: I know you've checked carefully, but
> > some part of your code is defining netcdf variables and attributes  
> in
> > a slightly different way on some MPI processes than others.
> >
> > The main way people debug this is through binary search: comment out
> > half of the define-mode portion; if the problem persists, comment  
> out
> > half of the remainder, else, try with the other half.
> >
> > You're not the first to encounter this problem.  Maybe this could  
> be a
> > warning and not an error, and maybe we should just have the define
> > mode view as rank 0 sees it be the one that wins if there's a
> > discrepancy.   I don't know how many people (if any) rely on the
> > current behavior to find problems.
> >
> > ==rob
> >
> > --
> > Rob Latham
> > Mathematics and Computer Science Division
> > Argonne National Lab, IL USA
> >
>
>
>
>
>
>
>
>