Problem on Blue Gene/P

Tue Jun 16 11:17:51 CDT 2009

Hi Rob

For the global attributes at least, I think nothing is important but the
root process, for the name, and even the value (I might be wrong on this
point, but in my case it is useless for everybody to know it).
About dimensions and variables, I would prefer to keep a check, even if I
don't really see why process n should know the size of the array he is
working on...

Julien

2009/6/16 Rob Ross <rross at mcs.anl.gov>

> Hi Jim,
>
> Yeah, I agree that (given that we're bcast()ing the header around to check
> it anyway) there isn't much advantage to what we're doing now.
>
> So what would the API look like? Would we make all the define mode calls
> independent and only valid on rank 0 of the communicator used to open the
> dataset? Do we make them collective but ignore values from all other
> processes?
>
> Rob
>
>
> On Jun 16, 2009, at 10:46 AM, Jim Edwards wrote:
>
>  Seems like it wouldn't be any harder to implement and would be more user
>> friendly if you use the value from the root process and ignore the others.
>> In my application there are several attributes which are set on the root
>> task and which are not needed on the other tasks except to meet the
>> requirement of pnetcdf, if we remove this requirement I could get rid of
>> several otherwise unnecessary mpi_bcast calls.
>>
>>
>>
>> On Tue, Jun 16, 2009 at 9:38 AM, Rob Ross <rross at mcs.anl.gov> wrote:
>> Hi Julien,
>>
>> Huh. I wouldn't have expected that. Perhaps we should adjust the PnetCDF
>> semantics for the *values* such that you get a value from some process, not
>> defined which one if they are different?
>>
>> Rob
>>
>>
>> On Jun 16, 2009, at 10:27 AM, Julien Bodart wrote:
>>
>> Hi Rob,
>>
>> This was in the value actually:
>>
>> current_time = time(NULL) ;
>> GSTRING_ASS(my_string, ctime(&current_time) ) ;
>> status =ncmpi_put_att_text ( nc_id, NC_GLOBAL, "time_stamp"      ,
>> (MPI_Offset) (my_string->len-1) , my_string->str ) ;
>>
>> GSTRING_ASS being a macro that return a string, where a string being a
>> char(str) and the number of element in the char (len)
>>
>> Regards,
>>
>> Julien
>>
>> 2009/6/16 Rob Ross <rross at mcs.anl.gov>
>> Hi Julien,
>>
>> So you were using the time in the name of the attribute, not in the value?
>>
>> Thanks,
>>
>> Rob
>>
>>
>> On Jun 16, 2009, at 7:22 AM, Julien Bodart wrote:
>>
>> Hi everybody,
>>
>> I finally manage to remove this bug, which was of course coming from my
>> source code!
>> The guilty: a "time" global attribute coming from the "ctime" function
>> which of course is different across a large number of processors... I know
>> this is silly but actually I was not expecting a check on the global
>> attributes, especially with an error message "NC definitions mismatch".
>> So I have to apologize for such a stupid mistake, but at the same time, it
>> reinforces the idea to rethink this check function.
>> Thanks again.
>>
>> Julien
>>
>> 2009/6/16 Wei-keng Liao <wkliao at ece.northwestern.edu>
>>
>>
>> I agree this suggestion. So, there are three options.
>>
>> 1. When pnetcdf is built with debug mode (at configure time), we will
>> enable
>>  the consistency checking across all processes. In this case, the error
>>  is considered fatal. For debugging purpose, this should be fine. Note
>> that
>>  the consistency checking may become costly, especially when the number of
>>  processes is large. We do not expect a production pnetcdf to be built
>> with
>>  the debug mode.
>>
>> 2. When built without debug mode, pnetcdf will only take process 0's
>> inputs and
>>  ignore all others. Also, consistency checking is disabled.
>>
>> 3. A middle ground: enable consistency checking but only process 0's
>> inputs
>>  are used to define variables, attributes, etc. if inconsistency is
>> detected.
>>  The error is not fatal, but only gives a warning message.
>>
>> If these are fine to the pnetcdf community, we will start to implement
>> them.
>>
>> Wei-keng
>>
>>
>> On Jun 15, 2009, at 8:21 PM, Yu-hengTseng wrote:
>>
>> Thanks Rob and Wei-keng,
>> It will be a good idea to make this check as a warming only. In most
>> realistic applications (including Community Atmospheric Model, within CCSM
>> development), it's almost impossible to have the same dimension for the same
>> array variable within different processes. Usually, 1 or 2 shift. Cheers,
>> Yu-heng Tseng
>> Department of Atmospheric Sciences
>> National Taiwan University
>> No. 1, Sec. 4, Roosevelt Rd, Taipei 106, Taiwan
>> tel: 886-2-33663918
>> email: yhtseng at as.ntu.edu.tw
>>
>> ----- Original Message -----
>> From: wkliao at ece.northwestern.edu
>> To: wkliao at ece.northwestern.edu;parallel-netcdf<
>> parallel-netcdf at lists.mcs.anl.gov>
>> Sent: 2009-06-16 03:00:01
>> Subject: Re: Problem on Blue Gene/P
>>
>> For example, when defining a new 2D array variable and the number of
>> processes is 2, P0 and P1.
>> The metadata (array dimensions, attributes, etc. in define mode) must
>> be the same between P0
>> and P1. If P0 uses 10x10 dimension values and P1 uses 10x11, then this
>> error message will
>> appear.
>>
>> Wei-keng
>>
>> On Jun 15, 2009, at 12:57 PM, Julien Bodart wrote:
>>
>> > Thanks everybody for your help.
>> >
>> > I am afraid I don't get the point "your code is defining netcdf
>> > variables and attributes in
>> > a slightly different way on some MPI processes than others"...
>> > depending on what?
>> >
>> > Another test I could try is to unable the check made by ncpmi_enddef
>> > if it is possible, and see which kind of output file I get.
>> > I don't know if it is possible to do it easily without recompiling
>> > the library.
>> >
>> > I will try anyway the binary debugging.
>> >
>> >
>> > 2009/6/15 Rob Latham
>> > On Fri, Jun 12, 2009 at 02:19:33PM +0200, Julien Bodart wrote:
>> > > While it does not create any problems on small cases, bigger cases
>> > stop at
>> > > the ncmpi_enddef call on some files (randomly, even with
>> > synchronisation in
>> > > between), saying that there is a mismatch between dimensions.
>> > After many
>> > > check it does not seems that there is something wrong with the
>> > dimensions. I
>> > > have no idea of how to solve the problem. Did anyone had similar
>> > problem?
>> > > Thanks for your help.
>> >
>> > Hi Julien. Wei-keng is right: I know you've checked carefully, but
>> > some part of your code is defining netcdf variables and attributes in
>> > a slightly different way on some MPI processes than others.
>> >
>> > The main way people debug this is through binary search: comment out
>> > half of the define-mode portion; if the problem persists, comment out
>> > half of the remainder, else, try with the other half.
>> >
>> > You're not the first to encounter this problem.  Maybe this could be a
>> > warning and not an error, and maybe we should just have the define
>> > mode view as rank 0 sees it be the one that wins if there's a
>> > discrepancy.   I don't know how many people (if any) rely on the
>> > current behavior to find problems.
>> >
>> > ==rob
>> >
>> > --
>> > Rob Latham
>> > Mathematics and Computer Science Division
>> > Argonne National Lab, IL USA
>> >
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20090616/4b3dfc84/attachment.htm>