DAcreate2d process layout order

[Sean Dettrick]

Barry,

just to follow up, I created a cartesian communicator workaround, 
without changing the order of x and y in the DA calls.

Instead I permuted my periodicity orders, created the 2D cartesian 
communicator, and then created with MPI_Cart_sub an array of 1D 
cartesian communicators, one for each dimension, with the dimension 
orders permuted.

Now I can use, say, Cartcomm1D[0] to send in the 0th (x) direction of 
the DA, with the periodicity matching DA_XPERIODIC.

I still can't use the 2D communicator - it still has confused dimension 
order and periodicity. But the 1D communicators are a good compromise.

Thanks for your help,
Sean

Barry Smith wrote:

>
> Ok, so it is laid-out like a matrix, not like x, y coordinates.
>
> Will put the change to DA on the list of things to do.
>
>
> Barry
>
> On Thu, 18 May 2006, Sean Dettrick wrote:
>
>> Barry Smith wrote:
>>
>>>
>>> Bill, does the MPI standard dictate this decomposition or
>>> could different implementations do it the opposite way?
>>> Then we'd have to make the DA logic a bit more complicated.
>>
>>
>> I don't have a copy of the standard, but to quote page 255 of "MPI, 
>> the complete reference" by Snir et al:
>> "Row-major numbering is always used for the processes in a Cartesian 
>> structure".
>> Their diagram in figure 6.1 matches my code output for coords 
>> couplets (i,j):
>>
>> 0 1 2 3
>> (0,0) (0,1) (0,2) (0,3)
>>
>> 4 5 6 7
>> (1,0) (1,1) (1,2) (1,3)
>>
>> 8 9 10 11
>> (2,0) (2,1) (2,2) (2,3)
>>
>>
>> By the way I agree with you, I *should* be able to swap the x and y 
>> myself. Just haven't had much luck yet in that regard.
>>
>> Sean
>>
>>>
>>>
>>> On Thu, 18 May 2006, Sean Dettrick wrote:
>>>
>>>> Barry Smith wrote:
>>>>
>>>>> On Thu, 18 May 2006, Sean Dettrick wrote:
>>>>>
>>>>>> Hi Barry,
>>>>>> the order is determined by MPI_Cart_create.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
>>>>> fastest and then the first (x-axis)? Hmmm, maybe we should change the
>>>>> DA? Changing it once and for all (not supporting both) is probably
>>>>> not a big deal and shouldn't break much (I hope).
>>>>
>>>>
>>>>
>>>> Hi Barry,
>>>>
>>>> it depends, what do you call x and what do you call y?
>>>> MPI_Cart_coords returns a vector, coords - I tend to say x is 
>>>> coords[0], y is coords[1] and z is coords[2]. For what it's worth, 
>>>> there's a short code appended to this email, which produces:
>>>>
>>>> rank = 0 has Cartesian coords = { 0, 0 }
>>>> rank = 1 has Cartesian coords = { 0, 1 }
>>>> rank = 2 has Cartesian coords = { 1, 0 }
>>>> rank = 3 has Cartesian coords = { 1, 1 }
>>>> rank = 0 has DA range x=[0,50) and y=[0,50)
>>>> rank = 1 has DA range x=[50,100) and y=[0,50)
>>>> rank = 2 has DA range x=[0,50) and y=[50,100)
>>>> rank = 3 has DA range x=[50,100) and y=[50,100)
>>>>
>>>>>>> I don't completely understand what goes wrong. Is it because YOUR
>>>>>>> application orders the processors related to geometry in the 
>>>>>>> following way?
>>>>>>>
>>>>>>> ^ y direction
>>>>>>> |
>>>>>>> 2 5 8
>>>>>>> 1 4 7
>>>>>>> 0 3 6
>>>>>>>
>>>>>>> -> x direction
>>>>>>>
>>>>>>> Or is this something inherent in MPI_Cart_create?
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> For my interpretation of x and y, MPI_Cart_create produces the 
>>>> above layout. But if I said x=coords[1] and y=coords[0], then it 
>>>> would match the one below.
>>>>
>>>>>>>
>>>>>>> PETSc does it so
>>>>>>>
>>>>>>> ^ y direction
>>>>>>> |
>>>>>>> 6 7 8
>>>>>>> 3 4 5
>>>>>>> 0 1 2
>>>>>>>
>>>>>>> -> x direction
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> Code and makefile attached ... hopefully within the message size 
>>>> limit.
>>>> Just make cartcommtest.
>>>>
>>>> Sean
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>