process id in DA

[Sean Dettrick]

On Nov 12, 2008, at 4:24 PM, Shao-Ching Huang wrote:

> Hi,
>
> Is there a PETSc call to obtain the "process coordinates" in a process
> grid, partitioned by a DA? (Something like what MPI_Cart_coords does.)
>
> Thanks,
>
> Shao-Ching
>

A while ago (May 2006) I found that the PETSc DA has its processes  
ordered in the TRANSPOSE way to that of MPI Cartesian communicators,  
so you might have to consider mapping back and forth between the two  
cartesian representations.

Barry said that he might change the DA so that the two agree with each  
other, but I don't know if he ever did...

Does anybody know the status of that?

In case it helps, below is part of our exchange on the matter.

Sean
---------------------------



Ok, so it is laid-out like a matrix, not like x, y coordinates.

Will put the change to DA on the list of things to do.


   Barry

On Thu, 18 May 2006, Sean Dettrick wrote:

> Barry Smith wrote:
>
>> Bill, does the MPI standard dictate this decomposition or
>> could different implementations do it the opposite way?
>> Then we'd have to make the DA logic a bit more complicated.
>
> I don't have a copy of the standard, but to quote page 255 of "MPI,  
> the complete reference" by Snir et al:
> "Row-major numbering is always used for the processes in a Cartesian  
> structure".
> Their diagram in figure 6.1 matches my code output for coords  
> couplets (i,j):
>
> 0      1       2       3
> (0,0)  (0,1)   (0,2)   (0,3)
>
> 4      5       6       7
> (1,0)  (1,1)   (1,2)   (1,3)
>
> 8      9      10      11
> (2,0)  (2,1)   (2,2)   (2,3)
>
>
> By the way I agree with you, I *should* be able to swap the x and y  
> myself. Just haven't had much luck yet in that regard.
>
> Sean
>
>> On Thu, 18 May 2006, Sean Dettrick wrote:
>>> Barry Smith wrote:
>>>> On Thu, 18 May 2006, Sean Dettrick wrote:
>>>>> Hi Barry,
>>>>> the order is determined by MPI_Cart_create.
>>>>
>>>>   Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
>>>> fastest and then the first (x-axis)? Hmmm, maybe we should change  
>>>> the
>>>> DA? Changing it once and for all (not supporting both) is probably
>>>> not a big deal and shouldn't break much (I hope).
>>> Hi Barry,
>>> it depends, what do you call x and what do you call y?
>>> MPI_Cart_coords returns a vector, coords - I tend to say x is  
>>> coords[0], y is coords[1] and z is coords[2]. For what it's worth,  
>>> there's a short code appended to this email, which produces:
>>> rank = 0 has Cartesian coords = { 0, 0 }
>>> rank = 1 has Cartesian coords = { 0, 1 }
>>> rank = 2 has Cartesian coords = { 1, 0 }
>>> rank = 3 has Cartesian coords = { 1, 1 }
>>> rank = 0 has DA range x=[0,50) and y=[0,50)
>>> rank = 1 has DA range x=[50,100) and y=[0,50)
>>> rank = 2 has DA range x=[0,50) and y=[50,100)
>>> rank = 3 has DA range x=[50,100) and y=[50,100)
>>>
>>>>>>   I don't completely understand what goes wrong. Is it because  
>>>>>> YOUR
>>>>>> application orders the processors related to geometry in the  
>>>>>> following way?
>>>>>>
>>>>>>    ^ y direction
>>>>>>    |
>>>>>>       2   5  8
>>>>>>       1   4  7
>>>>>>       0   3  6
>>>>>>
>>>>>>                    -> x direction
>>>>>> Or is this something inherent in MPI_Cart_create?
>>> For my interpretation of x and y, MPI_Cart_create produces the  
>>> above layout. But if I said x=coords[1] and y=coords[0], then it  
>>> would match the one below.
>>>>>> PETSc does it so
>>>>>>
>>>>>>    ^ y direction
>>>>>>    |
>>>>>>       6   7  8
>>>>>>       3   4  5
>>>>>>       0   1  2
>>>>>>
>>>>>>                    -> x direction
>>> Code and makefile attached ... hopefully within the message size  
>>> limit.
>>> Just make cartcommtest.
>>> Sean
>
>