DAcreate2d process layout order

[Barry Smith]

On Thu, 18 May 2006, Sean Dettrick wrote:

> Hi Barry,
> the order is determined by MPI_Cart_create.

    Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
fastest and then the first (x-axis)? Hmmm, maybe we should change the
DA? Changing it once and for all (not supporting both) is probably
not a big deal and shouldn't break much (I hope).
Bill, what do you think?

    Barry

>  I've tried to swap x and y in 
> the DA to make the problem go away but haven't succeeded yet.  Not enough 
> brain cells I suspect ... still thinking about it.
> Thanks for the response,
> Sean
>
> Barry Smith wrote:
>
>> 
>>
>>   Sean,
>>
>>    I don't completely understand what goes wrong. Is it because YOUR
>> application orders the processors related to geometry in the following way?
>>
>>     ^ y direction
>>     |
>>        2   5  8
>>        1   4  7
>>        0   3  6
>>
>>                     -> x direction
>> 
>> Or is this something inherent in MPI_Cart_create?
>> 
>> PETSc does it so
>>
>>     ^ y direction
>>     |
>>        6   7  8
>>        3   4  5
>>        0   1  2
>>
>>                     -> x direction
>>
>>   If you want to "force" the PETSc DA to match the first case you could 
>> just
>> "lie" to PETSc and treat the x direction as the y direction and the x as 
>> the y.
>> Thus you would flip the i and j indices. You could do the same trick in
>> 3d.
>> 
>> There is no easy way to change the DA to do the ordering as the first
>> case above or support both approaches (possible? yes, but ugly
>> duplicate code to handle the two cases).
>> 
>> I've cc:ed to Bill since he would know more about the details of 
>> MPI_cart_create().
>>
>>    Barry
>> 
>> On Mon, 15 May 2006, Sean Dettrick wrote:
>> 
>>> Hi,
>>> 
>>> I'm trying to use DACreate2d and KSP in my existing MPI application.  I 
>>> already have a Cartesian communicator established, and I set 
>>> PETSC_COMM_WORLD equal to it and then call PetscInitialize.
>>> 
>>> This works fine on a prime number of CPUs, because there is only one 
>>> possible ordered MPI layout in one dimension.  But with a non-prime number 
>>> there are two possible ordered layouts and it just happens that my 2D CPU 
>>> layout (determined by MPI_Cart_create) is the transpose of the PETSc 2D 
>>> CPU layout. Is there a way to organize the DA layout more explicitly than 
>>> with DACreate2d?   Or to tell PETSc to transpose its CPU order?  I also 
>>> wonder about the 3D case.
>>> 
>>> thanks
>>> Sean
>>> 
>>> 
>> 
>> 
>
>