DAcreate2d process layout order

[Sean Dettrick]

Barry Smith wrote:

>
>
>
>
> On Thu, 18 May 2006, Sean Dettrick wrote:
>
>> Hi Barry,
>> the order is determined by MPI_Cart_create.
>
>
>    Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
> fastest and then the first (x-axis)? 

I am not sure that my interpretation is correct.  I'll send you a small 
demo, so you can see for yourself if you agree with it.
Sean

> Hmmm, maybe we should change the
> DA? Changing it once and for all (not supporting both) is probably
> not a big deal and shouldn't break much (I hope).
> Bill, what do you think?
>
>    Barry
>
>>  I've tried to swap x and y in the DA to make the problem go away but 
>> haven't succeeded yet.  Not enough brain cells I suspect ... still 
>> thinking about it.
>> Thanks for the response,
>> Sean
>>
>> Barry Smith wrote:
>>
>>>
>>>
>>>   Sean,
>>>
>>>    I don't completely understand what goes wrong. Is it because YOUR
>>> application orders the processors related to geometry in the 
>>> following way?
>>>
>>>     ^ y direction
>>>     |
>>>        2   5  8
>>>        1   4  7
>>>        0   3  6
>>>
>>>                     -> x direction
>>>
>>> Or is this something inherent in MPI_Cart_create?
>>>
>>> PETSc does it so
>>>
>>>     ^ y direction
>>>     |
>>>        6   7  8
>>>        3   4  5
>>>        0   1  2
>>>
>>>                     -> x direction
>>>
>>>   If you want to "force" the PETSc DA to match the first case you 
>>> could just
>>> "lie" to PETSc and treat the x direction as the y direction and the 
>>> x as the y.
>>> Thus you would flip the i and j indices. You could do the same trick in
>>> 3d.
>>>
>>> There is no easy way to change the DA to do the ordering as the first
>>> case above or support both approaches (possible? yes, but ugly
>>> duplicate code to handle the two cases).
>>>
>>> I've cc:ed to Bill since he would know more about the details of 
>>> MPI_cart_create().
>>>
>>>    Barry
>>>
>>> On Mon, 15 May 2006, Sean Dettrick wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to use DACreate2d and KSP in my existing MPI 
>>>> application.  I already have a Cartesian communicator established, 
>>>> and I set PETSC_COMM_WORLD equal to it and then call PetscInitialize.
>>>>
>>>> This works fine on a prime number of CPUs, because there is only 
>>>> one possible ordered MPI layout in one dimension.  But with a 
>>>> non-prime number there are two possible ordered layouts and it just 
>>>> happens that my 2D CPU layout (determined by MPI_Cart_create) is 
>>>> the transpose of the PETSc 2D CPU layout. Is there a way to 
>>>> organize the DA layout more explicitly than with DACreate2d?   Or 
>>>> to tell PETSc to transpose its CPU order?  I also wonder about the 
>>>> 3D case.
>>>>
>>>> thanks
>>>> Sean
>>>>
>>>>
>>>
>>>
>>
>>
>
>