MPI-layout of PETSc

[Barry Smith]

   Use MPI_Comm_split() with the same color for all processors, then  
use the second integer argument to indicate the new rank you want for  
the process.
Choice the new rank so its x,y coordinate in the logical grid will  
match the y,x coordinate in the cartesian grid.

    Barry

On Jul 3, 2009, at 12:09 PM, Rolf Kuiper wrote:

> Hi Barry,
>
> I tried that already with:
> First way by copying:
> MPI_Comm_dup(PETSC_COMM_WORLD, &MyComm);
>
> Second way by creating:
> int dims[3] = {0,0,0};
> int ndims=3;
> MPI_Dims_create(NumberOfProcessors, ndims, dims);
> int false = 0; int true = 1;
> int periods[3] = { false, false, true };
> int reorder = true;
> MPI_Comm MyComm;
> MPI_Cart_create(PETSC_COMM_WORLD, ndims, dims, periods, reorder,  
> &MyComm);
>
> in the end then:
> PETSC_COMM_WORLD = MyComm;
>
> I test the MyComm with MPI_Topo_test(); and it is cartesian, yes.
> I can the coordinates of the cpus with MPI_Cart_coords(MyComm,  
> LocalRank, ndims, coords); , but I found no way to set/rearrange  
> these coordinates.
>
> Do you can help me in that case or have I to ask a MPI-support?
>
> Thanks for all,
> Rolf
>
>
> Am 03.07.2009 um 17:56 schrieb Barry Smith:
>>
>>   In designing the PETSc DA I did not (by ignorance) follow the  
>> layout approach of the MPI cartesian MPI_Cart_create (that gives  
>> the first local cpus first in the y-direction).
>> I had it put the first cpus in the x-direction.
>>
>>   What you need to do is create a new communicator that changes the  
>> order of the processors so that when used by the PETSc DA they lie  
>> out in the ordering that matches the other code. You will need to  
>> read up on the MPI_Cart stuff.
>>
>>   To change PETSC_COMM_WORLD you simply set PETSC_COMM_WORLD =  
>> yournewcom BEFORE calling PetscInitialize().
>>
>>   Barry
>>
>> On Jul 3, 2009, at 3:52 AM, Rolf Kuiper wrote:
>>
>>> Hi,
>>>
>>> Am 30.06.2009 um 02:24 schrieb Barry Smith:
>>>>
>>>> On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote:
>>>>
>>>>> Hi PETSc users,
>>>>>
>>>>> I ran into trouble in combining my developed PETSc application  
>>>>> with another code (based on another library called "ArrayLib").
>>>>> The problem is the parallel layout for MPI, e.g. in 2D with 6  
>>>>> cpus the ArrayLib code gives the names/ranks of the local cpus  
>>>>> first in y-direction, than in x (from last to first, in the same  
>>>>> way the MPI arrays are called, like 3Darray[z][y][x]):
>>>>>
>>>>> y
>>>>> ^
>>>>> | 2-4-6
>>>>> | 1-3-5
>>>>> |--------> x
>>>>>
>>>>> If I call DACreate() from PETSc, it will assume an ordering  
>>>>> according to names/ranks first set in x-direction, than in y:
>>>>>
>>>>> y
>>>>> ^
>>>>> | 4-5-6
>>>>> | 1-2-3
>>>>> |--------> x
>>>>>
>>>>> Of course, if I now communicate the boundary values, I mix up  
>>>>> the domain (build by the other program).
>>>>>
>>>>> Is there a possibility / a flag to set the name of the ranks?
>>>>> Due to the fact that my application is written and working in  
>>>>> curvilinear coordinates and not in cartesian, I cannot just  
>>>>> switch the directions.
>>>>
>>>> What we recommend in this case is to just change the meaning of  
>>>> x, y, and z when you use the PETSc DA.  This does mean changing  
>>>> your code that uses the PETSc DA.
>>>
>>> The code is used as a module for many codes, so I would prefer to  
>>> not change the code (and the meaning of directions, that's not  
>>> user-friendly), but 'just' change the communicator.
>>>
>>>> I do not understand why curvilinear coordinates has anything to  
>>>> do with it. Another choice is to create a new MPI communicator  
>>>> that has the different ordering of the ranks of the processors  
>>>> and then using that comm to create the PETSc DA objects; then you  
>>>> would not need to change your code that calls PETSc.
>>>
>>> I tried some time before to use the PetscSetCommWorld() routine,  
>>> but I can't find it anymore, how can I set a new communicator in  
>>> PETSc3.0?
>>> The communicator, I want to use, is the MPI_COMM_WORLD, which  
>>> takes the first described ordering.
>>> Now I read that the MPI_COMM_WORLD is the default communicator for  
>>> PETSc. But why is the ordering than different?
>>>
>>> Sorry for all this question, but (as you can see) I really don't  
>>> understand this comm problem at the moment,
>>> Thanks for all,
>>> Rolf
>>>
>>>> Unfortunately PETSc doesn't have any way to flip how the DA  
>>>> handles the layout automatically.
>>>>
>>>>  Barry
>>>>
>>>>>
>>>>> Thanks a lot for your help,
>>>>> Rolf
>>>>
>>>
>>
>