MPI-layout of PETSc

[Barry Smith]

    Send us the code to do the conversion and we'll include as a  
utility.

    Barry

On Jul 4, 2009, at 6:08 AM, Rolf Kuiper wrote:

> Thanks Barry!
> It's working. But by the way: You simply should offer such a second  
> communicator inside the PETSc-library.
>
> Thanks for all your help, the support we got from this mailing list  
> is amazing,
> Rolf
>
>
> Am 04.07.2009 um 01:44 schrieb Barry Smith:
>>
>> Use MPI_Comm_split() with the same color for all processors, then  
>> use the second integer argument to indicate the new rank you want  
>> for the process.
>> Choice the new rank so its x,y coordinate in the logical grid will  
>> match the y,x coordinate in the cartesian grid.
>>
>>  Barry
>>
>> On Jul 3, 2009, at 12:09 PM, Rolf Kuiper wrote:
>>
>>> Hi Barry,
>>>
>>> I tried that already with:
>>> First way by copying:
>>> MPI_Comm_dup(PETSC_COMM_WORLD, &MyComm);
>>>
>>> Second way by creating:
>>> int dims[3] = {0,0,0};
>>> int ndims=3;
>>> MPI_Dims_create(NumberOfProcessors, ndims, dims);
>>> int false = 0; int true = 1;
>>> int periods[3] = { false, false, true };
>>> int reorder = true;
>>> MPI_Comm MyComm;
>>> MPI_Cart_create(PETSC_COMM_WORLD, ndims, dims, periods, reorder,  
>>> &MyComm);
>>>
>>> in the end then:
>>> PETSC_COMM_WORLD = MyComm;
>>>
>>> I test the MyComm with MPI_Topo_test(); and it is cartesian, yes.
>>> I can the coordinates of the cpus with MPI_Cart_coords(MyComm,  
>>> LocalRank, ndims, coords); , but I found no way to set/rearrange  
>>> these coordinates.
>>>
>>> Do you can help me in that case or have I to ask a MPI-support?
>>>
>>> Thanks for all,
>>> Rolf
>>>
>>>
>>> Am 03.07.2009 um 17:56 schrieb Barry Smith:
>>>>
>>>> In designing the PETSc DA I did not (by ignorance) follow the  
>>>> layout approach of the MPI cartesian MPI_Cart_create (that gives  
>>>> the first local cpus first in the y-direction).
>>>> I had it put the first cpus in the x-direction.
>>>>
>>>> What you need to do is create a new communicator that changes the  
>>>> order of the processors so that when used by the PETSc DA they  
>>>> lie out in the ordering that matches the other code. You will  
>>>> need to read up on the MPI_Cart stuff.
>>>>
>>>> To change PETSC_COMM_WORLD you simply set PETSC_COMM_WORLD =  
>>>> yournewcom BEFORE calling PetscInitialize().
>>>>
>>>> Barry
>>>>
>>>> On Jul 3, 2009, at 3:52 AM, Rolf Kuiper wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Am 30.06.2009 um 02:24 schrieb Barry Smith:
>>>>>>
>>>>>> On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote:
>>>>>>
>>>>>>> Hi PETSc users,
>>>>>>>
>>>>>>> I ran into trouble in combining my developed PETSc application  
>>>>>>> with another code (based on another library called "ArrayLib").
>>>>>>> The problem is the parallel layout for MPI, e.g. in 2D with 6  
>>>>>>> cpus the ArrayLib code gives the names/ranks of the local cpus  
>>>>>>> first in y-direction, than in x (from last to first, in the  
>>>>>>> same way the MPI arrays are called, like 3Darray[z][y][x]):
>>>>>>>
>>>>>>> y
>>>>>>> ^
>>>>>>> | 2-4-6
>>>>>>> | 1-3-5
>>>>>>> |--------> x
>>>>>>>
>>>>>>> If I call DACreate() from PETSc, it will assume an ordering  
>>>>>>> according to names/ranks first set in x-direction, than in y:
>>>>>>>
>>>>>>> y
>>>>>>> ^
>>>>>>> | 4-5-6
>>>>>>> | 1-2-3
>>>>>>> |--------> x
>>>>>>>
>>>>>>> Of course, if I now communicate the boundary values, I mix up  
>>>>>>> the domain (build by the other program).
>>>>>>>
>>>>>>> Is there a possibility / a flag to set the name of the ranks?
>>>>>>> Due to the fact that my application is written and working in  
>>>>>>> curvilinear coordinates and not in cartesian, I cannot just  
>>>>>>> switch the directions.
>>>>>>
>>>>>> What we recommend in this case is to just change the meaning of  
>>>>>> x, y, and z when you use the PETSc DA.  This does mean changing  
>>>>>> your code that uses the PETSc DA.
>>>>>
>>>>> The code is used as a module for many codes, so I would prefer  
>>>>> to not change the code (and the meaning of directions, that's  
>>>>> not user-friendly), but 'just' change the communicator.
>>>>>
>>>>>> I do not understand why curvilinear coordinates has anything to  
>>>>>> do with it. Another choice is to create a new MPI communicator  
>>>>>> that has the different ordering of the ranks of the processors  
>>>>>> and then using that comm to create the PETSc DA objects; then  
>>>>>> you would not need to change your code that calls PETSc.
>>>>>
>>>>> I tried some time before to use the PetscSetCommWorld() routine,  
>>>>> but I can't find it anymore, how can I set a new communicator in  
>>>>> PETSc3.0?
>>>>> The communicator, I want to use, is the MPI_COMM_WORLD, which  
>>>>> takes the first described ordering.
>>>>> Now I read that the MPI_COMM_WORLD is the default communicator  
>>>>> for PETSc. But why is the ordering than different?
>>>>>
>>>>> Sorry for all this question, but (as you can see) I really don't  
>>>>> understand this comm problem at the moment,
>>>>> Thanks for all,
>>>>> Rolf
>>>>>
>>>>>> Unfortunately PETSc doesn't have any way to flip how the DA  
>>>>>> handles the layout automatically.
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>>
>>>>>>> Thanks a lot for your help,
>>>>>>> Rolf
>>>>>>
>>>>>
>>>>
>>>
>>
>