MPI-layout of PETSc

[Rolf Kuiper]

Thanks Barry!
It's working. But by the way: You simply should offer such a second  
communicator inside the PETSc-library.

Thanks for all your help, the support we got from this mailing list is  
amazing,
Rolf


Am 04.07.2009 um 01:44 schrieb Barry Smith:
>
>  Use MPI_Comm_split() with the same color for all processors, then  
> use the second integer argument to indicate the new rank you want  
> for the process.
> Choice the new rank so its x,y coordinate in the logical grid will  
> match the y,x coordinate in the cartesian grid.
>
>   Barry
>
> On Jul 3, 2009, at 12:09 PM, Rolf Kuiper wrote:
>
>> Hi Barry,
>>
>> I tried that already with:
>> First way by copying:
>> MPI_Comm_dup(PETSC_COMM_WORLD, &MyComm);
>>
>> Second way by creating:
>> int dims[3] = {0,0,0};
>> int ndims=3;
>> MPI_Dims_create(NumberOfProcessors, ndims, dims);
>> int false = 0; int true = 1;
>> int periods[3] = { false, false, true };
>> int reorder = true;
>> MPI_Comm MyComm;
>> MPI_Cart_create(PETSC_COMM_WORLD, ndims, dims, periods, reorder,  
>> &MyComm);
>>
>> in the end then:
>> PETSC_COMM_WORLD = MyComm;
>>
>> I test the MyComm with MPI_Topo_test(); and it is cartesian, yes.
>> I can the coordinates of the cpus with MPI_Cart_coords(MyComm,  
>> LocalRank, ndims, coords); , but I found no way to set/rearrange  
>> these coordinates.
>>
>> Do you can help me in that case or have I to ask a MPI-support?
>>
>> Thanks for all,
>> Rolf
>>
>>
>> Am 03.07.2009 um 17:56 schrieb Barry Smith:
>>>
>>>  In designing the PETSc DA I did not (by ignorance) follow the  
>>> layout approach of the MPI cartesian MPI_Cart_create (that gives  
>>> the first local cpus first in the y-direction).
>>> I had it put the first cpus in the x-direction.
>>>
>>>  What you need to do is create a new communicator that changes the  
>>> order of the processors so that when used by the PETSc DA they lie  
>>> out in the ordering that matches the other code. You will need to  
>>> read up on the MPI_Cart stuff.
>>>
>>>  To change PETSC_COMM_WORLD you simply set PETSC_COMM_WORLD =  
>>> yournewcom BEFORE calling PetscInitialize().
>>>
>>>  Barry
>>>
>>> On Jul 3, 2009, at 3:52 AM, Rolf Kuiper wrote:
>>>
>>>> Hi,
>>>>
>>>> Am 30.06.2009 um 02:24 schrieb Barry Smith:
>>>>>
>>>>> On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote:
>>>>>
>>>>>> Hi PETSc users,
>>>>>>
>>>>>> I ran into trouble in combining my developed PETSc application  
>>>>>> with another code (based on another library called "ArrayLib").
>>>>>> The problem is the parallel layout for MPI, e.g. in 2D with 6  
>>>>>> cpus the ArrayLib code gives the names/ranks of the local cpus  
>>>>>> first in y-direction, than in x (from last to first, in the  
>>>>>> same way the MPI arrays are called, like 3Darray[z][y][x]):
>>>>>>
>>>>>> y
>>>>>> ^
>>>>>> | 2-4-6
>>>>>> | 1-3-5
>>>>>> |--------> x
>>>>>>
>>>>>> If I call DACreate() from PETSc, it will assume an ordering  
>>>>>> according to names/ranks first set in x-direction, than in y:
>>>>>>
>>>>>> y
>>>>>> ^
>>>>>> | 4-5-6
>>>>>> | 1-2-3
>>>>>> |--------> x
>>>>>>
>>>>>> Of course, if I now communicate the boundary values, I mix up  
>>>>>> the domain (build by the other program).
>>>>>>
>>>>>> Is there a possibility / a flag to set the name of the ranks?
>>>>>> Due to the fact that my application is written and working in  
>>>>>> curvilinear coordinates and not in cartesian, I cannot just  
>>>>>> switch the directions.
>>>>>
>>>>> What we recommend in this case is to just change the meaning of  
>>>>> x, y, and z when you use the PETSc DA.  This does mean changing  
>>>>> your code that uses the PETSc DA.
>>>>
>>>> The code is used as a module for many codes, so I would prefer to  
>>>> not change the code (and the meaning of directions, that's not  
>>>> user-friendly), but 'just' change the communicator.
>>>>
>>>>> I do not understand why curvilinear coordinates has anything to  
>>>>> do with it. Another choice is to create a new MPI communicator  
>>>>> that has the different ordering of the ranks of the processors  
>>>>> and then using that comm to create the PETSc DA objects; then  
>>>>> you would not need to change your code that calls PETSc.
>>>>
>>>> I tried some time before to use the PetscSetCommWorld() routine,  
>>>> but I can't find it anymore, how can I set a new communicator in  
>>>> PETSc3.0?
>>>> The communicator, I want to use, is the MPI_COMM_WORLD, which  
>>>> takes the first described ordering.
>>>> Now I read that the MPI_COMM_WORLD is the default communicator  
>>>> for PETSc. But why is the ordering than different?
>>>>
>>>> Sorry for all this question, but (as you can see) I really don't  
>>>> understand this comm problem at the moment,
>>>> Thanks for all,
>>>> Rolf
>>>>
>>>>> Unfortunately PETSc doesn't have any way to flip how the DA  
>>>>> handles the layout automatically.
>>>>>
>>>>> Barry
>>>>>
>>>>>>
>>>>>> Thanks a lot for your help,
>>>>>> Rolf
>>>>>
>>>>
>>>
>>
>