MPI-layout of PETSc

[Rolf Kuiper]

Hi,

Am 30.06.2009 um 02:24 schrieb Barry Smith:
>
> On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote:
>
>> Hi PETSc users,
>>
>> I ran into trouble in combining my developed PETSc application with  
>> another code (based on another library called "ArrayLib").
>> The problem is the parallel layout for MPI, e.g. in 2D with 6 cpus  
>> the ArrayLib code gives the names/ranks of the local cpus first in  
>> y-direction, than in x (from last to first, in the same way the MPI  
>> arrays are called, like 3Darray[z][y][x]):
>>
>> y
>> ^
>> | 2-4-6
>> | 1-3-5
>> |--------> x
>>
>> If I call DACreate() from PETSc, it will assume an ordering  
>> according to names/ranks first set in x-direction, than in y:
>>
>> y
>> ^
>> | 4-5-6
>> | 1-2-3
>> |--------> x
>>
>> Of course, if I now communicate the boundary values, I mix up the  
>> domain (build by the other program).
>>
>> Is there a possibility / a flag to set the name of the ranks?
>> Due to the fact that my application is written and working in  
>> curvilinear coordinates and not in cartesian, I cannot just switch  
>> the directions.
>
>   What we recommend in this case is to just change the meaning of x,  
> y, and z when you use the PETSc DA.  This does mean changing your  
> code that uses the PETSc DA.

The code is used as a module for many codes, so I would prefer to not  
change the code (and the meaning of directions, that's not user- 
friendly), but 'just' change the communicator.

> I do not understand why curvilinear coordinates has anything to do  
> with it. Another choice is to create a new MPI communicator that has  
> the different ordering of the ranks of the processors and then using  
> that comm to create the PETSc DA objects; then you would not need to  
> change your code that calls PETSc.

I tried some time before to use the PetscSetCommWorld() routine, but I  
can't find it anymore, how can I set a new communicator in PETSc3.0?
The communicator, I want to use, is the MPI_COMM_WORLD, which takes  
the first described ordering.
Now I read that the MPI_COMM_WORLD is the default communicator for  
PETSc. But why is the ordering than different?

Sorry for all this question, but (as you can see) I really don't  
understand this comm problem at the moment,
Thanks for all,
Rolf

>   Unfortunately PETSc doesn't have any way to flip how the DA  
> handles the layout automatically.
>
>    Barry
>
>>
>> Thanks a lot for your help,
>> Rolf
>