<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Barry,<div><br></div><div>I tried that already with:</div><div>First way by copying:</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #642880">MPI_Comm_dup</span>(PETSC_COMM_WORLD, &MyComm);</div></div><div><br></div><div>Second way by creating:</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #7f0055">int</span> dims[3] = {0,0,0};</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #7f0055">int</span> ndims=3;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #642880">MPI_Dims_create</span>(NumberOfProcessors, ndims, dims);</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #7f0055">int</span> false = 0; <span style="color: #7f0055">int</span> true = 1;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #7f0055">int</span> periods[3] = { false, false, true };</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #7f0055">int</span> reorder = true;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #005032">MPI_Comm</span> MyComm;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #642880">MPI_Cart_create</span>(PETSC_COMM_WORLD, ndims, dims, periods, reorder, &MyComm);</div></div><div><br></div><div>in the end then:</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">PETSC_COMM_WORLD = MyComm;</div></div><div><br><div><div>I test the MyComm with <span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; "><span style="color: #642880">MPI_Topo_test</span>(); <span class="Apple-style-span" style="font-family: Helvetica; font-size: medium; ">and it is cartesian, yes.</span></span></div><div>I can the coordinates of the cpus with <span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; "><span style="color: #642880">MPI_Cart_coords</span>(MyComm, LocalRank, ndims, coords); <span class="Apple-style-span" style="font-family: Helvetica; font-size: medium; ">, but I found no way to set/rearrange these coordinates.</span></span></div><div><br></div><div>Do you can help me in that case or have I to ask a MPI-support?</div><div><br></div><div>Thanks for all,</div><div>Rolf</div><div><br></div><div><br></div><div>Am 03.07.2009 um 17:56 schrieb Barry Smith:</div><blockquote type="cite"><div><br> In designing the PETSc DA I did not (by ignorance) follow the layout approach of the MPI cartesian MPI_Cart_create (that gives the first local cpus first in the y-direction).<br>I had it put the first cpus in the x-direction.<br><br> What you need to do is create a new communicator that changes the order of the processors so that when used by the PETSc DA they lie out in the ordering that matches the other code. You will need to read up on the MPI_Cart stuff.<br><br> To change PETSC_COMM_WORLD you simply set PETSC_COMM_WORLD = yournewcom BEFORE calling PetscInitialize().<br><br> Barry<br><br>On Jul 3, 2009, at 3:52 AM, Rolf Kuiper wrote:<br><br><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Am 30.06.2009 um 02:24 schrieb Barry Smith:<br></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hi PETSc users,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I ran into trouble in combining my developed PETSc application with another code (based on another library called "ArrayLib").<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The problem is the parallel layout for MPI, e.g. in 2D with 6 cpus the ArrayLib code gives the names/ranks of the local cpus first in y-direction, than in x (from last to first, in the same way the MPI arrays are called, like 3Darray[z][y][x]):<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">y<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">^<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">| 2-4-6<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">| 1-3-5<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">|--------> x<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">If I call DACreate() from PETSc, it will assume an ordering according to names/ranks first set in x-direction, than in y:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">y<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">^<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">| 4-5-6<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">| 1-2-3<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">|--------> x<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Of course, if I now communicate the boundary values, I mix up the domain (build by the other program).<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Is there a possibility / a flag to set the name of the ranks?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Due to the fact that my application is written and working in curvilinear coordinates and not in cartesian, I cannot just switch the directions.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> What we recommend in this case is to just change the meaning of x, y, and z when you use the PETSc DA. This does mean changing your code that uses the PETSc DA.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The code is used as a module for many codes, so I would prefer to not change the code (and the meaning of directions, that's not user-friendly), but 'just' change the communicator.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">I do not understand why curvilinear coordinates has anything to do with it. Another choice is to create a new MPI communicator that has the different ordering of the ranks of the processors and then using that comm to create the PETSc DA objects; then you would not need to change your code that calls PETSc.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I tried some time before to use the PetscSetCommWorld() routine, but I can't find it anymore, how can I set a new communicator in PETSc3.0?<br></blockquote><blockquote type="cite">The communicator, I want to use, is the MPI_COMM_WORLD, which takes the first described ordering.<br></blockquote><blockquote type="cite">Now I read that the MPI_COMM_WORLD is the default communicator for PETSc. But why is the ordering than different?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Sorry for all this question, but (as you can see) I really don't understand this comm problem at the moment,<br></blockquote><blockquote type="cite">Thanks for all,<br></blockquote><blockquote type="cite">Rolf<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"> Unfortunately PETSc doesn't have any way to flip how the DA handles the layout automatically.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Barry<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Thanks a lot for your help,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Rolf<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><br></div></blockquote></div><br></div></body></html>