<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Aug 8, 2015 at 2:45 PM, Mani Chandra <span dir="ltr"><<a href="mailto:mc0710@gmail.com" target="_blank">mc0710@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks. Any suggestions for a fix?</div></blockquote><div><br></div><div>You have to deal with the right part of the domain in your application code. I have no idea</div><div>how you are handling this, and its not in the code below.</div><div><br></div><div> Matt</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Reorder the indices in arrayApplication?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Aug 8, 2015 at 2:19 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div>On Sat, Aug 8, 2015 at 1:52 PM, Mani Chandra <span dir="ltr"><<a href="mailto:mc0710@gmail.com" target="_blank">mc0710@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I'm having trouble interfacing petsc to an application which I think is related to the ordering of the nodes. Here's what I'm trying to do:</div><div><br></div><div>The application uses a structured grid with a global array having dimensions N1 x N2, which is then decomposed into a local array with dimensions NX1 x NX2.</div><div><br></div><div>I create a Petsc DMDA using </div><div><br></div><div><div> DMDACreate2d(MPI_COMM_WORLD,</div><div> DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC,</div><div> DMDA_STENCIL_BOX,</div><div> N1, N2,</div><div> N1/NX1, N2/NX2,</div><div> 1, nghost, PETSC_NULL, PETSC_NULL,</div><div> &dmda);</div></div><div><br></div><div>and then use this to create a vec:</div><div><br></div><div><div> DMCreateGlobalVector(dmda, &vec);</div></div><div><br></div><div>Now I copy the local contents of the application array to the petsc array using the following:</div><div><br></div><div>Let i, j be the application indices and iPetsc and jPetsc be petsc's indices, then:</div><div><div> </div><div>DMDAGetCorners(dmda, &iStart, &jStart, &kStart,</div><div> &iSize, &jSize, &kSize</div><div> );</div></div><div><br></div><div><br></div><div>double **arrayPetsc;</div><div><div>DMDAVecGetArray(dmda, vec, &arrayPetsc);</div></div><div><br></div><div>for (int j=0, jPetsc=jStart; j<NX2, jPetsc<jStart+jSize; j++, jPetsc++)</div><div>{</div><div> for (int i=0, iPetsc=iStart; i<NX1, iPetsc<iStart+iSize; i++, iPetsc++)</div><div> {</div><div> arrayPetsc[jPetsc][iPetsc] = arrayApplication[j][i];</div><div> }</div><div>}</div><div><br></div><div><div>DMDAVecRestoreArray(dmda, vec, &arrayPetsc);</div></div><div><br></div><div>Now if I VecView(vec, viewer) and look at the data that petsc has, it looks right when run with 1 proc, but if I use 4 procs it's all messed up (see attached plots).</div><div><br></div><div>I should probably be using the AO object but its not clear how. Could you help me out?</div></div></blockquote><div><br></div></div></div><div>It looks like you have the global order of processes reversed, meaning you have</div><div><br></div><div> 1 3</div><div><br></div><div> 0 2</div><div><br></div><div>and it should be</div><div><br></div><div> 2 3</div><div><br></div><div> 0 1 </div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div>Mani</div></div></blockquote></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888">-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>