<div dir="ltr">Hi Matthew,<div><br></div><div>Thank you for your response. It works right now.</div><div><br></div>dmplex before distribution<br>DM Object: 3 MPI processes<br> type: plex<br>DM_0x84000004_0 in 3 dimensions:<br> 0-cells: 7 0 0<br> 1-cells: 17 0 0<br> 2-cells: 17 0 0<br> 3-cells: 6 0 0<br>Labels:<br> depth: 4 strata with value/size (0 (7), 1 (17), 2 (17), 3 (6))<br>dmplex after distribution<br>DM Object: Parallel Mesh 3 MPI processes<br> type: plex<br>Parallel Mesh in 3 dimensions:<br> 0-cells: 5 5 5<br> 1-cells: 9 9 9<br> 2-cells: 7 7 7<br> 3-cells: 2 2 2<br>Labels:<br> depth: 4 strata with value/size (0 (5), 1 (9), 2 (7), 3 (2))<br><div> </div><div>Thanks,</div><div>Swarnava</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 14, 2019 at 8:47 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Aug 14, 2019 at 10:48 PM Swarnava Ghosh <<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Matthew,</div><div><br></div>"It looks like you are running things with the wrong 'mpirun' ", Could you please elaborate on this? I have another DMDA in my code, which is correctly being parallelized. </div></blockquote><div><br></div><div>Ah, I see it now. You are feeding in the initial mesh on every process. Normally one process generates the mesh</div><div>and the other ones just give 0 for num cells and vertices.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div>SG</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 14, 2019 at 7:35 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Aug 14, 2019 at 10:23 PM Swarnava Ghosh <<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matthew,<div><br></div><div>I added DMView(pCgdft->dmplex,PETSC_VIEWER_STDOUT_WORLD); before and after distribution, and I get the following:</div></div></blockquote><div><br></div><div>It looks like you are running things with the wrong 'mpirun'</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>dmplex before distribution<br>DM Object: 3 MPI processes<br> type: plex<br>DM_0x84000004_0 in 3 dimensions:<br> 0-cells: 7 7 7<br> 1-cells: 17 17 17<br> 2-cells: 17 17 17<br> 3-cells: 6 6 6<br>Labels:<br> depth: 4 strata with value/size (0 (7), 1 (17), 2 (17), 3 (6))<br>dmplex after distribution<br>DM Object: Parallel Mesh 3 MPI processes<br> type: plex<br>Parallel Mesh in 3 dimensions:<br> 0-cells: 7 7 7<br> 1-cells: 17 17 17<br> 2-cells: 17 17 17<br> 3-cells: 6 6 6<br>Labels:<br> depth: 4 strata with value/size (0 (7), 1 (17), 2 (17), 3 (6))<br></div><div><br></div><div>Thanks,</div><div>SG</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 14, 2019 at 6:48 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">DMView() the mesh before and after distribution, so we can see what we have.<div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 14, 2019 at 5:30 PM Swarnava Ghosh via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi PETSc team and users,<div><br></div><div>I am trying to create a 3D dmplex mesh using DMPlexCreateFromCellList, then distribute it, and find out the coordinates of the vertices owned by each process.</div><div>My cell list is as follows:</div><div>numCells: 6 <br>numVertices: 7 <br>numCorners: 4 <br>cells: <br>0 <br>3 <br>2 <br>1 <br>4 <br>0 <br>2 <br>1 <br>6 <br>4 <br>2 <br>1 <br>3 <br>6 <br>2 <br>1 <br>5 <br>4 <br>6 <br>1 <br>3 <br>5 <br>6 <br>1 <br>vertexCoords: <br>-6.043000 <br>-5.233392 <br>-4.924000 <br>-3.021500 <br>0.000000 <br>-4.924000 <br>-3.021500 <br>-3.488928 <br>0.000000 <br>-6.043000 <br>1.744464 <br>0.000000 <br>0.000000 <br>-5.233392 <br>-4.924000 <br>3.021500 <br>0.000000 <br>-4.924000 <br>3.021500 <br>-3.488928 <br>0.000000 <br></div><div><br></div><div>After reading this information, I do</div><div> ierr= DMPlexCreateFromCellList(PETSC_COMM_WORLD,3,pCgdft->numCellsESP,pCgdft->NESP,pCgdft->numCornersESP,interpolate,pCgdft->cellsESP,3,pCgdft->vertexCoordsESP,&pCgdft->dmplex);<br></div><div><br></div><div>ierr = DMPlexDistribute(pCgdft->dmplex,0,&pCgdft->dmplexSF, &distributedMesh);CHKERRQ(ierr);<br><br> if (distributedMesh) {<br> printf("mesh is distributed \n");<br> ierr = DMDestroy(&pCgdft->dmplex);CHKERRQ(ierr);<br> pCgdft->dmplex = distributedMesh;<br> }<br></div><div><br></div><div> DMGetCoordinates(pCgdft->dmplex,&VC);<br> VecView(VC,PETSC_VIEWER_STDOUT_WORLD);<br></div><div><br></div><div>On running this with 3 mpi processes, From VecView, I see that all the processes own all the vertices. Why is the dmplex not being distributed?</div><div><br></div><div>The VecView is :</div><div>Process [0]<br>-6.043<br>-5.23339<br>-4.924<br>-3.0215<br>0.<br>-4.924<br>-3.0215<br>-3.48893<br>0.<br>-6.043<br>1.74446<br>0.<br>0.<br>-5.23339<br>-4.924<br>3.0215<br>0.<br>-4.924<br>3.0215<br>-3.48893<br>0.<br>Process [1]<br>-6.043<br>-5.23339<br>-4.924<br>-3.0215<br>0.<br>-4.924<br>-3.0215<br>-3.48893<br>0.<br>-6.043<br>1.74446<br>0.<br>0.<br>-5.23339<br>-4.924<br>3.0215<br>0.<br>-4.924<br>3.0215<br>-3.48893<br>0.<br>Process [2]<br>-6.043<br>-5.23339<br>-4.924<br>-3.0215<br>0.<br>-4.924<br>-3.0215<br>-3.48893<br>0.<br>-6.043<br>1.74446<br>0.<br>0.<br>-5.23339<br>-4.924<br>3.0215<br>0.<br>-4.924<br>3.0215<br>-3.48893<br>0.<br></div><div><br></div><div>Thanks,</div><div>SG</div><div><br></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_4344847270432761447gmail-m_-5975795381903042793gmail-m_-3532065257364095128gmail-m_126871636754025037gmail-m_7904107780803670279gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_4344847270432761447gmail-m_-5975795381903042793gmail-m_-3532065257364095128gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_4344847270432761447gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>