<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sun, Mar 25, 2018 at 6:32 PM, Ellen M. Price <span dir="ltr"><<a href="mailto:ellen.price@cfa.harvard.edu" target="_blank">ellen.price@cfa.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am trying to understand some unusual behavior (at least, given my<br>
current understanding) in DMPlexDistribute. I have created a hex mesh<br>
and distributed it using the following snippet:<br>
<br>
/* "-refine" is an option set at runtime */<br>
PetscOptionsGetInt(NULL, NULL, "-refine", &refine, NULL);<br>
ierr = DMPlexCreateHexCylinderMesh(<wbr>PETSC_COMM_WORLD, refine,<br>
DM_BOUNDARY_NONE, &dmplex); CHKERRQ(ierr);<br>
DMView(dmplex, PETSC_VIEWER_STDOUT_WORLD);<br>
<br>
/* this is from the examples */<br>
ierr = DMPlexDistribute(dmplex, 0, NULL, &dmplex_dist); CHKERRQ(ierr);<br>
<br>
if (dmplex_dist)<br>
<br>
{<br>
<br>
DMDestroy(&dmplex);<br>
<br>
dmplex = dmplex_dist;<br>
<br>
}<br>
<br>
DMView(dmplex, PETSC_VIEWER_STDOUT_WORLD);<br>
<br>
So I do a view operation before and after the distribute call to see how<br>
the DM is structured. I do not understand what happens next:<br>
<br>
$ mpirun -n 4 ./myprogram -refine 2<br>
<br>
DM Object: 4 MPI processes<br>
type: plex<br>
DM_0x1f24d50_0 in 3 dimensions:<br>
0-cells: 445 445 445 445<br>
1-cells: 1196 1196 1196 1196<br>
2-cells: 1072 1072 1072 1072<br>
3-cells: 320 320 320 320<br>
Labels:<br>
depth: 4 strata with value/size (0 (445), 1 (1196), 2 (1072), 3 (320))<br>
DM Object: Parallel Mesh 4 MPI processes<br>
type: plex<br>
Parallel Mesh in 3 dimensions:<br>
0-cells: 445 445 445 445<br>
1-cells: 1196 1196 1196 1196<br>
2-cells: 1072 1072 1072 1072<br>
3-cells: 320 320 320 320<br>
Labels:<br>
depth: 4 strata with value/size (0 (445), 1 (1196), 2 (1072), 3 (320))<br>
<br>
No matter what I choose for the number of processors, every processor<br>
has a copy of all 320 cells (at this refinement level). Similar behavior<br>
at other refinement levels. The only thing that is changed is the name<br>
of the DM, to "Parallel Mesh". This is not what I would have expected<br>
given the description of DMPlexDistribute in the manual; I thought the<br>
cells would be split up between all available processors.<br>
<br>
I am also viewing this mesh in VTK and have noticed that the file size<br>
of the output scales with the number of processors, as if it is really<br>
writing each "copy" of the mesh and data stored in it to one big file.<br>
Again, not what I expected.<br>
<br>
Can someone clear up what is going on?</blockquote><div><br></div><div>It definitely does not work that way. Can you send the whole code you are running?</div><div>Its hard to tell where the problem might be from snippets. You could also try running</div><div>an example. For instance,</div><div><br></div><div> cd $PETSC_DIR/src/snes/examples/tutorials</div><div> make ex13</div><div><br></div><div>
<span></span>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">master *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ ./ex13 -cells 8,8 -dm_view</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">DM Object: Mesh 1 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: plex</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Mesh in 2 dimensions:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>0-cells: 81</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>1-cells: 208</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>2-cells: 128</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Labels:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>Face Sets: 1 strata with value/size (1 (32))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>marker: 1 strata with value/size (1 (64))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>depth: 3 strata with value/size (0 (81), 1 (208), 2 (128))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">master *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ $PETSC_DIR/arch-master-debug/bin/mpiexec -n 4 ./ex13 -cells 8,8 -dm_view</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">DM Object: Mesh 4 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: plex</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Mesh in 2 dimensions:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>0-cells: 26 26 27 25</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>1-cells: 57 57 58 56</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>2-cells: 32 32 32 32</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Labels:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>Face Sets: 1 strata with value/size (1 (8))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>marker: 1 strata with value/size (1 (17))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>depth: 3 strata with value/size (0 (26), 1 (57), 2 (32))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">master *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ $PETSC_DIR/arch-master-debug/bin/mpiexec -n 7 ./ex13 -cells 8,8 -dm_view</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">DM Object: Mesh 7 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: plex</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Mesh in 2 dimensions:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>0-cells: 17 16 16 17 17 17 19</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>1-cells: 34 33 33 35 34 35 35</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>2-cells: 18 18 18 19 18 19 18</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo">Labels:</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>Face Sets: 1 strata with value/size (1 (5))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>marker: 1 strata with value/size (1 (11))</p>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>depth: 3 strata with value/size (0 (17), 1 (34), 2 (18))</p>
<br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="HOEnZb"><font color="#888888"><br>
Ellen Price<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>