<div dir="ltr">Just one addition about one thing I've noticed.<div><br></div><div>The section:</div><div><font face="monospace, monospace" size="1"><br></font></div><div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1">PetscSection Object: 2 MPI processes</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> type not yet set</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1">Process 0:</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 0) dim 1 offset 0</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 1) dim 1 offset 1</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 2) dim 1 offset 2</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 3) dim 1 offset 3</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 4) dim -2 offset -8</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 5) dim -2 offset -9</font></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1"> ( 6) dim -2 offset -10</font></div></div><div style="color:rgb(0,0,0)"><div style="font-size:12.8px"><font face="monospace, monospace" size="1">Process 1:</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 0) dim 1 offset 4</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 1) dim 1 offset 5</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 2) dim 1 offset 6</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 3) dim 1 offset 7</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 4) dim 1 offset 8</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"> ( 5) dim 1 offset 9</font></div><div style="font-size:12.8px"><font face="monospace, monospace" size="1"><br></font></div></div><div style="color:rgb(0,0,0)"><font face="monospace, monospace" size="1">For the ghost values 4, 5, 6... is encoding the ghost values as rank = -(-2 + 1) and offset = -(-8 + 1) ?</font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 16, 2016 at 11:36 AM, Adrian Maldonado <span dir="ltr"><<a href="mailto:dmaldona@hawk.iit.edu" target="_blank">dmaldona@hawk.iit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I am trying to understand some of the data structures DMPlex/DMNetwork creates and the relationship among them.</div><div><br></div><div>As an example, I have an small test circuit (/src/ksp/ksp/examples/<wbr>tutorials/network/ex1.c).<br></div><div><br></div><div>This is a graph that consists on 6 edges and 4 vertices, each one of those having one degree of freedom. When ran with two processors, each rank will own 3 edges. Rank 0 will own one vertex (3 ghost) and Rank 1 will own 3 vertices.</div><div><br></div><div>These are some data structures for this problem. I am getting these data structures inside <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMNetworkDistribute.html" target="_blank">DMNetworkDistribute</a></div><div><br></div><div><font face="monospace, monospace" size="1">DM Object: Parallel Mesh 2 MPI processes<br></font></div><div><div><font face="monospace, monospace" size="1"> type: plex</font></div><div><font face="monospace, monospace" size="1">Parallel Mesh in 1 dimensions:</font></div><div><font face="monospace, monospace" size="1"> 0-cells: 4 3</font></div><div><font face="monospace, monospace" size="1"> 1-cells: 3 3</font></div><div><font face="monospace, monospace" size="1">Labels:</font></div><div><font face="monospace, monospace" size="1"> depth: 2 strata of sizes (4, 3)</font></div></div><div><br></div><div>This, as I understand, is printing a tree with all the vertices and edges in each processor (owned and ghost).</div><div><br></div><div><div><font face="monospace, monospace" size="1">PetscSection Object: 2 MPI processes</font></div><div><font face="monospace, monospace" size="1"> type not yet set</font></div><div><font face="monospace, monospace" size="1">Process 0:</font></div><div><font face="monospace, monospace" size="1"> ( 0) dim 1 offset 0</font></div><div><font face="monospace, monospace" size="1"> ( 1) dim 1 offset 1</font></div><div><font face="monospace, monospace" size="1"> ( 2) dim 1 offset 2</font></div><div><font face="monospace, monospace" size="1"> ( 3) dim 1 offset 3</font></div><div><font face="monospace, monospace" size="1"> ( 4) dim -2 offset -8</font></div><div><font face="monospace, monospace" size="1"> ( 5) dim -2 offset -9</font></div><div><font face="monospace, monospace" size="1"> ( 6) dim -2 offset -10</font></div><div><font face="monospace, monospace" size="1">Process 1:</font></div><div><font face="monospace, monospace" size="1"> ( 0) dim 1 offset 4</font></div><div><font face="monospace, monospace" size="1"> ( 1) dim 1 offset 5</font></div><div><font face="monospace, monospace" size="1"> ( 2) dim 1 offset 6</font></div><div><font face="monospace, monospace" size="1"> ( 3) dim 1 offset 7</font></div><div><font face="monospace, monospace" size="1"> ( 4) dim 1 offset 8</font></div><div><font face="monospace, monospace" size="1"> ( 5) dim 1 offset 9</font></div></div><div><br></div><div>This is a global PETSc section that gives me the global numbering for the owned points and (garbage?) negative values for ghost.</div><div><br></div><div>Until here everything is good. But then I print the PetscSF that is created by 'DMPlexDistribute'. This I do not understand:</div><div><br></div><div><div><font face="monospace, monospace" size="1">PetscSF Object: Migration SF 2 MPI processes</font></div><div><font face="monospace, monospace" size="1"> type: basic</font></div><div><font face="monospace, monospace" size="1"> sort=rank-order</font></div><div><font face="monospace, monospace" size="1"> [0] Number of roots=10, leaves=7, remote ranks=1</font></div><div><font face="monospace, monospace" size="1"> [0] 0 <- (0,0)</font></div><div><font face="monospace, monospace" size="1"> [0] 1 <- (0,1)</font></div><div><font face="monospace, monospace" size="1"> [0] 2 <- (0,3)</font></div><div><font face="monospace, monospace" size="1"> [0] 3 <- (0,6)</font></div><div><font face="monospace, monospace" size="1"> [0] 4 <- (0,7)</font></div><div><font face="monospace, monospace" size="1"> [0] 5 <- (0,8)</font></div><div><font face="monospace, monospace" size="1"> [0] 6 <- (0,9)</font></div><div><font face="monospace, monospace" size="1"> [1] Number of roots=0, leaves=6, remote ranks=1</font></div><div><font face="monospace, monospace" size="1"> [1] 0 <- (0,2)</font></div><div><font face="monospace, monospace" size="1"> [1] 1 <- (0,4)</font></div><div><font face="monospace, monospace" size="1"> [1] 2 <- (0,5)</font></div><div><font face="monospace, monospace" size="1"> [1] 3 <- (0,7)</font></div><div><font face="monospace, monospace" size="1"> [1] 4 <- (0,8)</font></div><div><font face="monospace, monospace" size="1"> [1] 5 <- (0,9)</font></div><div><font face="monospace, monospace" size="1"> [0] Roots referenced by my leaves, by rank</font></div><div><font face="monospace, monospace" size="1"> [0] 0: 7 edges</font></div><div><font face="monospace, monospace" size="1"> [0] 0 <- 0</font></div><div><font face="monospace, monospace" size="1"> [0] 1 <- 1</font></div><div><font face="monospace, monospace" size="1"> [0] 2 <- 3</font></div><div><font face="monospace, monospace" size="1"> [0] 3 <- 6</font></div><div><font face="monospace, monospace" size="1"> [0] 4 <- 7</font></div><div><font face="monospace, monospace" size="1"> [0] 5 <- 8</font></div><div><font face="monospace, monospace" size="1"> [0] 6 <- 9</font></div><div><font face="monospace, monospace" size="1"> [1] Roots referenced by my leaves, by rank</font></div><div><font face="monospace, monospace" size="1"> [1] 0: 6 edges</font></div><div><font face="monospace, monospace" size="1"> [1] 0 <- 2</font></div><div><font face="monospace, monospace" size="1"> [1] 1 <- 4</font></div><div><font face="monospace, monospace" size="1"> [1] 2 <- 5</font></div><div><font face="monospace, monospace" size="1"> [1] 3 <- 7</font></div><div><font face="monospace, monospace" size="1"> [1] 4 <- 8</font></div><div><font face="monospace, monospace" size="1"> [1] 5 <- 9</font></div></div><div><br></div><div>I understand that SF is a data structure that saves references to pieces of data that are now owned by the process (<a href="https://arxiv.org/pdf/1506.06194v1.pdf" target="_blank">https://arxiv.org/pdf/1506.<wbr>06194v1.pdf</a>, page 4).</div><div><br></div><div>Since the only ghost nodes appear in rank 0 (three ghost vertices) I would expect something like:</div><div><b>rank 0:</b></div><div> 4 - (1, 3) (to read: point 4 is owned by rank 1 and is rank's 1 point 3)</div><div> etc...</div><div><b>rank 1:</b></div><div> nothing</div><div> </div><div>Is my intuition correct? If so, what does the star forest that I get from DMPlexDistribute mean? I am printing the wrong thing?</div><div><br></div><div>Thank you</div><span class="HOEnZb"><font color="#888888"><div><div><br></div>-- <br><div><div dir="ltr">D. Adrian Maldonado, PhD Candidate<br>Electrical & Computer Engineering Dept.<br>Illinois Institute of Technology<br>3301 S. Dearborn Street, Chicago, IL 60616<br></div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">D. Adrian Maldonado, PhD Candidate<br>Electrical & Computer Engineering Dept.<br>Illinois Institute of Technology<br>3301 S. Dearborn Street, Chicago, IL 60616<br></div></div>
</div>