[petsc-users] Question about PETScSF usage in DMNetwork/DMPlex

[Adrian Maldonado]

Hi,

I am trying to understand some of the data structures DMPlex/DMNetwork
creates and the relationship among them.

As an example, I have an small test circuit
(/src/ksp/ksp/examples/tutorials/network/ex1.c).

This is a graph that consists on 6 edges and 4 vertices, each one of those
having one degree of freedom.  When ran with two processors, each rank will
own 3 edges. Rank 0 will own one vertex (3 ghost) and Rank 1 will own 3
vertices.

These are some data structures for this problem. I am getting these data
structures inside DMNetworkDistribute
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMNetworkDistribute.html>

DM Object: Parallel Mesh 2 MPI processes
  type: plex
Parallel Mesh in 1 dimensions:
  0-cells: 4 3
  1-cells: 3 3
Labels:
  depth: 2 strata of sizes (4, 3)

This, as  I understand, is printing a tree with all the vertices and edges
in each processor (owned and ghost).

PetscSection Object: 2 MPI processes
  type not yet set
Process 0:
  (   0) dim  1 offset   0
  (   1) dim  1 offset   1
  (   2) dim  1 offset   2
  (   3) dim  1 offset   3
  (   4) dim -2 offset  -8
  (   5) dim -2 offset  -9
  (   6) dim -2 offset -10
Process 1:
  (   0) dim  1 offset   4
  (   1) dim  1 offset   5
  (   2) dim  1 offset   6
  (   3) dim  1 offset   7
  (   4) dim  1 offset   8
  (   5) dim  1 offset   9

This is a global PETSc section that gives me the global numbering for the
owned points and (garbage?) negative values for ghost.

Until here everything is good. But then I print the PetscSF that is created
by  'DMPlexDistribute'. This I do not understand:

PetscSF Object: Migration SF 2 MPI processes
  type: basic
    sort=rank-order
  [0] Number of roots=10, leaves=7, remote ranks=1
  [0] 0 <- (0,0)
  [0] 1 <- (0,1)
  [0] 2 <- (0,3)
  [0] 3 <- (0,6)
  [0] 4 <- (0,7)
  [0] 5 <- (0,8)
  [0] 6 <- (0,9)
  [1] Number of roots=0, leaves=6, remote ranks=1
  [1] 0 <- (0,2)
  [1] 1 <- (0,4)
  [1] 2 <- (0,5)
  [1] 3 <- (0,7)
  [1] 4 <- (0,8)
  [1] 5 <- (0,9)
  [0] Roots referenced by my leaves, by rank
  [0] 0: 7 edges
  [0]    0 <- 0
  [0]    1 <- 1
  [0]    2 <- 3
  [0]    3 <- 6
  [0]    4 <- 7
  [0]    5 <- 8
  [0]    6 <- 9
  [1] Roots referenced by my leaves, by rank
  [1] 0: 6 edges
  [1]    0 <- 2
  [1]    1 <- 4
  [1]    2 <- 5
  [1]    3 <- 7
  [1]    4 <- 8
  [1]    5 <- 9

I understand that SF is a data structure that saves references to pieces of
data that are now owned by the process (
https://arxiv.org/pdf/1506.06194v1.pdf, page 4).

Since the only ghost nodes appear in rank 0 (three ghost vertices) I would
expect something like:
*rank 0:*
  4 - (1, 3)  (to read: point 4 is owned by rank 1 and is rank's 1 point 3)
  etc...
*rank 1:*
  nothing

Is my intuition correct? If so, what does the star forest that I get from
DMPlexDistribute mean? I am printing the wrong thing?

Thank you

-- 
D. Adrian Maldonado, PhD Candidate
Electrical & Computer Engineering Dept.
Illinois Institute of Technology
3301 S. Dearborn Street, Chicago, IL 60616
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160916/d78f39a3/attachment.html>