[petsc-users] Question about PETScSF usage in DMNetwork/DMPlex

[Matthew Knepley]

On Fri, Sep 16, 2016 at 12:54 PM, Adrian Maldonado <dmaldona at hawk.iit.edu>
wrote:

> Just one addition about one thing I've noticed.
>
> The section:
>
> PetscSection Object: 2 MPI processes
>   type not yet set
> Process 0:
>   (   0) dim  1 offset   0
>   (   1) dim  1 offset   1
>   (   2) dim  1 offset   2
>   (   3) dim  1 offset   3
>   (   4) dim -2 offset  -8
>   (   5) dim -2 offset  -9
>   (   6) dim -2 offset -10
> Process 1:
>   (   0) dim  1 offset   4
>   (   1) dim  1 offset   5
>   (   2) dim  1 offset   6
>   (   3) dim  1 offset   7
>   (   4) dim  1 offset   8
>   (   5) dim  1 offset   9
>
> For the ghost values 4, 5, 6... is encoding the ghost values as rank =
> -(-2 + 1) and offset = -(-8 + 1) ?
>

Yes, the encoding is

  -(val + 1)

Notice that this is reversible. Say my value is v, then my ghost value is

  -(v + 1)

and I can get my original value using

 -(-(v+1) + 1)

= -(-v - 1 + 1) = v

   Matt


> On Fri, Sep 16, 2016 at 11:36 AM, Adrian Maldonado <dmaldona at hawk.iit.edu>
> wrote:
>
>> Hi,
>>
>> I am trying to understand some of the data structures DMPlex/DMNetwork
>> creates and the relationship among them.
>>
>> As an example, I have an small test circuit (/src/ksp/ksp/examples/tutoria
>> ls/network/ex1.c).
>>
>> This is a graph that consists on 6 edges and 4 vertices, each one of
>> those having one degree of freedom.  When ran with two processors, each
>> rank will own 3 edges. Rank 0 will own one vertex (3 ghost) and Rank 1 will
>> own 3 vertices.
>>
>> These are some data structures for this problem. I am getting these data
>> structures inside DMNetworkDistribute
>> <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMNetworkDistribute.html>
>>
>> DM Object: Parallel Mesh 2 MPI processes
>>   type: plex
>> Parallel Mesh in 1 dimensions:
>>   0-cells: 4 3
>>   1-cells: 3 3
>> Labels:
>>   depth: 2 strata of sizes (4, 3)
>>
>> This, as  I understand, is printing a tree with all the vertices and
>> edges in each processor (owned and ghost).
>>
>> PetscSection Object: 2 MPI processes
>>   type not yet set
>> Process 0:
>>   (   0) dim  1 offset   0
>>   (   1) dim  1 offset   1
>>   (   2) dim  1 offset   2
>>   (   3) dim  1 offset   3
>>   (   4) dim -2 offset  -8
>>   (   5) dim -2 offset  -9
>>   (   6) dim -2 offset -10
>> Process 1:
>>   (   0) dim  1 offset   4
>>   (   1) dim  1 offset   5
>>   (   2) dim  1 offset   6
>>   (   3) dim  1 offset   7
>>   (   4) dim  1 offset   8
>>   (   5) dim  1 offset   9
>>
>> This is a global PETSc section that gives me the global numbering for the
>> owned points and (garbage?) negative values for ghost.
>>
>> Until here everything is good. But then I print the PetscSF that is
>> created by  'DMPlexDistribute'. This I do not understand:
>>
>> PetscSF Object: Migration SF 2 MPI processes
>>   type: basic
>>     sort=rank-order
>>   [0] Number of roots=10, leaves=7, remote ranks=1
>>   [0] 0 <- (0,0)
>>   [0] 1 <- (0,1)
>>   [0] 2 <- (0,3)
>>   [0] 3 <- (0,6)
>>   [0] 4 <- (0,7)
>>   [0] 5 <- (0,8)
>>   [0] 6 <- (0,9)
>>   [1] Number of roots=0, leaves=6, remote ranks=1
>>   [1] 0 <- (0,2)
>>   [1] 1 <- (0,4)
>>   [1] 2 <- (0,5)
>>   [1] 3 <- (0,7)
>>   [1] 4 <- (0,8)
>>   [1] 5 <- (0,9)
>>   [0] Roots referenced by my leaves, by rank
>>   [0] 0: 7 edges
>>   [0]    0 <- 0
>>   [0]    1 <- 1
>>   [0]    2 <- 3
>>   [0]    3 <- 6
>>   [0]    4 <- 7
>>   [0]    5 <- 8
>>   [0]    6 <- 9
>>   [1] Roots referenced by my leaves, by rank
>>   [1] 0: 6 edges
>>   [1]    0 <- 2
>>   [1]    1 <- 4
>>   [1]    2 <- 5
>>   [1]    3 <- 7
>>   [1]    4 <- 8
>>   [1]    5 <- 9
>>
>> I understand that SF is a data structure that saves references to pieces
>> of data that are now owned by the process (https://arxiv.org/pdf/1506.06
>> 194v1.pdf, page 4).
>>
>> Since the only ghost nodes appear in rank 0 (three ghost vertices) I
>> would expect something like:
>> *rank 0:*
>>   4 - (1, 3)  (to read: point 4 is owned by rank 1 and is rank's 1 point
>> 3)
>>   etc...
>> *rank 1:*
>>   nothing
>>
>> Is my intuition correct? If so, what does the star forest that I get from
>> DMPlexDistribute mean? I am printing the wrong thing?
>>
>> Thank you
>>
>> --
>> D. Adrian Maldonado, PhD Candidate
>> Electrical & Computer Engineering Dept.
>> Illinois Institute of Technology
>> 3301 S. Dearborn Street, Chicago, IL 60616
>>
>
>
>
> --
> D. Adrian Maldonado, PhD Candidate
> Electrical & Computer Engineering Dept.
> Illinois Institute of Technology
> 3301 S. Dearborn Street, Chicago, IL 60616
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160919/d9a1f93d/attachment.html>