[petsc-dev] Vertex assignment in DMPlexDistribute

Michael Lange michael.lange at imperial.ac.uk
Mon Nov 18 10:32:59 CST 2013


Brilliant, thanks a lot.

Michael

On 18/11/13 16:08, Matthew Knepley wrote:
> On Mon, Nov 18, 2013 at 9:47 AM, Matthew Knepley <knepley at gmail.com 
> <mailto:knepley at gmail.com>> wrote:
>
>     On Mon, Nov 18, 2013 at 3:24 AM, Michael Lange
>     <michael.lange at imperial.ac.uk
>     <mailto:michael.lange at imperial.ac.uk>> wrote:
>
>         Hi Matt,
>
>         I think there is a misunderstanding here. I am referring to
>         the case where DMPlexDistribute() is run with overlap=1 (which
>         is not the case in SNES ex12) and vertices in the overlap/halo
>         region are assigned to the wrong rank. This can lead to a case
>         where a proc may own a vertex that is not in its original
>         (non-overlapping) partition, although the attached cell is not
>         owned and will be marked as "ghost" by
>         DMPlexConstructGhostCells().
>
>
> Your fix is now merged to next:
>
> https://bitbucket.org/petsc/petsc/branch/knepley/fix-plex-partition-overlap
>
>     Matt
>
>         To illustrate this, I have attached an example consisting of a
>         unit square with 3 faces in each dimension and a section with
>         only vertex dofs. If run with two ranks, rank 1 will own all
>         its vertices (13 roots), whereas rank 0 only owns vertices not
>         in the overlap/halo of rank 1 (3 roots). My understanding is
>         that, since the original partition splits the square along its
>         diagonal, the vertex distribution should be 10 to 6 with the 4
>         diagonal vertices assigned to rank 1 and all other vertices
>         assigned according to the original partition. Is this correct,
>         or am I missing something here?
>
>
>     I have simplified the example so that I can easily do things in my
>     head. Now we just have 2 faces per side. Here is the run with
>     overlap = 0:
>
>     next *$:/PETSc3/petsc/petsc-dev$
>     /PETSc3/petsc/petsc-dev/arch-c-exodus-next/bin/mpiexec -host
>     localhost -n 2
>     /PETSc3/petsc/petsc-dev/arch-c-exodus-next/lib/plex_overlap-obj/plex_overlap
>     -dm_view -overlap 0
>     Parallel Mesh in 2 dimensions:
>       0-cells: 6 6
>       1-cells: 9 9
>       2-cells: 4 4
>     Labels:
>       depth: 3 strata of sizes (6, 9, 4)
>       exterior_facets: 1 strata of sizes (4)
>       marker: 2 strata of sizes (9, 3)
>     PetscSF Object: 2 MPI processes
>       type: basic
>         sort=rank-order
>       [0] Number of roots=19, leaves=5, remote ranks=1
>       [0] 4 <- (1,6)
>       [0] 5 <- (1,8)
>       [0] 7 <- (1,9)
>       [0] 10 <- (1,13)
>       [0] 11 <- (1,17)
>       [1] Number of roots=19, leaves=0, remote ranks=0
>
>     Each partition gets 4 cells and 6 vertices since it is split along
>     the diagonal. The overlap
>     region contains the 3 vertices and 2 faces that lie on the
>     diagonal, and they are all owned by proc 1.
>     Now if we run with an overlap of 1:
>
>     next *$:/PETSc3/petsc/petsc-dev$
>     /PETSc3/petsc/petsc-dev/arch-c-exodus-next/bin/mpiexec -host
>     localhost -n 2
>     /PETSc3/petsc/petsc-dev/arch-c-exodus-next/lib/plex_overlap-obj/plex_overlap
>     -dm_view -overlap 1
>     Parallel Mesh in 2 dimensions:
>       0-cells: 8 8
>       1-cells: 13 13
>       2-cells: 6 6
>     Labels:
>       depth: 3 strata of sizes (8, 13, 6)
>       exterior_facets: 1 strata of sizes (6)
>       marker: 2 strata of sizes (13, 5)
>     PetscSF Object: 2 MPI processes
>       type: basic
>         sort=rank-order
>       [0] Number of roots=27, leaves=19, remote ranks=1
>       [0] 0 <- (1,1)
>       [0] 2 <- (1,4)
>       [0] 6 <- (1,7)
>       [0] 7 <- (1,8)
>       [0] 8 <- (1,9)
>       [0] 9 <- (1,10)
>       [0] 10 <- (1,11)
>       [0] 11 <- (1,12)
>       [0] 12 <- (1,13)
>       [0] 14 <- (1,17)
>       [0] 15 <- (1,18)
>       [0] 16 <- (1,19)
>       [0] 17 <- (1,20)
>       [0] 18 <- (1,21)
>       [0] 19 <- (1,22)
>       [0] 20 <- (1,23)
>       [0] 21 <- (1,24)
>       [0] 25 <- (1,25)
>       [0] 26 <- (1,26)
>       [1] Number of roots=27, leaves=2, remote ranks=1
>       [1] 3 <- (0,1)
>       [1] 5 <- (0,5)
>
>     Each process gets 2 more cells (those who faces lie on the
>     diagonal), 2 more vertices and 4 more edges. This is correct. The
>     two overlap cells are ghost for proc 1, but the 4 edges and 2
>     vertices are owned. So you are correct, I need to mark all those
>     overlap points as "unownable" by the original process.
>
>       Thanks for finding this,
>
>           Matt
>
>         Many thanks for all your help
>         Michael
>
>         On 16/11/13 13:54, Matthew Knepley wrote:
>>         On Sat, Nov 16, 2013 at 7:22 AM, Michael Lange
>>         <michael.lange at imperial.ac.uk
>>         <mailto:michael.lange at imperial.ac.uk>> wrote:
>>
>>             Hi,
>>
>>             I notice that, when creating the point SF for the
>>             parallel partition in DMPlexDistribute, cells are
>>             assigned to procs according to the original partition but
>>             vertices aren't. Was this done by design or is this a bug?
>>
>>
>>         If this were true, there would be no communication for the P1
>>         test of SNES ex12. Here is running it with
>>         -interpolate 1 and -dm_view ::ascii_info_detail
>>
>>         PetscSF Object: 2 MPI processes
>>           type: basic
>>             sort=rank-order
>>           [0] Number of roots=19, leaves=5, remote ranks=1
>>           [0] 4 <- (1,6)
>>           [0] 5 <- (1,8)
>>           [0] 7 <- (1,9)
>>           [0] 10 <- (1,13)
>>           [0] 11 <- (1,17)
>>           [1] Number of roots=19, leaves=0, remote ranks=0
>>           [0] Roots referenced by my leaves, by rank
>>           [0] 1: 5 edges
>>           [0]    4 <- 6
>>           [0]    5 <- 8
>>           [0]    7 <- 9
>>           [0]    10 <- 13
>>           [0]    11 <- 17
>>           [1] Roots referenced by my leaves, by rank
>>
>>         So there are 3 vertices and 2 edges in the point SF.
>>
>>            Matt
>>
>>             In case it is a bug, I have attached a patch that fixes
>>             this by using the closure of the original partition instead.
>>
>>             Thanks and kind regards
>>             Michael
>>
>>
>>
>>
>>         -- 
>>         What most experimenters take for granted before they begin
>>         their experiments is infinitely more interesting than any
>>         results to which their experiments lead.
>>         -- Norbert Wiener
>
>
>
>
>     -- 
>     What most experimenters take for granted before they begin their
>     experiments is infinitely more interesting than any results to
>     which their experiments lead.
>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20131118/89c6d98b/attachment.html>


More information about the petsc-dev mailing list