[petsc-users] [EXTERNAL] Re: DMPlex/PetscSF How to determine if local topology is other rank's ghost?

[Toby Isaac]

>
> //======================================
> PetscSF Object: 2 MPI processes
>   type: basic
>   [0] Number of roots=29, leaves=5, remote ranks=1
>   [0] 9 <- (1,9)
>   [0] 11 <- (1,10)
>   [0] 12 <- (1,13)
>   [0] 20 <- (1,20)
>   [0] 27 <- (1,27)
>   [1] Number of roots=29, leaves=2, remote ranks=1
>   [1] 14 <- (0,13)
>   [1] 19 <- (0,18)
>   MultiSF sort=rank-order
> ## Reduce Leafdata
> [0] 0: 2 2 2 0 0
> [1] 0: 3 0
> ## Reduce Rootdata
> [0] 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 -686563120 0 0 0 0 0 0
> [0] 20: 0 0 0 0 0 0 0 0 0
> [1] 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0
> [1] 20: -527386800 0 0 0 0 0 0 32610 0

This sf is one where the leaves are numbered as though they are
sparsely drawn from a larger vector.

For example, `[0] 9 <- (1,9)` means that the leaf with local index 9
on rank 0 has a root at (rank 1, index 9); the next leaf is `[0] 11 <-
(1,10)`, meaning it has local index 11 and its root is at (rank 1,
index 10).
So `PetscSFReduceBegin()` is expecting to read the `leafdata` on rank
0 from indices 9, 11, ....

But you have given it a `leafdata` array that is just a contiguous
array that's the size of the number of leaves.

The index spaces for leaves and roots don't have to be the same, but
in the case of the point SF they always are.  You should make a change
like so:

```
--- PetscCalloc2(nleaves,&leafdata,nroots,&rootdata);
+++ PetscCalloc2(nroots,&leafdata,nroots,&rootdata);
```