[petsc-users] Question on reverse scatters from VecScatterCreateToAll

[Fabian Jakub]

Dear Petsc list,

I have a question regarding reverse vec-scatters:

I have a shared memory solver that I want to use on a distributed DMDA
and average its results.

The shared mem solver needs some of the global state.

So I want to create a full copy of a global vec on each master rank of a
machine, compute some result

and average the results back into the global vec.


For example: the parallel layout for a global vector is

3 ranks in comm_world, each with Ndof = 6, so global size is 18

Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED

Then I create a local vec on each rank, with sizes 9 on each rank_0 on
the sub comm, size 0 otherwise

      if ( sub_id == 0 ) then
        call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr)
      else
        Nlocal = 0
      endif
      call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr)

This yields for example (first 2 ranks in a shared mem subcomm, 3rd rank
on another machine):

Global rank_id  |   sub_rank_id   |   local_size_gvec |  local_size_lvec

0                       |  0                    |  
6                        | 18

1                       |  1                    |  
6                        | 0

2                       |  0                    |  
6                        | 18


To copy the global vec on each shared mem master, I do:

ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr)

VecScatterCreate(gvec, is, lvec, is, ctx, ierr)

VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr)

VecScatterEnd  (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr)


That works fine.

Then I want to do the reverse, i.e. add all values from the local vec to
the global vec on comm_world to generate the average of the results.

I tried:

VecSet(gvec, zero, ierr)

VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr)


I was hoping to get the sum of svec in the global vec, so that gvec /
comm_size(sub_comm_id==0) gives the mean.

However, I get the following error:


Nonconforming object sizes

Vector wrong size 18 for scatter 6 (scatter reverse and vector to != ctx
from size)


Going with the same approach with VecScatterCreateToAll
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html#VecScatterCreateToAll>
leads to the same issue.

Do you have suggestions on how I could/should achieve it?


Many thanks!

Fabian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200605/5d3c0871/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200605/5d3c0871/attachment.sig>