[petsc-users] Question on reverse scatters from VecScatterCreateToAll

[Fabian Jakub]

Darn, of course! This works!

My sincere apologies and many thanks for the swift reply!

Fabian

On 05.06.20 14:04, Dave May wrote:
>
>
> On Fri 5. Jun 2020 at 13:52, Fabian Jakub
> <Fabian.Jakub at physik.uni-muenchen.de
> <mailto:Fabian.Jakub at physik.uni-muenchen.de>> wrote:
>
>     Dear Petsc list,
>
>     I have a question regarding reverse vec-scatters:
>
>     I have a shared memory solver that I want to use on a distributed
>     DMDA and average its results.
>
>     The shared mem solver needs some of the global state.
>
>     So I want to create a full copy of a global vec on each master
>     rank of a machine, compute some result
>
>     and average the results back into the global vec.
>
>
>     For example: the parallel layout for a global vector is
>
>     3 ranks in comm_world, each with Ndof = 6, so global size is 18
>
>     Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED
>
>     Then I create a local vec on each rank, with sizes 9 on each
>     rank_0 on the sub comm, size 0 otherwise
>
>           if ( sub_id == 0 ) then
>             call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr)
>           else
>             Nlocal = 0
>           endif
>           call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr)
>
>     This yields for example (first 2 ranks in a shared mem subcomm,
>     3rd rank on another machine):
>
>     Global rank_id  |   sub_rank_id   |   local_size_gvec | 
>     local_size_lvec
>
>     0                       |  0                    |  
>     6                        | 18
>
>     1                       |  1                    |  
>     6                        | 0
>
>     2                       |  0                    |  
>     6                        | 18
>
>
>     To copy the global vec on each shared mem master, I do:
>
>     ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr)
>
>     VecScatterCreate(gvec, is, lvec, is, ctx, ierr)
>
>     VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr)
>
>     VecScatterEnd  (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr)
>
>
>     That works fine.
>
>     Then I want to do the reverse, i.e. add all values from the local
>     vec to the global vec on comm_world to generate the average of the
>     results.
>
>     I tried:
>
>     VecSet(gvec, zero, ierr)
>
>     VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr)
>
>
>     I was hoping to get the sum of svec in the global vec, so that
>     gvec / comm_size(sub_comm_id==0) gives the mean.
>
>     However, I get the following error:
>
>
>     Nonconforming object sizes
>
>     Vector wrong size 18 for scatter 6 (scatter reverse and vector to
>     != ctx from size)
>
>
>     Going with the same approach with VecScatterCreateToAll
>     <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html#VecScatterCreateToAll>
>     leads to the same issue.
>
>     Do you have suggestions on how I could/should achieve it?
>
>
>
> You have to flip the lvec, gvec args in reverse mode (as per the man
> pages states under Notes)
>
> Thanks
> Dave
>
>
>
>     Many thanks!
>
>     Fabian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200605/66bfc08e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200605/66bfc08e/attachment.sig>