<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Darn, of course! This works!</p>
<p>My sincere apologies and many thanks for the swift reply!</p>
<p>Fabian<br>
</p>
<div class="moz-cite-prefix">On 05.06.20 14:04, Dave May wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJ98EDpGoxXK+Q=HczLhJ9rfSsqii2M4GUBsWG=SOUYJRzf6yQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_quote">
<div dir="ltr">On Fri 5. Jun 2020 at 13:52, Fabian Jakub
<<a href="mailto:Fabian.Jakub@physik.uni-muenchen.de"
target="_blank" moz-do-not-send="true">Fabian.Jakub@physik.uni-muenchen.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>Dear Petsc list,</p>
<p>I have a question regarding reverse vec-scatters:</p>
<p>I have a shared memory solver that I want to use on a
distributed DMDA and average its results.</p>
<p>The shared mem solver needs some of the global state.</p>
<p>So I want to create a full copy of a global vec on
each master rank of a machine, compute some result</p>
<p>and average the results back into the global vec.<br>
</p>
<p><br>
</p>
<p>For example: the parallel layout for a global vector
is<br>
</p>
<p>3 ranks in comm_world, each with Ndof = 6, so global
size is 18<br>
</p>
<p>Then I create a mpi sub communicator with
MPI_COMM_TYPE_SHARED</p>
<p>Then I create a local vec on each rank, with sizes 9
on each rank_0 on the sub comm, size 0 otherwise</p>
<p> if ( sub_id == 0 ) then<br>
call VecGetSize(gvec, Nlocal, ierr); call
CHKERR(ierr)<br>
else<br>
Nlocal = 0<br>
endif<br>
call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec,
ierr)<br>
<br>
</p>
<p>This yields for example (first 2 ranks in a shared
mem subcomm, 3rd rank on another machine):</p>
<p>Global rank_id | sub_rank_id | local_size_gvec
| local_size_lvec<br>
</p>
<p>0 | 0 |
6 | 18<br>
</p>
<p>1 | 1 |
6 | 0</p>
<p>2 | 0 |
6 | 18<br>
</p>
<p><br>
</p>
<p>To copy the global vec on each shared mem master, I
do:</p>
<p>ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is,
ierr)</p>
<p>VecScatterCreate(gvec, is, lvec, is, ctx, ierr)</p>
<p>VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES,
SCATTER_FORWARD, ierr)</p>
<p>VecScatterEnd (ctx, gvec, lvec, INSERT_VALUES,
SCATTER_FORWARD, ierr)</p>
<p><br>
</p>
<p>That works fine.</p>
<p>Then I want to do the reverse, i.e. add all values
from the local vec to the global vec on comm_world to
generate the average of the results.</p>
<p>I tried:</p>
<p>VecSet(gvec, zero, ierr)</p>
<p>VecScatterBegin(ctx, gvec, lvec, ADD_VALUES,
SCATTER_REVERSE, ierr)</p>
<p><br>
</p>
<p>I was hoping to get the sum of svec in the global
vec, so that gvec / comm_size(sub_comm_id==0) gives
the mean.</p>
<p>However, I get the following error:</p>
<p><br>
</p>
<p>Nonconforming object sizes</p>
<p>Vector wrong size 18 for scatter 6 (scatter reverse
and vector to != ctx from size)</p>
<p><br>
</p>
<p>Going with the same approach with <a
href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html#VecScatterCreateToAll"
target="_blank" moz-do-not-send="true">VecScatterCreateToAll</a>
leads to the same issue.<br>
</p>
<p>Do you have suggestions on how I could/should achieve
it?</p>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div class="gmail_quote">
<div dir="auto">You have to flip the lvec, gvec args in
reverse mode (as per the man pages states under Notes)</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks</div>
<div dir="auto">Dave</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p><br>
</p>
<p>Many thanks!</p>
</div>
<div>
<p>Fabian<br>
</p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</body>
</html>