[petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process

[Junchao Zhang]

VecScatterCreateToAll() scatters the MPI vector to a sequential vector on
every rank (as if each rank has a duplicate of the same sequential vector).
If the sample code you provided is what you want, it is fine and we just
need  to implement a minor optimization in petsc to make it efficient.  But
if you want to put the scatter in a loop as follows,  then it is a very bad
code.

for (p=0; p<size; i++) {

  loc_tgt_size = 0;
  if (rank == p){
    loc_tgt_size = n;
  }
  ierr = VecCreateSeq(PETSC_COMM_SELF, loc_tgt_size, &tgt_vec); CHKERRQ(ierr);
  ierr = VecZeroEntries(tgt_vec); CHKERRQ(ierr);
  // Scatter source vector to target vector on one process
  ierr = ISCreateStride(PETSC_COMM_SELF, loc_tgt_size, 0, 1, &is);
CHKERRQ(ierr);
  ierr = VecScatterCreate(src_vec, is, tgt_vec, is, &sctx); CHKERRQ(ierr);
  ierr = VecScatterBegin(sctx, src_vec, tgt_vec, INSERT_VALUES,
SCATTER_FORWARD); CHKERRQ(ierr);
  ierr = VecScatterEnd(sctx, src_vec, tgt_vec, INSERT_VALUES,
SCATTER_FORWARD); CHKERRQ(ierr);

   ...
}

--Junchao Zhang


On Sat, Jul 3, 2021 at 1:27 PM Peder Jørgensgaard Olesen <pjool at mek.dtu.dk>
wrote:

> Yeah, scattering a parallel vector to a sequential on one rank was exactly
> what I wanted to do (apologies if I didn't phrase that clearly). A code
> like the one I shared does just what I needed, replacing size-1 with the
> desired target rank in the if-statement.
>
>
> Isn't what you describe what VecScatterCreateToAll is for?
>
>
>
> Med venlig hilsen / Best regards
>
> Peder
> ------------------------------
> *Fra:* Junchao Zhang <junchao.zhang at gmail.com>
> *Sendt:* 3. juli 2021 04:42:48
> *Til:* Peder Jørgensgaard Olesen
> *Cc:* Jed Brown; petsc-users at mcs.anl.gov
> *Emne:* Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on
> non-zeroth process
>
> Peder,
>   Your example scatters a parallel vector to a sequential vector on one
> rank.  It is a pattern like MPI_Gatherv.
>   I want to see how you scatter parallel vectors to sequential vectors on
> every rank.
>
> --Junchao Zhang
>
>
> On Fri, Jul 2, 2021 at 4:07 AM Peder Jørgensgaard Olesen <pjool at mek.dtu.dk>
> wrote:
>
>> Matt's method seems to work well, though instead of editing the actual
>> function I put the relevant parts directly into my code. I made the small
>> example attached here.
>>
>>
>> I might look into Star Forests at some point, though it's not really
>> touched upon in the manual (I will probably take a look at your paper,
>> https://arxiv.org/abs/2102.13018).
>>
>>
>> Med venlig hilsen / Best regards
>>
>> Peder
>> ------------------------------
>> *Fra:* Junchao Zhang <junchao.zhang at gmail.com>
>> *Sendt:* 1. juli 2021 16:38:29
>> *Til:* Jed Brown
>> *Cc:* Peder Jørgensgaard Olesen; petsc-users at mcs.anl.gov
>> *Emne:* Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on
>> non-zeroth process
>>
>> Peder,
>>   PETSCSF_PATTERN_ALLTOALL only supports MPI_Alltoall (not Alltoallv),
>> and is only used by petsc internally at few places.
>>   I suggest you can go with Matt's approach. After it solves your
>> problem, you can distill an example to demo the communication pattern. Then
>> we can see how to efficiently support that in petsc.
>>
>>   Thanks.
>> --Junchao Zhang
>>
>>
>> On Thu, Jul 1, 2021 at 7:42 AM Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Peder Jørgensgaard Olesen <pjool at mek.dtu.dk> writes:
>>>
>>> > Each process is assigned an indexed subset of the tasks (the tasks are
>>> of constant size), and, for each task index, the relevant data is scattered
>>> as a SEQVEC to the process (this is done for all processes in each step,
>>> using an adaption of the code in Matt's link). This way each process only
>>> receives just the data it needs to complete the task. While I'm currently
>>> working with very moderate size data sets I'll eventually need to handle
>>> something rather more massive, so I want to economize memory where possible
>>> and give each process only the data it needs.
>>>
>>> From the sounds of it, this pattern ultimately boils down to MPI_Gather
>>> being called P times where P is the size of the communicator. This will
>>> work okay when P is small, but it's much less efficient than calling
>>> MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF
>>> that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You
>>> can see an example.
>>>
>>>
>>> https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210703/278348d7/attachment-0001.html>