[petsc-users] Scattering a vector to/from a subset of processors

[Junchao Zhang]

I think your approach is correct.  Do you have an example code?

--Junchao Zhang


On Tue, Dec 5, 2023 at 5:15 PM Sreeram R Venkat <srvenkat at utexas.edu> wrote:

> Hi, I have a follow up question on this.
>
> Now, I'm trying to do a scatter and permutation of the vector. Under the
> same setup as the original example, here are the new Start and Finish
> states I want to achieve:
>  Start                                Finish
> Proc | Entries                 Proc | Entries
>     0   |  0,...,8                     0   | 0, 12, 24
>     1   |  9,...,17                   1   | 1, 13, 25
>     2   |  18,...,26                 2   | 2, 14, 26
>     3   |  27,...,35                 3   | 3, 15, 27
>     4   |  None                     4   | 4, 16, 28
>     5   |  None                     5   | 5, 17, 29
>     6   |  None                     6   | 6, 18, 30
>     7   |  None                     7   | 7, 19, 31
>     8   |  None                     8   | 8, 20, 32
>     9   |  None                     9   | 9, 21, 33
>     10   |  None                   10 | 10, 22, 34
>     11   |  None                   11  | 11, 23, 35
>
> So far, I've tried to use ISCreateGeneral(), with each process giving an
> idx array corresponding to the indices it wants (i.e. idx on P0 looks like
> [0,12,24] P1 -> [1,13, 25], and so on).
> Then I use this to create the VecScatter with VecScatterCreate(x, is, y,
> NULL, &scatter).
>
> However, when I try to do the scatter, I get some illegal memory access
> errors.
>
> Is there something wrong with how I define the index sets?
>
> Thanks,
> Sreeram
>
>
>
>
>
> On Thu, Oct 5, 2023 at 12:57 PM Sreeram R Venkat <srvenkat at utexas.edu>
> wrote:
>
>> Thank you. This works for me.
>>
>> Sreeram
>>
>> On Wed, Oct 4, 2023 at 6:41 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Hi, Sreeram,
>>> You can try this code. Since x, y are both MPI vectors, we just need to
>>> say we want to scatter x[0:N] to y[0:N]. The 12 index sets with your
>>> example on the 12 processes would be [0..8], [9..17], [18..26], [27..35],
>>> [], ..., [].  Actually, you can do it arbitrarily, say, with 12 index sets
>>> [0..17], [18..35], .., [].  PETSc will figure out how to do the
>>> communication.
>>>
>>> PetscInt rstart, rend, N;
>>> IS ix;
>>> VecScatter vscat;
>>> Vec y;
>>> MPI_Comm comm;
>>> VecType type;
>>>
>>> PetscObjectGetComm((PetscObject)x, &comm);
>>> VecGetType(x, &type);
>>> VecGetSize(x, &N);
>>> VecGetOwnershipRange(x, &rstart, &rend);
>>>
>>> VecCreate(comm, &y);
>>> VecSetSizes(y, PETSC_DECIDE, N);
>>> VecSetType(y, type);
>>>
>>> ISCreateStride(PetscObjectComm((PetscObject)x), rend - rstart, rstart, 1,
>>> &ix);
>>> VecScatterCreate(x, ix, y, ix, &vscat);
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Wed, Oct 4, 2023 at 6:03 PM Sreeram R Venkat <srvenkat at utexas.edu>
>>> wrote:
>>>
>>>> Suppose I am running on 12 processors, and I have a vector "v" of size
>>>> 36 partitioned over the first 4. v still uses the PETSC_COMM_WORLD, so it
>>>> has a layout of (9, 9, 9, 9, 0, 0, ..., 0). Now, I would like to
>>>> repartition it over all 12 processors, so that the layout becomes (3, 3, 3,
>>>> ..., 3). I've been trying to use VecScatter to do this, but I'm not sure
>>>> what IndexSets to use for the sender and receiver.
>>>>
>>>> The result I am trying to achieve is this:
>>>>
>>>> Assume the vector is v = <0, 1, 2, ..., 35>
>>>>
>>>>      Start                                Finish
>>>> Proc | Entries                 Proc | Entries
>>>>     0   |  0,...,8                     0   | 0, 1, 2
>>>>     1   |  9,...,17                   1   | 3, 4, 5
>>>>     2   |  18,...,26                 2   | 6, 7, 8
>>>>     3   |  27,...,35                 3   | 9, 10, 11
>>>>     4   |  None                     4   | 12, 13, 14
>>>>     5   |  None                     5   | 15, 16, 17
>>>>     6   |  None                     6   | 18, 19, 20
>>>>     7   |  None                     7   | 21, 22, 23
>>>>     8   |  None                     8   | 24, 25, 26
>>>>     9   |  None                     9   | 27, 28, 29
>>>>     10   |  None                   10 | 30, 31, 32
>>>>     11   |  None                   11  | 33, 34, 35
>>>>
>>>> Appreciate any help you can provide on this.
>>>>
>>>> Thanks,
>>>> Sreeram
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231205/ad97b138/attachment-0001.html>