[petsc-dev] Understanding Vecscatter with Kokkos Vecs

[Fande Kong]

On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang at gmail.com>
wrote:

> VecScatter (i.e., SF, the two are the same thing) setup (building various
> index lists, rank lists) is done on the CPU.  is1, is2 must be host data.
>

Just out of curiosity, is1 and is2 can not be created on a GPU device in
the first place? That being said, it is technically impossible? Or we just
did not implement them yet?

Fande,


> When the SF is used to communicate device data, indices are copied to the
> device..
>
> --Junchao Zhang
>
>
> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan at gmail.com>
> wrote:
>
>> I'm trying to understand how VecScatters work with GPU-native Kokkos
>> Vecs.
>>
>> Specifically, I'm interested in what will happen in code like in
>> src/vec/vec/tests/ex22.c,
>>
>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr);
>>
>> (from
>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44
>> )
>>
>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the
>> command line. But is1 and is2 are (I think), always
>> CPU/host data. Assuming that the scatter itself can happen on the GPU,
>> the indices must make it to the device somehow - are they copied there when
>> the scatter is created? Is there a way to create the scatter using indices
>> already on the GPU (Maybe using SF more directly)?
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210218/d9eca87a/attachment.html>