[petsc-dev] Understanding Vecscatter with Kokkos Vecs

[Junchao Zhang]

VecScatter (i.e., SF, the two are the same thing) setup (building various
index lists, rank lists) is done on the CPU.  is1, is2 must be host data.
When the SF is used to communicate device data, indices are copied to the
device..

--Junchao Zhang


On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan at gmail.com>
wrote:

> I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs.
>
> Specifically, I'm interested in what will happen in code like in
> src/vec/vec/tests/ex22.c,
>
> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr);
>
> (from
> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44)
>
> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the
> command line. But is1 and is2 are (I think), always
> CPU/host data. Assuming that the scatter itself can happen on the GPU, the
> indices must make it to the device somehow - are they copied there when the
> scatter is created? Is there a way to create the scatter using indices
> already on the GPU (Maybe using SF more directly)?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210218/68293550/attachment.html>