[petsc-dev] Understanding Vecscatter with Kokkos Vecs

[Junchao Zhang]

On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd at gmail.com> wrote:

>
>
> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang at gmail.com>
> wrote:
>
>> VecScatter (i.e., SF, the two are the same thing) setup (building various
>> index lists, rank lists) is done on the CPU.  is1, is2 must be host data.
>>
>
> Just out of curiosity, is1 and is2 can not be created on a GPU device in
> the first place? That being said, it is technically impossible? Or we just
> did not implement them yet?
>
Simply because we do not have an ISCUDA class.


>
> Fande,
>
>
>> When the SF is used to communicate device data, indices are copied to the
>> device..
>>
>> --Junchao Zhang
>>
>>
>> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan at gmail.com>
>> wrote:
>>
>>> I'm trying to understand how VecScatters work with GPU-native Kokkos
>>> Vecs.
>>>
>>> Specifically, I'm interested in what will happen in code like in
>>> src/vec/vec/tests/ex22.c,
>>>
>>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr);
>>>
>>> (from
>>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44
>>> )
>>>
>>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the
>>> command line. But is1 and is2 are (I think), always
>>> CPU/host data. Assuming that the scatter itself can happen on the GPU,
>>> the indices must make it to the device somehow - are they copied there when
>>> the scatter is created? Is there a way to create the scatter using indices
>>> already on the GPU (Maybe using SF more directly)?
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210218/2767dde0/attachment.html>