[petsc-users] MemCpy (HtoD and DtoH) in Krylov solver

[Matthew Knepley]

On Wed, Jul 17, 2019 at 3:33 PM Xiangdong <epscodes at gmail.com> wrote:

> Thanks for your explanation.
>
> Is it correct to say that ONLY the VecNorm, VecDot, VecMDot, MatMult
> operations in GMRES are done on GPU and everything else is done on CPU?
>

You are missing VecAXPY(). If instead you list what is not done, I think it
is only the duiagonalization of H by Givens rotations, but
this would be latency dominated on a GPU and probably slower than the CPU.

  Thanks,

     Matt


> Thanks.
>
> Xiangdong
>
> On Wed, Jul 17, 2019 at 7:34 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Also, MPI communication is done from the host, so every mat-vec will do a
>> "CopySome" call from the device, do MPI comms, and then the next time you
>> do GPU work it will copy from the host to get the update.
>>
>> On Tue, Jul 16, 2019 at 10:22 PM Matthew Knepley via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> On Tue, Jul 16, 2019 at 9:07 PM Xiangdong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I am new to petsc gpu and have a simple question.
>>>>
>>>> When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are
>>>> VECSEQCUDA  with GMRES(or GCR) and pcnone, I found that during each krylov
>>>> iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does
>>>> that mean the Krylov solve is not 100% on GPU and the solve still needs
>>>> some work from CPU? What are these MemCpys for during the each iteration?
>>>>
>>>
>>> We have GPU experts on the list, but there is definitely a communication
>>> because we do not do orthogonalization on the GPU,
>>> just the BLAS ops. This is a very small amount of data, so it just
>>> contributed latency, and I would guess that it is less then kernel
>>> launch latency.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Thank you.
>>>>
>>>> Best,
>>>> Xiangdong
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190717/335d7321/attachment.html>