[petsc-users] MemCpy (HtoD and DtoH) in Krylov solver

[Smith, Barry F.]

   1) What preconditioner are you using? If any.

   2) Where/how are you getting this information about the MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize this same sort of information to plan future optimizations. 

   3) Are you using more than 1 MPI rank?

  If you use the master branch (which we highly recommend for anyone using GPUs and PETSc) the -log_view option will log communication between CPU and GPU and display it in the summary table. This is useful for seeing exactly what operations are doing vector communication between the CPU/GPU.

  We welcome all feedback on the GPUs since it previously has only been lightly used.

   Barry


> On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello everyone,
> 
> I am new to petsc gpu and have a simple question. 
> 
> When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are VECSEQCUDA  with GMRES(or GCR) and pcnone, I found that during each krylov iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and the solve still needs some work from CPU? What are these MemCpys for during the each iteration?
> 
> Thank you.
> 
> Best,
> Xiangdong