<div dir="ltr">Yes, nvprof can give the size of the data as well as the amount of time for data movement. See the attached snapshots.<div><br></div><div>I can understand some of the numbers, but not the HtoD case.</div><div><br></div><div>In DtoH1, it is the data movement from VecMDot. The size of data is 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = 8*128*8 = 8192. My question is: instead of calling cublasDdot nv times, why do you implement your own kernels? I guess it must be for performance, but can you explain a little more?</div><div><br></div><div>In DtoH2, it is the data movement from VecNorm. The size of data is 8B, which is just the sizeof(PetscScalar).</div><div><br></div><div>In DtoD1, it is the data movement from VecAXPY. The size of data is 17.952MB, which is exactly sizeof(PetscScalar)*length(b).</div><div><br></div><div>However, I do not understand the number in HostToDevice in gmres for np=1. The size of data movement is 1.032KB. I thought this is related to the updated upper Hessenberg matrix, but the number does not match. Can anyone help me understand the data movement of HToD in GMRES for np=1?</div><div><br></div><div>Thank you.</div><div><br></div><div>Best,</div><div>Xiangdong</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at">rupp@iue.tuwien.ac.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
as you can see from the screenshot, the communication is merely for <br>
scalars from the dot-products and/or norms. These are needed on the host <br>
for the control flow and convergence checks and is true for any <br>
iterative solver.<br>
<br>
Best regards,<br>
Karli<br>
<br>
<br>
<br>
On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote:<br>
> <br>
> <br>
> On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a> <br>
> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>> wrote:<br>
> <br>
> <br>
> 1) What preconditioner are you using? If any.<br>
> <br>
> Currently I am using none as I want to understand how gmres works on GPU.<br>
> <br>
> <br>
> 2) Where/how are you getting this information about the<br>
> MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize<br>
> this same sort of information to plan future optimizations.<br>
> <br>
> I am using nvprof and nvvp from cuda toolkit. It looks like there are <br>
> one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 <br>
> case. See the attached snapshots.<br>
> <br>
> 3) Are you using more than 1 MPI rank?<br>
> <br>
> <br>
> I tried both np=1 and np=2. Attached please find snapshots from nvvp for <br>
> both np=1 and np=2 cases. The figures showing gpu calls with two pure <br>
> gmres iterations.<br>
> <br>
> Thanks.<br>
> Xiangdong<br>
> <br>
> <br>
> If you use the master branch (which we highly recommend for<br>
> anyone using GPUs and PETSc) the -log_view option will log<br>
> communication between CPU and GPU and display it in the summary<br>
> table. This is useful for seeing exactly what operations are doing<br>
> vector communication between the CPU/GPU.<br>
> <br>
> We welcome all feedback on the GPUs since it previously has only<br>
> been lightly used.<br>
> <br>
> Barry<br>
> <br>
> <br>
> > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users<br>
> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>> wrote:<br>
> ><br>
> > Hello everyone,<br>
> ><br>
> > I am new to petsc gpu and have a simple question.<br>
> ><br>
> > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x<br>
> are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during<br>
> each krylov iteration, there are one call MemCpy(HtoD) and one call<br>
> MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and<br>
> the solve still needs some work from CPU? What are these MemCpys for<br>
> during the each iteration?<br>
> ><br>
> > Thank you.<br>
> ><br>
> > Best,<br>
> > Xiangdong<br>
> <br>
</blockquote></div>