<div dir="ltr">Thanks, Barry and Karli! You comments are very helpful. At this point, I just started learning how to load the linear solve to GPU. Leave a copy of matrix on CPU is fine at the moment for me.<div><br></div><div>If the GPUs I am going to work on are Nvidia GPUs, can AIJVIENNACL and AIJCUSPARSE provide very similar performance based on your experience?</div><div><br></div><div>Thank you. <br><div><br></div><div>Xiangdong</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 23, 2019 at 6:50 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> On Jul 23, 2019, at 12:24 PM, Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>> wrote:<br>
> <br>
> Hi,<br>
> <br>
>> I have two quick questions related to run gpu solvers.<br>
>> 1) # of MPI processes vs # of GPUs. Is it true that we should set these two numbers equal if most of computations are done on GPU? For one case I tested, with only one GPU, running with np=2 is 15% slower than np=1 (probably due to mpi communications). I am curious in what case, one will benefit by having more mpi processes than number of GPUs.<br>
> <br>
> yes, usually you want to have the same number of MPI processes as the number of GPUs on each node, unless you have a lot of work for the CPU and only little work for the GPU. If you oversubscribe the GPU, it will not get any faster (unlike multi-core CPUs); instead, the extra management overhead will make it slow down.<br>
> <br>
> <br>
>> 2) When I use MatLoad(A,viewer) to load a binary format data into a aijcusparse format A, how many matrices are created? Does it involve creating a intermediate aij matrix A' on CPU then convert it to aijcuspare A on GPU? I am not sure whether such intermediate aij matrix exist. If yes, What is the life time of this matrix? Is it destroyed right after the converting?<br>
> <br>
> GPU matrices are essentially AIJ matrices with additional GPU data members. That is, if you MatLoad(), the data will be copied into the CPU buffers first and then pushed down to the GPU when needed. The CPU buffers will never be free'd, but might be updated from the latest GPU data to allow for fallback operations that have no GPU implementation.<br>
<br>
Eventually we should provide a mechanism that allows users to free the CPU side of matrices (and vectors) if the user knows that they will never need them again on the CPU. Honestly we are still working on the basics, not yet at this level of optimization. If you would benefit from such a feature now feel free to make a pull request. <br>
<br>
Barry<br>
<br>
<br>
<br>
> <br>
> Best regards,<br>
> Karli<br>
> <br>
> <br>
> <br>
> <br>
> <br>
>> On Fri, Jul 19, 2019 at 12:08 PM Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a> <mailto:<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>>> wrote:<br>
>> Hi Xiangdong,<br>
>> > I can understand some of the numbers, but not the HtoD case.<br>
>> ><br>
>> > In DtoH1, it is the data movement from VecMDot. The size of data is<br>
>> > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 =<br>
>> 8*128*8<br>
>> > = 8192. My question is: instead of calling cublasDdot nv times,<br>
>> why do<br>
>> > you implement your own kernels? I guess it must be for<br>
>> performance, but<br>
>> > can you explain a little more?<br>
>> Yes, this is a performance optimization. We've used several<br>
>> dot-products<br>
>> (suffers from kernel launch latency) as well as matrix-vector-products<br>
>> (suffers extra matrix setup) in the past; in both cases, there was<br>
>> extra<br>
>> memory traffic, thus impacting performance.<br>
>> The reason why the data size is 8192 is to get around a separate<br>
>> reduction stage on the GPU (i.e. a second kernel launch). By moving the<br>
>> data to the CPU and doing the reduction there, one is faster than doing<br>
>> it on the GPU and then moving only a few numbers. This has to do with<br>
>> PCI-Express latency: It takes about the same time to send a single byte<br>
>> as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes<br>
>> the<br>
>> limiting factor.<br>
>> > In DtoH2, it is the data movement from VecNorm. The size of data<br>
>> is 8B,<br>
>> > which is just the sizeof(PetscScalar).<br>
>> This is most likely the result required for the control flow on the CPU.<br>
>> > In DtoD1, it is the data movement from VecAXPY. The size of data is<br>
>> > 17.952MB, which is exactly sizeof(PetscScalar)*length(b).<br>
>> This is a vector assignment. If I remember correctly, it uses the<br>
>> memcpy-routines and hence shows up as a separate DtoD instead of just a<br>
>> kernel. It matches the time required for scal_kernel_val (scaling a<br>
>> vector by a scalar), so it runs at full bandwidth on the GPU.<br>
>> > However, I do not understand the number in HostToDevice in gmres for<br>
>> > np=1. The size of data movement is 1.032KB. I thought this is<br>
>> related to<br>
>> > the updated upper Hessenberg matrix, but the number does not<br>
>> match. Can<br>
>> > anyone help me understand the data movement of HToD in GMRES for<br>
>> np=1?<br>
>> 1032 = (128+1)*8, so this might be some auxiliary work information on<br>
>> the GPU. I could figure out the exact source of these transfers, but<br>
>> that is some effort. Let me know whether this is important information<br>
>> for you, then I can do it.<br>
>> Best regards,<br>
>> Karli<br>
>> ><br>
>> > Thank you.<br>
>> ><br>
>> > Best,<br>
>> > Xiangdong<br>
>> ><br>
>> > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a><br>
>> <mailto:<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>><br>
>> > <mailto:<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a> <mailto:<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>>>> wrote:<br>
>> ><br>
>> > Hi,<br>
>> ><br>
>> > as you can see from the screenshot, the communication is<br>
>> merely for<br>
>> > scalars from the dot-products and/or norms. These are needed<br>
>> on the<br>
>> > host<br>
>> > for the control flow and convergence checks and is true for any<br>
>> > iterative solver.<br>
>> ><br>
>> > Best regards,<br>
>> > Karli<br>
>> ><br>
>> ><br>
>> ><br>
>> > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote:<br>
>> > ><br>
>> > ><br>
>> > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F.<br>
>> > <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>><br>
>> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>><br>
>> > > <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>><br>
>> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a> <mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>>>> wrote:<br>
>> > ><br>
>> > ><br>
>> > > 1) What preconditioner are you using? If any.<br>
>> > ><br>
>> > > Currently I am using none as I want to understand how<br>
>> gmres works<br>
>> > on GPU.<br>
>> > ><br>
>> > ><br>
>> > > 2) Where/how are you getting this information<br>
>> about the<br>
>> > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like<br>
>> to utilize<br>
>> > > this same sort of information to plan future<br>
>> optimizations.<br>
>> > ><br>
>> > > I am using nvprof and nvvp from cuda toolkit. It looks<br>
>> like there<br>
>> > are<br>
>> > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per<br>
>> iteration for np=1<br>
>> > > case. See the attached snapshots.<br>
>> > ><br>
>> > > 3) Are you using more than 1 MPI rank?<br>
>> > ><br>
>> > ><br>
>> > > I tried both np=1 and np=2. Attached please find snapshots<br>
>> from<br>
>> > nvvp for<br>
>> > > both np=1 and np=2 cases. The figures showing gpu calls<br>
>> with two<br>
>> > pure<br>
>> > > gmres iterations.<br>
>> > ><br>
>> > > Thanks.<br>
>> > > Xiangdong<br>
>> > ><br>
>> > ><br>
>> > > If you use the master branch (which we highly<br>
>> recommend for<br>
>> > > anyone using GPUs and PETSc) the -log_view option will log<br>
>> > > communication between CPU and GPU and display it in<br>
>> the summary<br>
>> > > table. This is useful for seeing exactly what<br>
>> operations are<br>
>> > doing<br>
>> > > vector communication between the CPU/GPU.<br>
>> > ><br>
>> > > We welcome all feedback on the GPUs since it previously<br>
>> > has only<br>
>> > > been lightly used.<br>
>> > ><br>
>> > > Barry<br>
>> > ><br>
>> > ><br>
>> > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users<br>
>> > > <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>> > <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>> > wrote:<br>
>> > > ><br>
>> > > > Hello everyone,<br>
>> > > ><br>
>> > > > I am new to petsc gpu and have a simple question.<br>
>> > > ><br>
>> > > > When I tried to solve Ax=b where A is<br>
>> MATAIJCUSPARSE and b<br>
>> > and x<br>
>> > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found<br>
>> that<br>
>> > during<br>
>> > > each krylov iteration, there are one call MemCpy(HtoD) and<br>
>> > one call<br>
>> > > MemCpy(DtoH). Does that mean the Krylov solve is not<br>
>> 100% on<br>
>> > GPU and<br>
>> > > the solve still needs some work from CPU? What are these<br>
>> > MemCpys for<br>
>> > > during the each iteration?<br>
>> > > ><br>
>> > > > Thank you.<br>
>> > > ><br>
>> > > > Best,<br>
>> > > > Xiangdong<br>
>> > ><br>
>> ><br>
<br>
</blockquote></div>