[petsc-dev] [petsc-maint #88993] Petsc with Cuda 4.0 and Multiple GPUs

[Dave Nystrom]

Hi Barry,

Barry Smith writes:
 > Dave,
 > 
 > I cannot explain why it does not use the MatMult_SeqAIJCusp() - it does for me.

Do you get good performance running a problem like ex2?

 > Have you updated to the latest cusp/thrust? From the mecurial repositories?

I did try the latest version of cusp from mercurial initially but the build
failed.  So I am currently using the latest cusp tarball.  I did not try the
latest version of thrust but instead was just using what came with the
released version of Cuda 4.0.  I could try the mercurial versions of both.

 > There is a difference, in your new 4.0 build you added
 > --download-txpetscgpu=yes BTW: that doesn't work for me with the latest
 > cusp and thrust from the mecurial repositories can you try reconfiguring
 > and making without that?

Yes, I can try that.  Maybe that is why my original build with cusp from
mercurial failed.

Thanks for your help,

Dave

 > Barry
 > 
 > On Oct 2, 2011, at 9:08 PM, Dave Nystrom wrote:
 > 
 >> Barry Smith writes:
 >>> On Oct 2, 2011, at 6:39 PM, Dave Nystrom wrote:
 >>> 
 >>>> Thanks for the update.  I don't believe I have gotten a run with good
 >>>> performance yet, either from C or Fortran.  I wish there was an easy way for
 >>>> me to force use of only one of my gpus.  I don't want to have to pull one of
 >>>> the gpus in order to see if that is complicating things with Cuda 4.0.  I'll
 >>>> be eager to hear if you make any progress on figuring things out.
 >>>> 
 >>>> Do you understand yet why the petsc ex2.c example is able to parse the
 >>>> "-cuda_show_devices" argument but ex2f.F does not?
 >>> 
 >>> Matt put the code in the wrong place in PETSc, that is all, no big
 >>> existentialist reason. I will fix that.
 >> 
 >> Thanks.  I'll look forward to testing out the new version.
 >> 
 >> Dave
 >> 
 >>> Barry
 >>> 
 >>>> 
 >>>> Thanks,
 >>>> 
 >>>> Dave
 >>>> 
 >>>> Barry Smith writes:
 >>>>> It is not doing the MatMult operation on the GPU and hence needs to move
 >>>>> the vectors back and forth for each operation (since MatMult is done on
 >>>>> the CPU with the vector while vector operations are done on the GPU) hence
 >>>>> the terrible performance.
 >>>>> 
 >>>>> Not sure why yet. It is copying the Mat down for me from C.
 >>>>> 
 >>>>> Barry
 >>>>> 
 >>>>> On Oct 2, 2011, at 4:18 PM, Dave Nystrom wrote:
 >>>>> 
 >>>>>> In case it might be useful, I have attached two log files of runs with the
 >>>>>> ex2f petsc example from src/ksp/ksp/examples/tutorials.  One was run back in
 >>>>>> April with petsc-dev linked to Cuda 3.2.  It shows excellent runtime
 >>>>>> performance.  The other was run today with petsc-dev checked out of the
 >>>>>> mercurial repo yesterday morning and linked to Cuda 4.0.  In addition to the
 >>>>>> differences in run time performance, I also do not see an entry for
 >>>>>> MatCUSPCopyTo in the profiling section.  I'm not sure what the significance
 >>>>>> of that is.  I do observe that the run time for PCApply is about the same for
 >>>>>> the two cases.  I think I would expect that to be the case even if the
 >>>>>> problem were partitioned across two gpus.  However, it does make me wonder if
 >>>>>> the absence of MatCUSPCopyTo in the profiling section of the Cuda 4.0 log
 >>>>>> file is an indication that the matrix was not actually copied to the gpu.
 >>>>>> I'm not sure yet how to check for that.  Hope this might be useful.
 >>>>>> 
 >>>>>> Thanks,
 >>>>>> 
 >>>>>> Dave
 >>>>>> 
 >>>>>> 
 >>>>>> <ex2f_3200_3200_cuda_yes_cuda_3.2.log><ex2f_3200_3200_cuda_yes_cuda_4.0.log>
 >>>>>> Dave Nystrom writes:
 >>>>>>> Matthew Knepley writes:
 >>>>>>>> On Sat, Oct 1, 2011 at 11:26 PM, Dave Nystrom <Dave.Nystrom at tachyonlogic.com> wrote:
 >>>>>>>>> Barry Smith writes:
 >>>>>>>>>> On Oct 1, 2011, at 9:22 PM, Dave Nystrom wrote:
 >>>>>>>>>>> Hi Barry,
 >>>>>>>>>>> 
 >>>>>>>>>>> I've sent a couple more emails on this topic.  What I am trying to do at the
 >>>>>>>>>>> moment is to figure out how to have a problem run on only one gpu if it will
 >>>>>>>>>>> fit in the memory of that gpu.  Back in April when I had built petsc-dev with
 >>>>>>>>>>> Cuda 3.2, petsc would only use one gpu if you had multiple gpus on your
 >>>>>>>>>>> machine.  In order to use multiple gpus for a problem, one had to use
 >>>>>>>>>>> multiple threads with a separate thread assigned to control each gpu.  But
 >>>>>>>>>>> Cuda 4.0 has, I believe, made that transparent and under the hood.  So now
 >>>>>>>>>>> when I run a small example problem such as
 >>>>>>>>>>> src/ksp/ksp/examples/tutorials/ex2f.F with an 800x800 problem, it gets
 >>>>>>>>>>> partitioned to run on both of the gpus in my machine.  The result is a very
 >>>>>>>>>>> large performance hit because of communication back and forth from one gpu to
 >>>>>>>>>>> the other via the cpu.
 >>>>>>>>>> 
 >>>>>>>>>> How do you know there is lots of communication from the GPU to the CPU? In
 >>>>>>>>>> the -log_summary? Nope because PETSc does not manage anything like that
 >>>>>>>>>> (that is one CPU process using both GPUs).
 >>>>>>>>> 
 >>>>>>>>> What I believe is that it is being managed by Cuda 4.0, not by petsc.
 >>>>>>>>> 
 >>>>>>>>>>> So this problem with a 3200x3200 grid runs 5x slower
 >>>>>>>>>>> now than it did with Cuda 3.2.  I believe if one is programming down at the
 >>>>>>>>>>> cuda level, it is possible to have a smaller problem run on only one gpu so
 >>>>>>>>>>> that there is communication only between the cpu and gpu and only at the
 >>>>>>>>>>> start and end of the calculation.
 >>>>>>>>>>> 
 >>>>>>>>>>> To me, it seems like what is needed is a petsc option to specify the number
 >>>>>>>>>>> of gpus to run on that can somehow get passed down to the cuda level through
 >>>>>>>>>>> cusp and thrust.  I fear that the short term solution is going to have to be
 >>>>>>>>>>> for me to pull one of the gpus out of my desktop system but it would be nice
 >>>>>>>>>>> if there was a way to tell petsc and friends to just use one gpu when I want
 >>>>>>>>>>> it to.
 >>>>>>>>>>> 
 >>>>>>>>>>> If necessary, I can send a couple of log files to demonstrate what I am
 >>>>>>>>>>> trying to describe regarding the performance hit.
 >>>>>>>>>> 
 >>>>>>>>>> I am not convinced that the poor performance you are getting now has
 >>>>>>>>>> anything to do with using both GPUs. Please run a PETSc program with the
 >>>>>>>>>> command -cuda_show_devices
 >>>>>>>>> 
 >>>>>>>>> I ran the following command:
 >>>>>>>>> 
 >>>>>>>>> ex2f -m 8 -n 8 -ksp_type cg -pc_type jacobi -log_summary -cuda_show_devices
 >>>>>>>>> -mat_type aijcusp -vec_type cusp -options_left
 >>>>>>>>> 
 >>>>>>>>> The result was a report that there was one option left, that being
 >>>>>>>>> -cuda_show_devices.  I am using a copy of petsc-dev that I cloned and built
 >>>>>>>>> this morning.
 >>>>>>>> 
 >>>>>>>> What do you have at src/sys/object/pinit.c:825? You should see the code
 >>>>>>>> that processes this option. You should be able to break there in the
 >>>>>>>> debugger and see what happens. This sounds again like you are not
 >>>>>>>> processing options correctly.
 >>>>>>> 
 >>>>>>> Hi Matt,
 >>>>>>> 
 >>>>>>> I'll take a look at that in a bit and see if I can figure out what is going
 >>>>>>> on.  I do see the code that you mention that processes the arguments that
 >>>>>>> Barry mentioned.  In terms of processing options correctly, at least in this
 >>>>>>> case I am actually running one of the petsc examples rather than my own
 >>>>>>> code.  And it seems to correctly process the other command line arguments.
 >>>>>>> Anyway, I'll write more after I have had a chance to investigate more.
 >>>>>>> 
 >>>>>>> Thanks,
 >>>>>>> 
 >>>>>>> Dave
 >>>>>>> 
 >>>>>>>> Matt
 >>>>>>>> 
 >>>>>>>>>> What are the choices?  You can then pick one of them and run with
 >>>>>>>>> -cuda_set_device integer
 >>>>>>>>> 
 >>>>>>>>> The -cuda_set_device option does not appear to be recognized either, even
 >>>>>>>>> if I choose an integer like 0.
 >>>>>>>>> 
 >>>>>>>>>> Does this change things?
 >>>>>>>>> 
 >>>>>>>>> I suspect it would change things if I could get it to work.
 >>>>>>>>> 
 >>>>>>>>> Thanks,
 >>>>>>>>> 
 >>>>>>>>> Dave
 >>>>>>>>> 
 >>>>>>>>>> Barry
 >>>>>>>>>> 
 >>>>>>>>>>> 
 >>>>>>>>>>> Thanks,
 >>>>>>>>>>> 
 >>>>>>>>>>> Dave
 >>>>>>>>>>> 
 >>>>>>>>>>> Barry Smith writes:
 >>>>>>>>>>>> Dave,
 >>>>>>>>>>>> 
 >>>>>>>>>>>> We have no mechanism in the PETSc code for a PETSc single CPU process to
 >>>>>>>>>>>> use two GPUs at the same time. However you could have two MPI processes
 >>>>>>>>>>>> each using their own GPU.
 >>>>>>>>>>>> 
 >>>>>>>>>>>> The one tricky part is you need to make sure each MPI process uses a
 >>>>>>>>>>>> different GPU. We currently do not have a mechanism to do this assignment
 >>>>>>>>>>>> automatically. I think it can be done with cudaSetDevice(). But I don't
 >>>>>>>>>>>> know the details, sending this to petsc-dev at mcs.anl.gov where more people
 >>>>>>>>>>>> may know.
 >>>>>>>>>>>> 
 >>>>>>>>>>>> PETSc-folks,
 >>>>>>>>>>>> 
 >>>>>>>>>>>> We need a way to have this setup automatically.
 >>>>>>>>>>>> 
 >>>>>>>>>>>> Barry
 >>>>>>>>>>>> 
 >>>>>>>>>>>> On Oct 1, 2011, at 5:43 PM, Dave Nystrom wrote:
 >>>>>>>>>>>> 
 >>>>>>>>>>>>> I'm running petsc on a machine with Cuda 4.0 and 2 gpus.  This is a desktop
 >>>>>>>>>>>>> machine with a single processor.  I know that Cuda 4.0 has support for
 >>>>>>>>>>>>> running on multiple gpus but don't know if petsc uses that.  But suppose I
 >>>>>>>>>>>>> have a problem that will fit in the memory for a single gpu.  Will petsc run
 >>>>>>>>>>>>> the problem on a single gpu or does it split it between the 2 gpus and incur
 >>>>>>>>>>>>> the communication overhead of copying data between the two gpus?
 >>>>>>>>>>>>> 
 >>>>>>>>>>>>> Thanks,
 >>>>>>>>>>>>> 
 >>>>>>>>>>>>> Dave
 >>>>>>>> 
 >>>>>>>> -- 
 >>>>>>>> What most experimenters take for granted before they begin their experiments
 >>>>>>>> is infinitely more interesting than any results to which their experiments
 >>>>>>>> lead.
 >>>>>>>> -- Norbert Wiener
 >>>>> 
 >>> 
 >