On Sat, Oct 1, 2011 at 11:26 PM, Dave Nystrom <span dir="ltr"><<a href="mailto:Dave.Nystrom@tachyonlogic.com">Dave.Nystrom@tachyonlogic.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">Barry Smith writes:<br>
><br>
> On Oct 1, 2011, at 9:22 PM, Dave Nystrom wrote:<br>
><br>
> > Hi Barry,<br>
> ><br>
> > I've sent a couple more emails on this topic. What I am trying to do at the<br>
> > moment is to figure out how to have a problem run on only one gpu if it will<br>
> > fit in the memory of that gpu. Back in April when I had built petsc-dev with<br>
> > Cuda 3.2, petsc would only use one gpu if you had multiple gpus on your<br>
> > machine. In order to use multiple gpus for a problem, one had to use<br>
> > multiple threads with a separate thread assigned to control each gpu. But<br>
> > Cuda 4.0 has, I believe, made that transparent and under the hood. So now<br>
> > when I run a small example problem such as<br>
> > src/ksp/ksp/examples/tutorials/ex2f.F with an 800x800 problem, it gets<br>
> > partitioned to run on both of the gpus in my machine. The result is a very<br>
> > large performance hit because of communication back and forth from one gpu to<br>
> > the other via the cpu.<br>
><br>
> How do you know there is lots of communication from the GPU to the CPU? In<br>
> the -log_summary? Nope because PETSc does not manage anything like that<br>
> (that is one CPU process using both GPUs).<br>
<br>
</div>What I believe is that it is being managed by Cuda 4.0, not by petsc.<br>
<div class="im"><br>
> > So this problem with a 3200x3200 grid runs 5x slower<br>
> > now than it did with Cuda 3.2. I believe if one is programming down at the<br>
> > cuda level, it is possible to have a smaller problem run on only one gpu so<br>
> > that there is communication only between the cpu and gpu and only at the<br>
> > start and end of the calculation.<br>
> ><br>
> > To me, it seems like what is needed is a petsc option to specify the number<br>
> > of gpus to run on that can somehow get passed down to the cuda level through<br>
> > cusp and thrust. I fear that the short term solution is going to have to be<br>
> > for me to pull one of the gpus out of my desktop system but it would be nice<br>
> > if there was a way to tell petsc and friends to just use one gpu when I want<br>
> > it to.<br>
> ><br>
> > If necessary, I can send a couple of log files to demonstrate what I am<br>
> > trying to describe regarding the performance hit.<br>
><br>
> I am not convinced that the poor performance you are getting now has<br>
> anything to do with using both GPUs. Please run a PETSc program with the<br>
> command -cuda_show_devices<br>
<br>
</div>I ran the following command:<br>
<br>
ex2f -m 8 -n 8 -ksp_type cg -pc_type jacobi -log_summary -cuda_show_devices -mat_type aijcusp -vec_type cusp -options_left<br>
<br>
The result was a report that there was one option left, that being<br>
-cuda_show_devices. I am using a copy of petsc-dev that I cloned and built<br>
this morning.</blockquote><div><br></div><div>What do you have at src/sys/object/pinit.c:825? You should see the code that processes this option. You</div><div>should be able to break there in the debugger and see what happens. This sounds again like you are</div>
<div>not processing options correctly.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">
> What are the choices? You can then pick one of them and run with -cuda_set_device integer<br>
<br>
</div>The -cuda_set_device option does not appear to be recognized either, even if<br>
I choose an integer like 0.<br>
<br>
> Does this change things?<br>
<br>
I suspect it would change things if I could get it to work.<br>
<br>
Thanks,<br>
<font color="#888888"><br>
Dave<br>
</font><div><div></div><div class="h5"><br>
> Barry<br>
><br>
> ><br>
> > Thanks,<br>
> ><br>
> > Dave<br>
> ><br>
> > Barry Smith writes:<br>
> >> Dave,<br>
> >><br>
> >> We have no mechanism in the PETSc code for a PETSc single CPU process to<br>
> >> use two GPUs at the same time. However you could have two MPI processes<br>
> >> each using their own GPU.<br>
> >><br>
> >> The one tricky part is you need to make sure each MPI process uses a<br>
> >> different GPU. We currently do not have a mechanism to do this assignment<br>
> >> automatically. I think it can be done with cudaSetDevice(). But I don't<br>
> >> know the details, sending this to <a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a> where more people<br>
> >> may know.<br>
> >><br>
> >> PETSc-folks,<br>
> >><br>
> >> We need a way to have this setup automatically.<br>
> >><br>
> >> Barry<br>
> >><br>
> >> On Oct 1, 2011, at 5:43 PM, Dave Nystrom wrote:<br>
> >><br>
> >>> I'm running petsc on a machine with Cuda 4.0 and 2 gpus. This is a desktop<br>
> >>> machine with a single processor. I know that Cuda 4.0 has support for<br>
> >>> running on multiple gpus but don't know if petsc uses that. But suppose I<br>
> >>> have a problem that will fit in the memory for a single gpu. Will petsc run<br>
> >>> the problem on a single gpu or does it split it between the 2 gpus and incur<br>
> >>> the communication overhead of copying data between the two gpus?<br>
> >>><br>
> >>> Thanks,<br>
> >>><br>
> >>> Dave<br>
> >>><br>
> >><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>