<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, May 29, 2017 at 11:19 AM, Xinzhe Wu <span dir="ltr"><<a href="mailto:xinzhe.wu1990@gmail.com" target="_blank">xinzhe.wu1990@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear all,<div><br></div><div>We have developed the codes with PETSc + SLEPc which works well on CPU version. Now we want to try these codes with GPU + MPI, but get some weird errors shown as below. </div><div><br></div><div>I have found someone talked about this problem here <a href="http://lists.mcs.anl.gov/pipermail/petsc-dev/2016-March/018836.html" target="_blank">http://lists.mcs.anl.gov/<wbr>pipermail/petsc-dev/2016-<wbr>March/018836.html</a> , but I can hardly understand it. Can anyone help me with these issues?</div></div></blockquote><div><br></div><div>The answer is here:</div><div><br></div><div><pre style="white-space:pre-wrap;color:rgb(0,0,0)">>>>><i> I think the error messages you get is pretty descriptive regarding the root cause. You are probably running out of GPU memory. Since you are running on a GTX 285 you can't use MPS [1] therefore each MPI process has its own context on the GPU. Each context needs to initialize some data on the GPU (used for local variables and so on). The required amount needed for this depends on the size of the GPUs (essentially correlates with the maximum number of concurrently active threads). This can easily be 50-100MB. So with only 1GB of GPU memory you are probably using all GPUs memory for context data and nothing is available for your application. Unfortunately there is no good way to debug this with GeForce. On Tesla nvidia-smi does show you all processes that have a context on a GPU together with their memory consumption.</i></pre></div><div>It appears that you are running out of GPU memory. This can happen if you use too many</div><div>MPI processes for a single GPU.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thank you in advance!<br><div><br></div><div><br></div><div><div><font color="#ff0000">[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</font></div><div><font color="#ff0000">[0]PETSC ERROR: Error in external library</font></div><div><font color="#ff0000">[0]PETSC ERROR: CUBLAS error 1</font></div><div><font color="#ff0000">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.</font></div><div><font color="#ff0000">[0]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</font></div><div><font color="#ff0000">[2]PETSC ERROR: Error in external library</font></div><div><font color="#ff0000">[2]PETSC ERROR: CUBLAS error 1</font></div><div><font color="#ff0000">[2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.</font></div><div><font color="#ff0000">[2]PETSC ERROR: Petsc Development GIT revision: v3.7.6-3965-gf375733 GIT Date: 2017-05-28 10:32:02 -0500</font></div><div><font color="#ff0000">[2]PETSC ERROR: ./hyperh on a arch-linux2-c-debug named romeo44 by xinzhewu Mon May 29 18:03:58 2017</font></div><div><font color="#ff0000">[2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-visibility=0 --with-shared-libraries=0 --with-cuda=1 --with-thrust=1 --with-precision=double --with-clanguage=c --with-pestc-arch=linux-c-no-<wbr>debug-complex --with-scalar-type=complex</font></div><div><font color="#ff0000">[2]PETSC ERROR: #1 PetscInitialize() line 906 in /home/xinzhewu/Petsc-GPUs/<wbr>petsc/src/sys/objects/pinit.c</font></div><div><font color="#ff0000">[2]PETSC ERROR: #2 SlepcInitialize() line 259 in /home/xinzhewu/Petsc-GPUs/<wbr>slepc/src/sys/slepcinit.c</font></div><span class="gmail-HOEnZb"><font color="#888888"><div><font color="#ff0000"><br></font></div><div><font color="#ff0000"><br></font></div>-- <br><div class="gmail-m_5237013412536882204gmail_signature"><div dir="ltr"><div>Xinzhe WU<br>Ph.D Student of Computer Science<br>Maison de la Simulation, CNRS USR3441<br>Building 565, CEA Saclay<br>91191, Gif-sur-Yvette, France<br>Tel: <a href="tel:+33%201%2069%2008%2059%2093" value="+33169085993" target="_blank">+33 (0) 1 69 08 59 93</a></div></div></div>
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">http://www.caam.rice.edu/~mk51/</a><br></div></div></div>
</div></div>