<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 129.75pt 72.0pt 129.7pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:314527523;
mso-list-type:hybrid;
mso-list-template-ids:-1976421762 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">Good, it is running now, but performances are really poor: I tried on 3 nodes with 2 GPUs and 12 CPU threads each, MPI+CUDA performs much worse than pure MPI.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I have a few thoughts on why this might be happening:<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>My problem has dense matrices but on GPU I use –mat_type aijcusp<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>I did something wrong in setting the problem up, that causes a bunch of useless memory traffic I could avoid<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>The real power of GPUs might show up only eventually as the problem gets bigger [I tested a 100000x200 matrix]<o:p></o:p></p>
<p class="MsoPlainText">Is any of these likely to be causing this poor performance?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">There is also the banner on PETSc’s website saying that using GPUs effectively is not that easy. Can anyone point me to further reference on how to improve this?<o:p></o:p></p>
<p class="MsoPlainText">I suspect that I can achieve more by choosing proper solution methods and other options of PETSc/SLEPc.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Any hint highly appreciated.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Massimiliano<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">-----Original Message-----</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">From: Jose E. Roman [mailto:jroman@dsic.upv.es]</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">Sent: 07 August 2015 18:44</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">To: Leoni, Massimiliano</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">Cc: petsc-dev@mcs.anl.gov; slepc-maint@upv.es</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">Subject: Re: [petsc-dev] [GPU - slepc] Hands-on exercise 4 (SVD) not working</span></p>
<p class="MsoPlainText">> <span lang="EN-US" style="mso-fareast-language:EN-GB">with GPU and default configurations</span></p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> Yes, there seems to be a problem with the default SVD solver (SVDCROSS). I</p>
<p class="MsoPlainText">> will fix it in the master branch in the next days. Meanwhile, you can run the</p>
<p class="MsoPlainText">> example with -svd_type trlanczos</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> Thanks for reporting this.</p>
<p class="MsoPlainText">> Jose</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> > El 7/8/2015, a las 16:31, Leoni, Massimiliano <<a href="mailto:Massimiliano.Leoni@Rolls-Royce.com"><span style="color:windowtext;text-decoration:none">Massimiliano.Leoni@Rolls-</span></a></p>
<p class="MsoPlainText"><a href="mailto:Massimiliano.Leoni@Rolls-Royce.com"><span style="color:windowtext;text-decoration:none">> Royce.com</span></a>> escribió:</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Hi everybody!</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > I kept experimenting with slepc and GPUs, and when I turned to SVD I</p>
<p class="MsoPlainText">> found out that the hands-on exercise on SVD [#4] doesn’t run properly.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > If I run it on CPU it works fine, whereas… $ mpirun -np 1 slepcSVD</p>
<p class="MsoPlainText">> > -file $SLEPC_DIR/share/slepc/datafiles/matrices/rdb200.petsc -bv_type</p>
<p class="MsoPlainText">> > vecs -mat_type aijcusp -on_error_abort</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Singular value problem stored in file.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Reading REAL matrix from a binary file...</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: BVScaleColumn() line 380 in</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/sys/classes/bv/interface/bvop</p>
<p class="MsoPlainText">> > s.c Scalar value must be same on all processes, argument # 3 [0]PETSC</p>
<p class="MsoPlainText">> > ERROR:</p>
<p class="MsoPlainText">> > ----------------------------------------------------------------------</p>
<p class="MsoPlainText">> > -- [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation</p>
<p class="MsoPlainText">> > Violation, probably memory access out of range [0]PETSC ERROR: Try</p>
<p class="MsoPlainText">> > option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR:</p>
<p class="MsoPlainText">> > or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">
<span style="color:windowtext;text-decoration:none">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</span></a></p>
<p class="MsoPlainText">> > [0]PETSC ERROR: or try <a href="http://valgrind.org">
<span style="color:windowtext;text-decoration:none">http://valgrind.org</span></a> on GNU/linux and Apple Mac</p>
<p class="MsoPlainText">> > OS X to find memory corruption errors [0]PETSC ERROR: likely location</p>
<p class="MsoPlainText">> > of problem given in stack below [0]PETSC ERROR: ---------------------</p>
<p class="MsoPlainText">> > Stack Frames ------------------------------------</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not</p>
<p class="MsoPlainText">> available,</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: INSTEAD the line number of the start of the function</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: is given.</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] PetscAbortErrorHandler line 56</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/petsc/src/sys/error/errabort.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] PetscError line 363</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/petsc/src/sys/error/err.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] BVScaleColumn line 377</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/sys/classes/bv/interface/bvop</p>
<p class="MsoPlainText">> > s.c [0]PETSC ERROR: [0] EPSFullLanczos line 357</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/eps/impls/krylov/epskrylov.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Symm line 41</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/eps/impls/krylov/krylovschur/</p>
<p class="MsoPlainText">> > ks-symm.c [0]PETSC ERROR: [0] EPSSolve line 83</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/eps/interface/epssolve.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] SVDSolve_Cross line 155</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/svd/impls/cross/cross.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: [0] SVDSolve line 92</p>
<p class="MsoPlainText">> > /gpfs/rrcfd/rruk-students/apps/slepc/src/svd/interface/svdsolve.c</p>
<p class="MsoPlainText">> > [0]PETSC ERROR: User provided function() line 0 in unknown file</p>
<p class="MsoPlainText">> > (null)</p>
<p class="MsoPlainText">> > ----------------------------------------------------------------------</p>
<p class="MsoPlainText">> > ---- mpirun noticed that process rank 0 with PID 11264 on node gpu3</p>
<p class="MsoPlainText">> > exited on signal 11 (Segmentation fault).</p>
<p class="MsoPlainText">> > ----------------------------------------------------------------------</p>
<p class="MsoPlainText">> > ----</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > I am using the same command line options that work just fine on hands-on</p>
<p class="MsoPlainText">> exercises 1 and 2, which feature EPS solvers.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Any hint appreciated.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Best regards,</p>
<p class="MsoPlainText">> > Massimiliano Leoni</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > The data contained in, or attached to, this e-mail, may contain confidential</p>
<p class="MsoPlainText">> information. If you have received it in error you should notify the sender</p>
<p class="MsoPlainText">> immediately by reply e-mail, delete the message from your system and</p>
<p class="MsoPlainText">> contact +44 (0) 1332 622800(Security Operations Centre) if you need</p>
<p class="MsoPlainText">> assistance. Please do not copy it for any purpose, or disclose its contents to</p>
<p class="MsoPlainText">> any other person.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > An e-mail response to this address may be subject to interception or</p>
<p class="MsoPlainText">> monitoring for operational reasons or for lawful business practices.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > (c) 2015 Rolls-Royce plc</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Registered office: 62 Buckingham Gate, London SW1E 6AT Company</p>
<p class="MsoPlainText">> number: 1003142. Registered in England.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
<P><br>The data contained in, or attached to, this e-mail, may contain confidential information. If you have received it in error you should notify the sender immediately by reply e-mail, delete the message from your system and contact +44 (0) 1332 622800(Security Operations Centre) if you need assistance. Please do not copy it for any purpose, or disclose its contents to any other person.</P>
<P>An e-mail response to this address may be subject to interception or monitoring for operational reasons or for lawful business practices.</P>
<P>(c) 2015 Rolls-Royce plc</P><span lang=EN-GB>
<P dir=ltr align=left>Registered office: 62 Buckingham Gate, London SW1E 6AT Company number: 1003142. Registered in England.</span></P></body>
</html>