<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 129.75pt 72.0pt 129.7pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">> -----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">> From: Karl Rupp [mailto:rupp@iue.tuwien.ac.at]<o:p></o:p></p>
<p class="MsoPlainText">> Sent: 10 August 2015 14:13<o:p></o:p></p>
<p class="MsoPlainText">> To: Leoni, Massimiliano<o:p></o:p></p>
<p class="MsoPlainText">> Cc: slepc-maint@upv.es; petsc-dev@mcs.anl.gov<o:p></o:p></p>
<p class="MsoPlainText">> Subject: Re: [petsc-dev] [GPU - slepc] Hands-on exercise 4 (SVD) not working<o:p></o:p></p>
<p class="MsoPlainText">> with GPU and default configurations<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> Maybe you forgot to call SlepcFinalize()?<o:p></o:p></p>
<p class="MsoPlainText">Unfortunately it's not it, if I omit SlepcFinalize() an error message shows up at runtime to remind me.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> Ok, this is actually a relatively GPU-friendly setup, because CPUs have<o:p></o:p></p>
<p class="MsoPlainText">> reduced the gap in terms of FLOPs quite a bit (see for example<o:p></o:p></p>
<p class="MsoPlainText">> http://www.karlrupp.net/2013/06/cpu-gpu-and-mic-hardware-<o:p></o:p></p>
<p class="MsoPlainText">> characteristics-over-time/ )<o:p></o:p></p>
<p class="MsoPlainText">Read, thanks for sharing!<o:p></o:p></p>
<p class="MsoPlainText"><o:p></o:p></p>
<p class="MsoPlainText">> I'd suggest to convince your supervisor into buying/using a cluster with<o:p></o:p></p>
<p class="MsoPlainText">> current hardware and enjoy a higher speedup compared to what you could<o:p></o:p></p>
<p class="MsoPlainText">> get in an ideal setting with a GPU from 2010 anyway ;-)<o:p></o:p></p>
<p class="MsoPlainText">This could partly be overcome as I was told I *might*, eventually, have access to a big cluster with many NVIDIA Tesla K20.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> (Having said that, I carefully estimate that you can get some<o:p></o:p></p>
<p class="MsoPlainText">> performance gains for SVD if you deep-dive into the existing SVD<o:p></o:p></p>
<p class="MsoPlainText">> implementation, carefully redesign it to minimize CPU<->GPU<o:p></o:p></p>
<p class="MsoPlainText">> communication, and use optimized library routines from the BLAS 3<o:p></o:p></p>
<p class="MsoPlainText">> operations. Currently there is not enough GPU-infrastructure in PETSc to<o:p></o:p></p>
<p class="MsoPlainText">> achieve this via command line parameters only.)<o:p></o:p></p>
<p class="MsoPlainText">Mmm, can you give a rough estimate of the effort involved in this?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">>From: Matthew Knepley [mailto:knepley@gmail.com] <o:p></o:p></p>
<p class="MsoPlainText">>Sent: 10 August 2015 14:28<o:p></o:p></p>
<p class="MsoPlainText">>To: Leoni, Massimiliano<o:p></o:p></p>
<p class="MsoPlainText">>Cc: Karl Rupp; slepc-maint@upv.es; petsc-dev@mcs.anl.gov<o:p></o:p></p>
<p class="MsoPlainText">>Subject: Re: [petsc-dev] [GPU - slepc] Hands-on exercise 4 (SVD) not working with GPU and default configurations<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">>Try calling PetscLogBegin() after PetscInitialize(). We have now put in an error if this is not initialized correctly.<o:p></o:p></p>
<p class="MsoPlainText">This didn’t do the trick, unfortunately <o:p></o:p></p>
<p class="MsoPlainText">Do I have to pull from the repo and rebuild?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">[In general, can I pull and rebuild without running configure again?]<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">>I agree with Karl that not much speedup can be expected with GPUs. This is the fault of dishonest marketing. None<o:p></o:p></p>
<p class="MsoPlainText">>of the computations in PETSc are limited by the computation rate, rather they are limited by memory bandwidth. The<o:p></o:p></p>
<p class="MsoPlainText">>bandwidth is at best 2-3x better, and less for modern CPUs. The dense SVD can be better than this, but you are<o:p></o:p></p>
<p class="MsoPlainText">>eventually limited by offload times and memory latency. The story of 100x, or even 10x, speedups is just a fraud.<o:p></o:p></p>
<p class="MsoPlainText">I remember reading this in one of the petsc reports [the “Preliminary evaluation” one?].<o:p></o:p></p>
<p class="MsoPlainText">I’ll see what I can do<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="color:black">Best regards,<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="color:black">Massimiliano<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<P><br>The data contained in, or attached to, this e-mail, may contain confidential information. If you have received it in error you should notify the sender immediately by reply e-mail, delete the message from your system and contact +44 (0) 1332 622800(Security Operations Centre) if you need assistance. Please do not copy it for any purpose, or disclose its contents to any other person.</P>
<P>An e-mail response to this address may be subject to interception or monitoring for operational reasons or for lawful business practices.</P>
<P>(c) 2015 Rolls-Royce plc</P><span lang=EN-GB>
<P dir=ltr align=left>Registered office: 62 Buckingham Gate, London SW1E 6AT Company number: 1003142. Registered in England.</span></P></body>
</html>