<div dir="ltr"><div>When I speak about totally inconsistent results between single and multi-GPU in my previous e-mail, I mean with PCASM. I don't see the same with PCJACOBI (even though, in this case, this is useless since it crashes after a few iterations), so I'm doubting about what could be the source of the problem. <br>
</div>When constructing the matrix for the multi-GPU case, I use this sequence of calls:<br><div><br>MatSetType(m_mat, MATMPIAIJCUSP);<br>MatSetBlockSize(m_mat, blockSize);<br>MatMPIAIJSetPreallocation(m_mat, dnz, dnnz, onz, onnz);<br>
<br></div><div>and my sparsity pattern is computed accordingly, considering that I have an AIJ matrix instead of the BAIJ I use on the "CPU version".<br></div><div><br><br><br><div><div><br><div class="gmail_quote">
---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Andrea Lani</b> <span dir="ltr"><<a href="mailto:andrea.lani@gmail.com">andrea.lani@gmail.com</a>></span><br>Date: Fri, Jan 17, 2014 at 9:47 PM<br>
Subject: Re: [petsc-dev] GPU preconditioners<br>To: Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br>Cc: For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>><br>
<br><br><div bgcolor="#FFFFFF"><div>Ok, thanks.</div><div><br></div><div>In fact, I have another major problem: when running on multi-GPU with PETSc my results are totally inconsistent compared to a single GPU .</div><div>
<br></div><div>In my code, for now, I'm assuming a 1-1 correspondence between CPU and GPU: I run on 8 cores and 8 GPUs (4 K10). How can I enforce this in the PETSc solver? Is it automatically done or do I have to specify some options?</div>
<div><br></div><div>Thanks again</div><span class=""><font color="#888888"><div><br></div><div>Andrea</div></font></span><div><div class="h5"><div><br></div><div><br></div><div><br></div><div><br><br>On Jan 17, 2014, at 8:48 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
<br></div><div></div><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jan 17, 2014 at 1:29 PM, Andrea Lani <span dir="ltr"><<a href="mailto:andrea.lani@gmail.com" target="_blank"></a><a href="mailto:andrea.lani@gmail.com" target="_blank">andrea.lani@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div>Dear Devs,<br></div><br>is the PCBJACOBI solver fully ported to GPU in the latest petsc-dev version? if not, is there any intention to do so in the near future? <br>
<br>I have a convection-dominated (with strong discontinuities in the flow field) MHD problem where PCASM and PCBJACOBI both work fine with KSPGMRES.</div></blockquote><div><br></div><div>These can be done on the GPU, but the key is the inner PC. Right now, we have no ILU0 or equivalent, which I think is what</div>
<div>you want. You can try the AMG variants, but I am guessing they would not be great for convection dominated flow.</div><div><br></div><div>Maybe Karl has a better suggestion?</div><div><br></div><div> Thanks,</div><div>
<br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Looking for a speedup in my code (which is using a petsc-dev version about two-months old), the only GPU alternative I found was PCJACOBI. This is indeed considerably faster on GPU, but does not converge (neither does, consistently, its CPU counterpart) if not for a few iterations before blowing up. Any other alternative for GPU-based preconditioners or solvers worth trying at the moment?<br>
<br></div><div>Thanks in advance for your advice<span><font color="#888888"><br><br></font></span></div><span><font color="#888888"><div>Andrea<br></div><div><br><br clear="all"></div><div><div>
<br>-- <br><span style="color:rgb(102,102,102)">Dr. Andrea Lani </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(102,102,102)">Senior Research Engineer, PhD</span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">Aeronautics & Aerospace dept., CFD group </span><span style="color:rgb(102,102,102)"><br>
Von Karman Institute for Fluid Dynamics </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">
Chausse de Waterloo 72, </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">B-1640, Rhode-Saint-Genese, Belgium</span><span style="color:rgb(102,102,102)"><br>fax : <a href="tel:%2B32-2-3599600" value="+3223599600" target="_blank">+32-2-3599600</a> </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(153,153,153)"><span style="color:rgb(102,102,102)">
work : <a href="tel:%2B32-2-3599769" value="+3223599769" target="_blank">+32-2-3599769</a> </span> <span></span></span><u style="background-color:rgb(51,51,255);color:rgb(153,153,153)"><br><span style="background-color:rgb(255,255,255);color:rgb(0,0,153)"><a href="mailto:lani@vki.ac.be" target="_blank"></a><a href="mailto:lani@vki.ac.be" target="_blank">lani@vki.ac.be</a></span></u>
</div></div></font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div>
</div></blockquote></div></div></div></div><br><br clear="all"><br>-- <br><span style="color:rgb(102,102,102)">Dr. Andrea Lani </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(102,102,102)">Senior Research Engineer, PhD</span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">Aeronautics & Aerospace dept., CFD group </span><span style="color:rgb(102,102,102)"><br>
Von Karman Institute for Fluid Dynamics </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">
Chausse de Waterloo 72, </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">B-1640, Rhode-Saint-Genese, Belgium</span><span style="color:rgb(102,102,102)"><br>fax : +32-2-3599600 </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(153,153,153)"><span style="color:rgb(102,102,102)">
work : +32-2-3599769 </span> <span style="background-color:rgb(255,255,255)"></span></span><u style="background-color:rgb(51,51,255);color:rgb(153,153,153)"><br><span style="background-color:rgb(255,255,255);color:rgb(0,0,153)"><a href="mailto:lani@vki.ac.be" target="_blank">lani@vki.ac.be</a></span></u>
</div></div></div></div>