<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1" bgcolor="#ffffff">
<div style="direction: ltr;font-family: Arial;color: #000000;font-size: 14pt;">Please share how that goes. I'd love to hear how it goes for you.<br>
<br>
Dave<br>
<div><br>
</div>
<div style="font-family: Times New Roman; color: rgb(0, 0, 0); font-size: 16px;">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF493589"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Andrea Lani [andrea.lani@gmail.com]<br>
<b>Sent:</b> Monday, May 06, 2013 11:40 AM<br>
<b>To:</b> Nystrom, William D<br>
<b>Cc:</b> Paul Mullowney; petsc-dev@mcs.anl.gov; Nystrom, William D<br>
<b>Subject:</b> Re: [petsc-dev] Preconditioners for GPU<br>
</font><br>
</div>
<div></div>
<div>
<div>Ok, I ll try jacobi. I am also testing MHD cases by the way (interaction of Solar wind and earth Magnetosphere, both steady and unsteady) :-)</div>
<div>Thx</div>
<div>Andrea<br>
<br>
On May 6, 2013, at 7:17 PM, "Nystrom, William D" <<a href="mailto:wdn@lanl.gov" target="_blank">wdn@lanl.gov</a>> wrote:<br>
<br>
</div>
<div></div>
<blockquote type="cite">
<div>
<div style="direction: ltr; font-family: Arial; color: rgb(0, 0, 0); font-size: 14pt;">
Andrea,<br>
<br>
Have you tried jacobi preconditioning with gpus. I've found it to work pretty<br>
well, surprisingly, and it does work in parallel with mpi. It is also worth seeing<br>
if you can use the ell or hyb matrix formats. I have seen a reduction in compute<br>
time using ell/hyb matrix formats of typically 2x up to as much as 5x when the<br>
iteration counts get large. You should also try both the cusp and cusparse<br>
mat_type. I've tried the above on extended mhd problems. YMMV.<br>
<br>
Having said that, I am looking forward to trying better preconditioners for gpus<br>
as they become available. But it would be interesting to see if jacobi preconditioning<br>
on the gpu solves your problem faster than your best preconditioning option on<br>
the cpu. Also, all my problems have been SPD.<br>
<br>
Dave<br>
<div><br>
<div style="font-family: Tahoma; font-size: 13px;"><font size="2"><span style="font-size: 10pt;">--
<br>
Dave Nystrom<br>
LANL HPC-5<br>
Phone: 505-667-7913<br>
Email: <a href="mailto:wdn@lanl.gov" target="_blank"></a><a href="mailto:wdn@lanl.gov" target="_blank">wdn@lanl.gov</a><br>
Smail: Mail Stop B272<br>
Group HPC-5<br>
Los Alamos National Laboratory<br>
Los Alamos, NM 87545<br>
</span></font><br>
</div>
</div>
<div style="font-family: Times New Roman; color: rgb(0, 0, 0); font-size: 16px;">
<hr tabindex="-1">
<div id="divRpF374268" style="direction: ltr;"><font color="#000000" face="Tahoma" size="2"><b>From:</b>
<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank">petsc-dev-bounces@mcs.anl.gov</a> [<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>] on behalf of Andrea Lani [<a href="mailto:andrea.lani@gmail.com" target="_blank">andrea.lani@gmail.com</a>]<br>
<b>Sent:</b> Monday, May 06, 2013 10:52 AM<br>
<b>To:</b> Paul Mullowney<br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank"></a><a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-dev] Preconditioners for GPU<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div>Thanks Paul! this only work for sequential cases, right? in this case, what performance benefit should I expect compared to a fully CPU-based version, if any? is anything available also for parallel runs?<br>
</div>
<br>
Andrea <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, May 6, 2013 at 5:48 PM, Paul Mullowney <span dir="ltr">
<<a href="mailto:paulm@txcorp.com" target="_blank"></a><a href="mailto:paulm@txcorp.com" target="_blank">paulm@txcorp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#FFFFFF">Hi Andrea,<br>
<br>
The matrix type, aijcusparse has an ILU(n) (and ICC(n) for symmetric problems) preconditioner. The factorization is done on the CPU. The solves are done on the GPU via the cusparse library.
<br>
<br>
In the configure, do --download-txpetscgpu=yes<br>
<br>
I would also look at:<br>
<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSEQAIJCUSPARSE.html" target="_blank"></a><a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSEQAIJCUSPARSE.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSEQAIJCUSPARSE.html</a><br>
<br>
for information on the aijcusparse class.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Paul</font></span>
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">On Sun, May 5, 2013 at 4:26 PM, Andrea Lani <span dir="ltr"><<a href="mailto:andrea.lani@gmail.com" target="_blank"></a><a href="mailto:andrea.lani@gmail.com" target="_blank">andrea.lani@gmail.com</a>></span> wrote:<br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#FFFFFF">
<div>Thanks, Matt! Let me re-iterate on the question... are there other available preconditioners already ported to GPU, which are not based on AMG (which is typically not suitable for my convection-dominated CFD problems) and apart from BICGSTABCUSP? </div>
</div>
</blockquote>
<div><br>
</div>
<div>I believe that the txpetscgpu package has triangular solves for the GPU.</div>
<div><br>
</div>
<div>BiCGStab is a Krylov method.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#FFFFFF">
<div><span><font color="#888888">Andrea</font></span>
<div>
<div><br>
<br>
On May 5, 2013, at 9:53 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank"></a><a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div>
<div dir="ltr">On Sun, May 5, 2013 at 2:48 PM, Andrea Lani <span dir="ltr"><<a href="mailto:andrea.lani@gmail.com" target="_blank"></a><a href="mailto:andrea.lani@gmail.com" target="_blank">andrea.lani@gmail.com</a>></span> wrote:<br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Dear Developers,<br>
<br>
Could you please tell me the list of preconditioners for non-symmetric systems fully ported to multi-GPU and available in the current development version?<br>
</blockquote>
<div><br>
</div>
<div>In my opinion, there are no truly multi-GPU preconditioners anywhere. We can imagine them, like AMG with Chebychev smoothers, but</div>
<div>I have not really seen any of them reliably work. The CUSP SA-AMG is the closest one in this category, and Steve Dalton is working on</div>
<div>it this summer at ANL.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Thanks in advance<br>
<span><font color="#888888"><br>
Andrea</font></span></blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener </div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<span style="color: rgb(102, 102, 102);">Dr. Andrea Lani
</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">Senior Research Engineer, PhD</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">Aeronautics & Aerospace dept., CFD group
</span><span style="color: rgb(102, 102, 102);"><br>
Von Karman Institute for Fluid Dynamics </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">Chausse de Waterloo 72, </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">B-1640, Rhode-Saint-Genese, Belgium</span><span style="color: rgb(102, 102, 102);"><br>
fax : +32-2-3599600 </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(153, 153, 153);"><span style="color: rgb(102, 102, 102);">work : +32-2-3599769 </span>
<span style="background-color: rgb(255, 255, 255);"></span></span><u style="background-color: rgb(51, 51, 255); color: rgb(153, 153, 153);"><br>
<span style="background-color: rgb(255, 255, 255); color: rgb(0, 0, 153);"><a href="mailto:lani@vki.ac.be" target="_blank"></a><a href="mailto:lani@vki.ac.be" target="_blank">lani@vki.ac.be</a></span></u>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>