<div dir="ltr"><div>Well, I have 9 equations, so 9x9 I guess... <br><br>I hope the one you are mentioning was a major bug, because what I get is seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual of +0.72, on 8-cores/GPU I get -1.00 at the first time step, just to make an example. Can this be due to the bug you are saying or you can suspect something more?<br>
<br>What should I do then? wait for the valgrind fix which is underway and then update? Can you please notify me when this is fixed? I'm writing a final report for a project and I would like to include this feature fully fixed if possible.<br>
<br>Another question, what do you exactly mean by "order the unknowns properly" in this case? <br></div><div>Thanks a lot!<br><br></div>Andrea <br></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Jan 17, 2014 at 10:02 PM, Karl Rupp <span dir="ltr"><<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Andrea,<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In fact, I have another major problem: when running on multi-GPU with<br>
PETSc my results are totally inconsistent compared to a single GPU .<br>
</blockquote>
<br></div>
This was a bug which was fixed a couple of days ago. It is in branch 'next', but not yet merged to master since it has another valgrind issue I haven't nailed down yet.<div class="im"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In my code, for now, I'm assuming a 1-1 correspondence between CPU and<br>
GPU: I run on 8 cores and 8 GPUs (4 K10). How can I enforce this in the<br>
PETSc solver? Is it automatically done or do I have to specify some options?<br>
</blockquote>
<br></div>
One MPI rank maps to one logical GPU. In your case, please run with 8 MPI ranks and distribute them equally over the nodes equipped with the GPUs.<br>
<br>
As for the preconditioners: We haven't added any new preconditioners recently. Preconditioning on GPUs is a very problem-specific thing due to the burden of PCI-Express latency. Massively parallel approaches such as Sparse Approximate Inverses perform well in terms of theoretical FLOP counts, but are poor in terms of convergence and pretty expensive in terms of memory when running many simultaneous factorizations. ILU on the GPU can be fast if you order the unknowns properly and have only few nonzeros per row, but it is not great in terms of convergence rate either. PCI-Express bandwidth and latency is really a problem here...<br>
<br>
How large are your blocks when using a block-Jacobi preconditioner for your problem? In the order of 3x3 or (much) larger?<br>
<br>
Best regards,<br>
Karli<br>
<br>
</blockquote></div><br><br clear="all"><br>-- <br><span style="color:rgb(102,102,102)">Dr. Andrea Lani </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(102,102,102)">Senior Research Engineer, PhD</span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">Aeronautics & Aerospace dept., CFD group </span><span style="color:rgb(102,102,102)"><br>
Von Karman Institute for Fluid Dynamics </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">
Chausse de Waterloo 72, </span><br style="color:rgb(102,102,102)"><span style="color:rgb(102,102,102)">B-1640, Rhode-Saint-Genese, Belgium</span><span style="color:rgb(102,102,102)"><br>fax : +32-2-3599600 </span><br style="color:rgb(102,102,102)">
<span style="color:rgb(153,153,153)"><span style="color:rgb(102,102,102)">
work : +32-2-3599769 </span> <span style="background-color:rgb(255,255,255)"></span></span><u style="background-color:rgb(51,51,255);color:rgb(153,153,153)"><br><span style="background-color:rgb(255,255,255);color:rgb(0,0,153)"><a href="mailto:lani@vki.ac.be" target="_blank">lani@vki.ac.be</a></span></u>
</div>