<div dir="ltr"><div><br></div><div>Barry, Jed</div><br><div>Paul and I developed a polynomial Gauss-Seidel smoother and ILUTP, ILU(0) based smoothers that employ iterative</div><div>(Neuman series, or RIchardson iteration ) for the triangular solves (faster than Jacobi, does not diverge, finite due to nitpotent</div><div>strictly upper triangular part of the U - we also use LDU (row scaled)</div><div><br></div><div>glad to send along more details as we have a paper in flight (revision being sent to NLAA this week).</div><div><br></div><div>the problems we are solving with PeleLM are quite similar to those described in the attached papers</div><div>by Prenter et al (2020) and Jomo et al (2021) - namely cut-cell, immersed boundary</div><div><br></div><div>We are seeing a 5x speed-up on the NREL eagle machine with NVIDIA V100 and a reduced 1.5 - 2x speed-up on</div><div>crusher with AMD MI250X GPUs (over the direct triangular solver ILU's). We also see 5X with the MFIX-Exa model for ECP.</div><div><br></div><div>this work was motivated by Edmond CHow and Hartwiz Anzt looking at Jacobi for triangular systems.</div><div><br></div><div>I also have a new GMRES formulation (talking about this at CEED) - that is leading to good results</div><div>for Krylov Schur eigenvalues as well.</div><div><br></div><div>Cheers and best regards</div><div>Steve</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 2, 2022 at 9:34 AM Paul Mullowney <<a href="mailto:paulmullowney@gmail.com" target="_blank">paulmullowney@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div id="gmail-m_7386655642987321857gmail-m_-8522094473160406613gmail-:2f2" aria-label="Message Body" role="textbox" aria-multiline="true" style="direction:ltr;min-height:85px">The implementation is being (slowly) moved into Hypre. We have primarily used this technique with ILU-based smoothers for AMG. We did some comparisons against other smoothers like GS but not with Chebyshev or Polynomial. <div><br></div><div>For the problems we cared about, ILU was an effective smoother. The power series representation of the solve provided some nice speedups. I'ved cc'ed Steve Thomas who could say more.<div><br></div><div>-Paul</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 31, 2022 at 10:14 PM Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Do you have a test that compares this with a polynomial smoother for the original problem (like Chebyshev for SPD)?<br>
<br>
Paul Mullowney <<a href="mailto:paulmullowney@gmail.com" target="_blank">paulmullowney@gmail.com</a>> writes:<br>
<br>
> One could also approximate the SOR triangular solves with a Neumann series,<br>
> where each term in the series is a SpMV (great for GPUs). The number of<br>
> terms needed in the series is matrix dependent.<br>
> We've seen this work to great effect for some problems.<br>
><br>
> -Paul<br>
><br>
> On Wed, Jul 27, 2022 at 8:05 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br>
><br>
>><br>
>> There are multicolor versions of SOR that theoretically offer good<br>
>> parallelism on GPUs but at the cost of multiple phases and slower<br>
>> convergence rates. Unless someone already has one coded for CUDA or Kokkos<br>
>> it would take a good amount of code to produce one that offers (but does<br>
>> not necessarily guarantee) reasonable performance on GPUs.<br>
>><br>
>> > On Jul 27, 2022, at 7:57 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br>
>> ><br>
>> > Unfortunately, MatSOR is a really bad operation for GPUs. We can make it<br>
>> use sparse triangular primitives from cuSPARSE, but those run on GPU at<br>
>> about 20x slower than MatMult with the same sparse matrix. So unless MatSOR<br>
>> reduces iteration count by 20x compared to your next-best preconditioning<br>
>> option, you'll be better off finding a different preconditioner. This might<br>
>> be some elements of multigrid or polynomial smoothing with point-block<br>
>> Jacobi. If you can explain a bit about your application, we may be able to<br>
>> offer some advice.<br>
>> ><br>
>> > Han Tran <<a href="mailto:hantran@cs.utah.edu" target="_blank">hantran@cs.utah.edu</a>> writes:<br>
>> ><br>
>> >> Hello,<br>
>> >><br>
>> >> Running my example using VECMPICUDA for VecSetType(), and MATMPIAIJCUSP<br>
>> for MatSetType(), I have the profiling results as shown below. It is seen<br>
>> that MatSOR() has %F of GPU, only has GpuToCpu count and size. Is it<br>
>> correct that PETSc currently does not have MatSOR implemented on GPU? It<br>
>> would be appreciated if you can provide an explanation on how MatSOR()<br>
>> currently use GPU. From this example, MatSOR takes a considerable time<br>
>> relatively compared to other functions.<br>
>> >><br>
>> >> Thank you.<br>
>> >><br>
>> >> -Han<br>
>> >><br>
>> >><br>
>> ------------------------------------------------------------------------------------------------------------------------<br>
>> >> Event Count Time (sec) Flop<br>
>> --- Global --- --- Stage ---- Total GPU - CpuToGpu - -<br>
>> GpuToCpu - GPU<br>
>> >> Max Ratio Max Ratio Max Ratio Mess AvgLen<br>
>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size<br>
>> Count Size %F<br>
>> >><br>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
>> >><br>
>> >> --- Event Stage 0: Main Stage<br>
>> >><br>
>> >> BuildTwoSided 220001 1.0 3.9580e+02139.9 0.00e+00 0.0 2.0e+00<br>
>> 4.0e+00 2.2e+05 4 0 0 0 20 4 0 0 0 20 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> BuildTwoSidedF 220000 1.0 3.9614e+02126.4 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 2.2e+05 4 0 0 0 20 4 0 0 0 20 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecMDot 386001 1.0 6.3426e+01 1.5 1.05e+11 1.0 0.0e+00<br>
>> 0.0e+00 3.9e+05 1 11 0 0 35 1 11 0 0 35 3311 26012 386001<br>
>> 1.71e+05 0 0.00e+00 100<br>
>> >> VecNorm 496001 1.0 5.0877e+01 1.2 5.49e+10 1.0 0.0e+00<br>
>> 0.0e+00 5.0e+05 1 6 0 0 45 1 6 0 0 45 2159 3707 110000<br>
>> 4.87e+04 0 0.00e+00 100<br>
>> >> VecScale 496001 1.0 7.9951e+00 1.0 2.75e+10 1.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 6869 13321 0<br>
>> 0.00e+00 0 0.00e+00 100<br>
>> >> VecCopy 110000 1.0 1.9323e+00 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecSet 330017 1.0 5.4319e+00 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecAXPY 110000 1.0 1.5820e+00 1.0 1.22e+10 1.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 15399 35566 0<br>
>> 0.00e+00 0 0.00e+00 100<br>
>> >> VecMAXPY 496001 1.0 1.1505e+01 1.0 1.48e+11 1.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 16 0 0 0 0 16 0 0 0 25665 39638 0<br>
>> 0.00e+00 0 0.00e+00 100<br>
>> >> VecAssemblyBegin 110000 1.0 1.2021e+00 1.2 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 1.1e+05 0 0 0 0 10 0 0 0 0 10 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecAssemblyEnd 110000 1.0 1.5988e-01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecScatterBegin 496001 1.0 1.3002e+01 1.0 0.00e+00 0.0 9.9e+05<br>
>> 1.3e+04 1.0e+00 0 0100100 0 0 0100100 0 0 0 110000<br>
>> 4.87e+04 0 0.00e+00 0<br>
>> >> VecScatterEnd 496001 1.0 1.8988e+01 1.3 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> VecNormalize 496001 1.0 5.8797e+01 1.1 8.24e+10 1.0 0.0e+00<br>
>> 0.0e+00 5.0e+05 1 9 0 0 45 1 9 0 0 45 2802 4881 110000<br>
>> 4.87e+04 0 0.00e+00 100<br>
>> >> VecCUDACopyTo 716001 1.0 3.4483e+01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 716001<br>
>> 3.17e+05 0 0.00e+00 0<br>
>> >> VecCUDACopyFrom 1211994 1.0 5.1752e+01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 0<br>
>> 0.00e+00 1211994 5.37e+05 0<br>
>> >> MatMult 386001 1.0 4.8436e+01 1.0 1.90e+11 1.0 7.7e+05<br>
>> 1.3e+04 0.0e+00 1 21 78 78 0 1 21 78 78 0 7862 16962 0<br>
>> 0.00e+00 0 0.00e+00 100<br>
>> >> MatMultAdd 110000 1.0 6.2666e+01 1.1 6.03e+10 1.0 2.2e+05<br>
>> 1.3e+04 1.0e+00 1 7 22 22 0 1 7 22 22 0 1926 16893 440000<br>
>> 3.39e+05 0 0.00e+00 100<br>
>> >> MatSOR 496001 1.0 5.1821e+02 1.1 2.83e+11 1.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 10 31 0 0 0 10 31 0 0 0 1090 0 0<br>
>> 0.00e+00 991994 4.39e+05 0<br>
>> >> MatAssemblyBegin 110000 1.0 3.9732e+02109.2 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 1.1e+05 4 0 0 0 10 4 0 0 0 10 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> MatAssemblyEnd 110000 1.0 5.3015e-01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> MatZeroEntries 110000 1.0 1.3179e+01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> MatCUSPARSCopyTo 220000 1.0 3.2805e+01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 220000<br>
>> 2.41e+05 0 0.00e+00 0<br>
>> >> KSPSetUp 110000 1.0 3.5344e-02 1.3 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> KSPSolve 110000 1.0 6.8304e+02 1.0 8.20e+11 1.0 7.7e+05<br>
>> 1.3e+04 8.8e+05 13 89 78 78 80 13 89 78 78 80 2401 14311 496001<br>
>> 2.20e+05 991994 4.39e+05 66<br>
>> >> KSPGMRESOrthog 386001 1.0 7.2820e+01 1.4 2.10e+11 1.0 0.0e+00<br>
>> 0.0e+00 3.9e+05 1 23 0 0 35 1 23 0 0 35 5765 30176 386001<br>
>> 1.71e+05 0 0.00e+00 100<br>
>> >> PCSetUp 110000 1.0 1.8825e-02 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> PCApply 496001 1.0 5.1857e+02 1.1 2.83e+11 1.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 10 31 0 0 0 10 31 0 0 0 1090 0 0<br>
>> 0.00e+00 991994 4.39e+05 0<br>
>> >> SFSetGraph 1 1.0 2.0936e-05 1.1 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> SFSetUp 1 1.0 2.5347e-03 1.0 0.00e+00 0.0 4.0e+00<br>
>> 3.3e+03 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> SFPack 496001 1.0 3.0026e+00 1.1 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >> SFUnpack 496001 1.0 1.1296e-01 1.0 0.00e+00 0.0 0.0e+00<br>
>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0<br>
>> 0.00e+00 0 0.00e+00 0<br>
>> >><br>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
>><br>
>><br>
</blockquote></div>
</blockquote></div>