<div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 10pt;line-height:115%;font-size:11pt;font-family:"Calibri","sans-serif"">Thank you for your advice! My sparse matrix seems to be very
stiff so I have decided to concentrate on the direct solvers. I have very good
results with MUMPS. Due to a lack of time I haven’t got a good result with SuperLU_DIST
and haven’t compiled PETSc with Pastix yet but I have a feeling that MUMPS is
the best. I have run a sequential test case with built-in PETSc LU (-pc_type lu
-ksp_type preonly) and MUMPs (-pc_type lu -ksp_type preonly
-pc_factor_mat_solver_type mumps) with default settings and found that MUMPs was
about 50 times faster than the built-in LU and used about 3 times less RAM. Do
you have any idea why it could be? </p>
<p class="MsoNormal" style="margin:0in 0in 10pt;line-height:115%;font-size:11pt;font-family:"Calibri","sans-serif"">My test case has about 100,000 complex equations with about 3,000,000
non-zeros. PETSc was compiled with the following options: ./configure
--with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g
--with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi --with-scalar-type=complex
--with-clanguage=c --with-openmp --with-debugging=0 COPTFLAGS='-mkl=parallel
-O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' FOPTFLAGS='-mkl=parallel
-O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' CXXOPTFLAGS='-mkl=parallel
-O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' --download-superlu_dist
--download-mumps --download-scalapack --download-metis --download-cmake
--download-parmetis --download-ptscotch. </p>
<p class="MsoNormal" style="margin:0in 0in 10pt;line-height:115%;font-size:11pt;font-family:"Calibri","sans-serif"">Running MUPMS in parallel using MPI also gave me a significant
gain in performance (about 10 times on a single cluster node). </p>
<span style="font-size:11pt;line-height:115%;font-family:"Calibri","sans-serif"">Could you, please, advise me whether I can adjust
some options for the direct solvers to improve performance? Should I try MUMPS
in OpenMP mode?</span>
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">As Jed said high frequency is hard. AMG, as-is, can be adapted (<a href="https://link.springer.com/article/10.1007/s00466-006-0047-8" target="_blank">https://link.springer.com/article/10.1007/s00466-006-0047-8</a>) with parameters.<div>AMG for convection: use richardson/sor and not chebyshev smoothers and in smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0).</div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <<a href="mailto:Alexey.V.Kozlov.2@nd.edu" target="_blank">Alexey.V.Kozlov.2@nd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks a lot! I'll check them out.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> These are small enough that likely sparse direct solvers are the best use of your time and for general efficiency. <div><br></div><div> PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and Pastix. I recommend configuring PETSc for all three of them and then comparing them for problems of interest to you.</div><div><br></div><div> --download-superlu_dist --download-mumps --download-pastix --download-scalapack (used by MUMPS) --download-metis --download-parmetis --download-ptscotch </div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <<a href="mailto:Alexey.V.Kozlov.2@nd.edu" target="_blank">Alexey.V.Kozlov.2@nd.edu</a>> wrote:</div><br><div><div dir="ltr">Thanks for the tips! My matrix is complex and unsymmetric. My typical test case has of the order of one million equations. I use a 2nd-order finite-difference scheme with 19-point stencil, so my typical test case uses several GB of RAM.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Unfortunately, those are hard problems in which the "good" methods are technical and hard to make black-box. There are "sweeping" methods that solve on 2D "slabs" with PML boundary conditions, H-matrix based methods, and fancy multigrid methods. Attempting to solve with STRUMPACK is probably the easiest thing to try (--download-strumpack).<br>
<br>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html" rel="noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html</a><br>
<br>
Is the matrix complex symmetric?<br>
<br>
Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a 3D problem like this if you have enough memory. I'm assuming the memory or time is unacceptable and you want an iterative method with much lower setup costs.<br>
<br>
Alexey Kozlov <<a href="mailto:Alexey.V.Kozlov.2@nd.edu" target="_blank">Alexey.V.Kozlov.2@nd.edu</a>> writes:<br>
<br>
> Dear all,<br>
><br>
> I am solving a convected wave equation in a frequency domain. This equation<br>
> is a 3D Helmholtz equation with added first-order derivatives and mixed<br>
> derivatives, and with complex coefficients. The discretized PDE results in<br>
> a sparse linear system (about 10^6 equations) which is solved in PETSc. I<br>
> am having difficulty with the code convergence at high frequency, skewed<br>
> grid, and high Mach number. I suspect it may be due to the preconditioner I<br>
> use. I am currently using the ILU preconditioner with the number of fill<br>
> levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the art<br>
> has evolved and there are better preconditioners for Helmholtz-like<br>
> problems. Could you, please, advise me on a better preconditioner?<br>
><br>
> Thanks,<br>
> Alexey<br>
><br>
> -- <br>
> Alexey V. Kozlov<br>
><br>
> Research Scientist<br>
> Department of Aerospace and Mechanical Engineering<br>
> University of Notre Dame<br>
><br>
> 117 Hessert Center<br>
> Notre Dame, IN 46556-5684<br>
> Phone: (574) 631-4335<br>
> Fax: (574) 631-8355<br>
> Email: <a href="mailto:akozlov@nd.edu" target="_blank">akozlov@nd.edu</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,255)"><font size="4"><span style="font-family:"comic sans ms",sans-serif">Alexey V. Kozlov</span></font><br><br>Research Scientist<br>Department of Aerospace and Mechanical Engineering<br>University of Notre Dame<br><br>117 Hessert Center<br>Notre Dame, IN 46556-5684<br>Phone: (574) 631-4335<br>Fax: (574) 631-8355<br>Email: <a href="mailto:akozlov@nd.edu" target="_blank">akozlov@nd.edu</a></span><br></div></div></div></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,255)"><font size="4"><span style="font-family:"comic sans ms",sans-serif">Alexey V. Kozlov</span></font><br><br>Research Scientist<br>Department of Aerospace and Mechanical Engineering<br>University of Notre Dame<br><br>117 Hessert Center<br>Notre Dame, IN 46556-5684<br>Phone: (574) 631-4335<br>Fax: (574) 631-8355<br>Email: <a href="mailto:akozlov@nd.edu" target="_blank">akozlov@nd.edu</a></span><br></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,255)"><font size="4"><span style="font-family:comic sans ms,sans-serif">Alexey V. Kozlov</span></font><br><br>Research Scientist<br>Department of Aerospace and Mechanical Engineering<br>University of Notre Dame<br><br>117 Hessert Center<br>Notre Dame, IN 46556-5684<br>Phone: (574) 631-4335<br>Fax: (574) 631-8355<br>Email: <a href="mailto:akozlov@nd.edu" target="_blank">akozlov@nd.edu</a></span><br></div></div></div></div></div></div>