[petsc-users] Preconditioner for Helmholtz-like problem

[Alexey Kozlov]

Thank you for your advice! My sparse matrix seems to be very stiff so I
have decided to concentrate on the direct solvers. I have very good results
with MUMPS. Due to a lack of time I haven’t got a good result with
SuperLU_DIST and haven’t compiled PETSc with Pastix yet but I have a
feeling that MUMPS is the best. I have run a sequential test case with
built-in PETSc LU (-pc_type lu -ksp_type preonly) and MUMPs (-pc_type lu
-ksp_type preonly -pc_factor_mat_solver_type mumps) with default settings
and found that MUMPs was about 50 times faster than the built-in LU and
used about 3 times less RAM. Do you have any idea why it could be?

My test case has about 100,000 complex equations with about 3,000,000
non-zeros. PETSc was compiled with the following options: ./configure
--with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g
--with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi
--with-scalar-type=complex --with-clanguage=c --with-openmp
--with-debugging=0 COPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2
-no-prec-div -fp-model fast=2' FOPTFLAGS='-mkl=parallel -O2 -mavx
-axCORE-AVX2 -no-prec-div -fp-model fast=2' CXXOPTFLAGS='-mkl=parallel -O2
-mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' --download-superlu_dist
--download-mumps --download-scalapack --download-metis --download-cmake
--download-parmetis --download-ptscotch.

Running MUPMS in parallel using MPI also gave me a significant gain in
performance (about 10 times on a single cluster node).
Could you, please, advise me whether I can adjust some options for the
direct solvers to improve performance? Should I try MUMPS in OpenMP mode?

On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <mfadams at lbl.gov> wrote:

> As Jed said high frequency is hard. AMG, as-is,  can be adapted (
> https://link.springer.com/article/10.1007/s00466-006-0047-8) with
> parameters.
> AMG for convection: use richardson/sor and not chebyshev smoothers and in
> smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0).
> Mark
>
> On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu>
> wrote:
>
>> Thanks a lot! I'll check them out.
>>
>> On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <bsmith at petsc.dev> wrote:
>>
>>>
>>>   These are small enough that likely sparse direct solvers are the best
>>> use of your time and for general efficiency.
>>>
>>>   PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and
>>> Pastix. I recommend configuring PETSc for all three of them and then
>>> comparing them for problems of interest to you.
>>>
>>>    --download-superlu_dist --download-mumps --download-pastix
>>> --download-scalapack (used by MUMPS) --download-metis --download-parmetis
>>> --download-ptscotch
>>>
>>>   Barry
>>>
>>>
>>> On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu>
>>> wrote:
>>>
>>> Thanks for the tips! My matrix is complex and unsymmetric. My typical
>>> test case has of the order of one million equations. I use a 2nd-order
>>> finite-difference scheme with 19-point stencil, so my typical test case
>>> uses several GB of RAM.
>>>
>>> On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Unfortunately, those are hard problems in which the "good" methods are
>>>> technical and hard to make black-box.  There are "sweeping" methods that
>>>> solve on 2D "slabs" with PML boundary conditions, H-matrix based methods,
>>>> and fancy multigrid methods.  Attempting to solve with STRUMPACK is
>>>> probably the easiest thing to try (--download-strumpack).
>>>>
>>>>
>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html
>>>>
>>>> Is the matrix complex symmetric?
>>>>
>>>> Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a 3D
>>>> problem like this if you have enough memory.  I'm assuming the memory or
>>>> time is unacceptable and you want an iterative method with much lower setup
>>>> costs.
>>>>
>>>> Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu> writes:
>>>>
>>>> > Dear all,
>>>> >
>>>> > I am solving a convected wave equation in a frequency domain. This
>>>> equation
>>>> > is a 3D Helmholtz equation with added first-order derivatives and
>>>> mixed
>>>> > derivatives, and with complex coefficients. The discretized PDE
>>>> results in
>>>> > a sparse linear system (about 10^6 equations) which is solved in
>>>> PETSc. I
>>>> > am having difficulty with the code convergence at high frequency,
>>>> skewed
>>>> > grid, and high Mach number. I suspect it may be due to the
>>>> preconditioner I
>>>> > use. I am currently using the ILU preconditioner with the number of
>>>> fill
>>>> > levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the
>>>> art
>>>> > has evolved and there are better preconditioners for Helmholtz-like
>>>> > problems. Could you, please, advise me on a better preconditioner?
>>>> >
>>>> > Thanks,
>>>> > Alexey
>>>> >
>>>> > --
>>>> > Alexey V. Kozlov
>>>> >
>>>> > Research Scientist
>>>> > Department of Aerospace and Mechanical Engineering
>>>> > University of Notre Dame
>>>> >
>>>> > 117 Hessert Center
>>>> > Notre Dame, IN 46556-5684
>>>> > Phone: (574) 631-4335
>>>> > Fax: (574) 631-8355
>>>> > Email: akozlov at nd.edu
>>>>
>>>
>>>
>>> --
>>> Alexey V. Kozlov
>>>
>>> Research Scientist
>>> Department of Aerospace and Mechanical Engineering
>>> University of Notre Dame
>>>
>>> 117 Hessert Center
>>> Notre Dame, IN 46556-5684
>>> Phone: (574) 631-4335
>>> Fax: (574) 631-8355
>>> Email: akozlov at nd.edu
>>>
>>>
>>>
>>
>> --
>> Alexey V. Kozlov
>>
>> Research Scientist
>> Department of Aerospace and Mechanical Engineering
>> University of Notre Dame
>>
>> 117 Hessert Center
>> Notre Dame, IN 46556-5684
>> Phone: (574) 631-4335
>> Fax: (574) 631-8355
>> Email: akozlov at nd.edu
>>
>

-- 
Alexey V. Kozlov

Research Scientist
Department of Aerospace and Mechanical Engineering
University of Notre Dame

117 Hessert Center
Notre Dame, IN 46556-5684
Phone: (574) 631-4335
Fax: (574) 631-8355
Email: akozlov at nd.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20201016/440e75d5/attachment.html>