[petsc-users] Preconditioner for Helmholtz-like problem

Fri Oct 16 23:42:22 CDT 2020

> On Oct 16, 2020, at 11:33 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Oct 16, 2020 at 11:48 PM Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu <mailto:Alexey.V.Kozlov.2 at nd.edu>> wrote:
> Thank you for your advice! My sparse matrix seems to be very stiff so I have decided to concentrate on the direct solvers. I have very good results with MUMPS. Due to a lack of time I haven’t got a good result with SuperLU_DIST and haven’t compiled PETSc with Pastix yet but I have a feeling that MUMPS is the best. I have run a sequential test case with built-in PETSc LU (-pc_type lu -ksp_type preonly) and MUMPs (-pc_type lu -ksp_type preonly -pc_factor_mat_solver_type mumps) with default settings and found that MUMPs was about 50 times faster than the built-in LU and used about 3 times less RAM. Do you have any idea why it could be?
> 
> The numbers do not sound realistic, but of course we do not have your particular problem. In particular, the memory figure seems impossible. 

   They are probably using a different ordering. Remember each external direct solver manages its own orderings and doesn't share even their names. (Not nice community behavior).

> 
> My test case has about 100,000 complex equations with about 3,000,000 non-zeros. PETSc was compiled with the following options: ./configure --with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g --with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi --with-scalar-type=complex --with-clanguage=c --with-openmp --with-debugging=0 COPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' FOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' CXXOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' --download-superlu_dist --download-mumps --download-scalapack --download-metis --download-cmake --download-parmetis --download-ptscotch.
> 
> Running MUPMS in parallel using MPI also gave me a significant gain in performance (about 10 times on a single cluster node).
> 
> Again, this does not appear to make sense. The performance should be limited by memory bandwidth, and a single cluster node will not usually have
> 10x the bandwidth of a CPU, although it might be possible with a very old CPU.
> 
> It would help to understand the performance if you would send the output of -log_view.
> 
>   Thanks,
> 
>     Matt
> 
> Could you, please, advise me whether I can adjust some options for the direct solvers to improve performance? Should I try MUMPS in OpenMP mode?

    Look at the orderings and other options that MUMPs supports and try them out. Most can be accessed from the command line. You can run with -help to get a real brief summary of them but should read the MUMPs users manual for full details.

> 
> On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> As Jed said high frequency is hard. AMG, as-is,  can be adapted (https://link.springer.com/article/10.1007/s00466-006-0047-8 <https://link.springer.com/article/10.1007/s00466-006-0047-8>) with parameters.
> AMG for convection: use richardson/sor and not chebyshev smoothers and in smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0).
> Mark
> 
> On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu <mailto:Alexey.V.Kozlov.2 at nd.edu>> wrote:
> Thanks a lot! I'll check them out.
> 
> On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> 
>   These are small enough that likely sparse direct solvers are the best use of your time and for general efficiency. 
> 
>   PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and Pastix. I recommend configuring PETSc for all three of them and then comparing them for problems of interest to you.
> 
>    --download-superlu_dist --download-mumps --download-pastix --download-scalapack (used by MUMPS) --download-metis --download-parmetis --download-ptscotch 
> 
>   Barry
> 
> 
>> On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu <mailto:Alexey.V.Kozlov.2 at nd.edu>> wrote:
>> 
>> Thanks for the tips! My matrix is complex and unsymmetric. My typical test case has of the order of one million equations. I use a 2nd-order finite-difference scheme with 19-point stencil, so my typical test case uses several GB of RAM.
>> 
>> On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <jed at jedbrown.org <mailto:jed at jedbrown.org>> wrote:
>> Unfortunately, those are hard problems in which the "good" methods are technical and hard to make black-box.  There are "sweeping" methods that solve on 2D "slabs" with PML boundary conditions, H-matrix based methods, and fancy multigrid methods.  Attempting to solve with STRUMPACK is probably the easiest thing to try (--download-strumpack).
>> 
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html>
>> 
>> Is the matrix complex symmetric?
>> 
>> Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a 3D problem like this if you have enough memory.  I'm assuming the memory or time is unacceptable and you want an iterative method with much lower setup costs.
>> 
>> Alexey Kozlov <Alexey.V.Kozlov.2 at nd.edu <mailto:Alexey.V.Kozlov.2 at nd.edu>> writes:
>> 
>> > Dear all,
>> >
>> > I am solving a convected wave equation in a frequency domain. This equation
>> > is a 3D Helmholtz equation with added first-order derivatives and mixed
>> > derivatives, and with complex coefficients. The discretized PDE results in
>> > a sparse linear system (about 10^6 equations) which is solved in PETSc. I
>> > am having difficulty with the code convergence at high frequency, skewed
>> > grid, and high Mach number. I suspect it may be due to the preconditioner I
>> > use. I am currently using the ILU preconditioner with the number of fill
>> > levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the art
>> > has evolved and there are better preconditioners for Helmholtz-like
>> > problems. Could you, please, advise me on a better preconditioner?
>> >
>> > Thanks,
>> > Alexey
>> >
>> > -- 
>> > Alexey V. Kozlov
>> >
>> > Research Scientist
>> > Department of Aerospace and Mechanical Engineering
>> > University of Notre Dame
>> >
>> > 117 Hessert Center
>> > Notre Dame, IN 46556-5684
>> > Phone: (574) 631-4335
>> > Fax: (574) 631-8355
>> > Email: akozlov at nd.edu <mailto:akozlov at nd.edu>
>> 
>> 
>> -- 
>> Alexey V. Kozlov
>> 
>> Research Scientist
>> Department of Aerospace and Mechanical Engineering
>> University of Notre Dame
>> 
>> 117 Hessert Center
>> Notre Dame, IN 46556-5684
>> Phone: (574) 631-4335
>> Fax: (574) 631-8355
>> Email: akozlov at nd.edu <mailto:akozlov at nd.edu>
> 
> 
> 
> -- 
> Alexey V. Kozlov
> 
> Research Scientist
> Department of Aerospace and Mechanical Engineering
> University of Notre Dame
> 
> 117 Hessert Center
> Notre Dame, IN 46556-5684
> Phone: (574) 631-4335
> Fax: (574) 631-8355
> Email: akozlov at nd.edu <mailto:akozlov at nd.edu>
> 
> 
> -- 
> Alexey V. Kozlov
> 
> Research Scientist
> Department of Aerospace and Mechanical Engineering
> University of Notre Dame
> 
> 117 Hessert Center
> Notre Dame, IN 46556-5684
> Phone: (574) 631-4335
> Fax: (574) 631-8355
> Email: akozlov at nd.edu <mailto:akozlov at nd.edu>
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20201016/8704bbdb/attachment.html>