[petsc-users] Usage of AMG as preconditioner

[Michael Werner]

Matthew Knepley writes:

> On Fri, Sep 28, 2018 at 8:13 AM Michael Werner 
> <michael.werner at dlr.de>
> wrote:
>
>>
>> Matthew Knepley writes:
>>
>> > On Fri, Sep 28, 2018 at 7:43 AM Michael Werner
>> > <michael.werner at dlr.de>
>> > wrote:
>> >
>> >>
>> >> Matthew Knepley writes:
>> >>
>> >> > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner
>> >> > <michael.werner at dlr.de>
>> >> > wrote:
>> >> >
>> >> >> Hello,
>> >> >>
>> >> >> I'm having trouble with getting the AMG preconditioners
>> >> >> working. I
>> >> >> tried all of them (gamg, ml, hypre-boomeramg), with 
>> >> >> varying
>> >> >> degrees of "success":
>> >> >>
>> >> >> - GAMG:
>> >> >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual
>> >> >> -ksp_max_it
>> >> >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE
>> >> >> -pc_gamg_agg_nsmooths 1 -ksp_view
>> >> >> always crashes with the following error:
>> >> >> Error: error code 77
>> >> >> [0] KSPSolve() line 780 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c
>> >> >> [0] KSPSolve_GMRES() line 233 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c
>> >> >> [0] KSPInitialResidual() line 67 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c
>> >> >> [0] KSP_PCApply() line 281 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h
>> >> >> [0] PCApply() line 462 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c
>> >> >> [0] PCApply_MG() line 377 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c
>> >> >> [0] PCMGMCycle_Private() line 20 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c
>> >> >> [0] KSPSolve() line 780 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c
>> >> >> [0] KSPSolve_Chebyshev() line 381 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c
>> >> >> [0] Petsc has generated inconsistent data
>> >> >> [0] Eigen estimator failed: DIVERGED_NANORINF at 
>> >> >> iteration 0
>> >> >>
>> >> >> When I'm using a different solver for 
>> >> >> -mg_levels_ksp_type,
>> >> >> such
>> >> >> as
>> >> >> gmres, GAMG no longer crashes, but I don't see 
>> >> >> convergence
>> >> >> of
>> >> >> the
>> >> >> problem (convergence history and ksp_view output are
>> >> >> attached
>> >> >> below).
>> >> >>
>> >> >
>> >> > It uses unpreconditioned GMRES to estimate spectral bounds
>> >> > for
>> >> > the operator
>> >> > before using a Chebychev smoother.
>> >> > If your matrix does not have a nice, connected, positive
>> >> > spectrum,
>> >> > Chebychev will not work. However, the fact that
>> >> > you get DIVERGED_NANORINF in the estimator tells me that 
>> >> > you
>> >> > have a problem
>> >> > in the matrix.
>> >> >
>> >>
>> >> The error above (DIVERGED_NANORINF) only appears for
>> >> -mg_levels_ksp_type chebyshev. When I use GMRES
>> >> (-mg_levels_ksp_type gmres) there are no errors, the KSP 
>> >> just
>> >> never converges.
>> >>
>> >> >
>> >> >> - Hypre
>> >> >> With the default settings, BoomerAMG just returns a 
>> >> >> Vector
>> >> >> of
>> >> >> all
>> >> >> zeros after one iteration.
>> >> >> When I change the relaxation type
>> >> >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get 
>> >> >> similar
>> >> >> results than with GAMG: the solver works without errors, 
>> >> >> but
>> >> >> doesn't converge. The output for Hypre is also attached
>> >> >> below.
>> >> >>
>> >> >> - ML
>> >> >> With default settings the result is just like Boomeramg: 
>> >> >> a
>> >> >> vector
>> >> >> of all zeros after one iteration.
>> >> >> When I change -mg_levels_ksp_type the behaviour is 
>> >> >> identical
>> >> >> to
>> >> >> GAMG.
>> >> >>
>> >> >>
>> >> >> Since none of the packages worked, I'm assuming that the
>> >> >> error
>> >> >> lies with me/ my code,
>> >> >
>> >> >
>> >> > It looks like a value in the matrix might be bad.
>> >> >
>> >> >
>> >> >> so I'll give a short overview over what I'm
>> >> >> trying to do.
>> >> >> The matrix I'm trying to precondition is the Jacobian of 
>> >> >> a
>> >> >> flow
>> >> >> field originating from an unstructured finite-volume CFD
>> >> >> code.
>> >
>> >
>> > Compressible or incompressible?
>> >
>> >   Mstt
>> >
>>
>> The matrix is generated by a compressible CFD code. The current
>> test case is quasi-incompressible due to very low flow velocity
>> (flow around a cylinder).
>>
>
> Ah. I do not believe Multigrid is going to work for compressible 
> flow, at
> least not the
> straightforward kind of multigrid that we have in PETSc. MG is 
> designed for
> elliptic
> problems. Hyprebolic problems require extensive changes and its 
> still a
> research problem
> to get it working. Definitely no freely available package has it 
> working,
> and others cannot be
> tested by us.
>
> Moreover, in the incompressible limit, the compressible 
> formulation can
> become very stiff,
> leading to failure of solvers. That is why you get the slightly
> compressible (adiabatic?) formulations.
>
>   Thanks,
>
>      Matt
>
Ah, too bad, I was already expecting something like that, because 
most of the applications I found so far were FEM/ elasticity 
problems. Then I'll have to see if I can find some other suitable 
solution.

Kind regards,
Michael

>
>> Kind regards,
>> Michael
>>
>> >
>> >> It
>> >> >> has a blocked structure as each node of the original mesh
>> >> >> holds
>> >> >> several variables (pressure, density, 
>> >> >> velocities). However,
>> >> >> I'm
>> >> >> not using a DM-Plex since I get the assembled Jacobian in
>> >> >> binary
>> >> >> format from the external CFD code.
>> >> >> When I'm using direct (lu) I get correct results, so the
>> >> >> basic
>> >> >> code should be fine.
>> >> >
>> >> >
>> >> > So LU works for the same problem that gave the NANORINF in
>> >> > GMRES?
>> >> >
>> >> > I would recommend using a Manufactured Solution to check 
>> >> > the
>> >> > operation. Then
>> >> > you can start by feeding in the exact solution, and see 
>> >> > that
>> >> > nothing fails.
>> >> > Also, I
>> >> > would make a very small test case, so that you can send 
>> >> > the
>> >> > matrix and/or
>> >> > code
>> >> > to us to check.
>> >> >
>> >>
>> >> Yes, LU works for the same matrix/ problem. I didn't change
>> >> anything besides using -pc_type lu instead of -pc_type gamg.
>> >> What do you mean by "Manufactured Solution"?
>> >> Concerning the test case: I'll set one up and send you the
>> >> matrix
>> >> in binary format.
>> >>
>> >> >
>> >> >> However, for larger problems lu is not
>> >> >> feasible due the very large memory requirements, 
>> >> >> therefore I
>> >> >> wanted to switch over to multigrid.
>> >> >>
>> >> >> Currently, PETSc has no information about the original
>> >> >> geometry. I
>> >> >> tried using setCoordinates, but for Hypre and ml it 
>> >> >> didn't
>> >> >> make
>> >> >> a
>> >> >> difference, and GAMG crashed with an error:
>> >> >>
>> >> >
>> >> > This is only an optimization, and currently GAMG does not
>> >> > know
>> >> > what to
>> >> > do for multiple fields (it handles single field problems 
>> >> > by
>> >> > building the
>> >> > nullspace of the symbol for the coarse grid). You could
>> >> > provide
>> >> > this when
>> >> > you want to optimize things.
>> >> >
>> >> >   Thanks,
>> >> >
>> >> >     Matt
>> >>
>> >> Ok, thanks for the clarification. In that case I won't 
>> >> bother
>> >> with
>> >> it for now. No need to optimize the code before the other
>> >> problems
>> >> are solved.
>> >>
>> >> Thank you for your answer!
>> >>
>> >> Kind regards,
>> >> Michael
>> >> >
>> >> >
>> >> >> [0] PCSetCoordinates() line 1883 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c
>> >> >> [0] PCSetCoordinates_AGG() line 199 in
>> >> >>
>> >> >>
>> >>
>> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c
>> >> >> [0] Petsc has generated inconsistent data
>> >> >> [0] Don't know how to create null space for ndm=2, ndf=4.
>> >> >> Use
>> >> >> MatSetNearNullSpace.
>> >> >>
>> >> >> I would be glad if you could give me some advice on how 
>> >> >> to
>> >> >> deal
>> >> >> with this.
>> >> >>
>> >> >> Kind regards,
>> >> >> Michael
>> >> >>
>> >> >>
>> >>
>> >>
>>