[petsc-users] troubleshooting AMG, coupled navier stokes, large eigenvalues on coarsest level

[Mark Adams]

On Tue, Nov 7, 2017 at 3:00 PM, Mark Lohry <mlohry at gmail.com> wrote:

> I've now got gamg running on matrix-free newton-krylov with the jacobian
> provided by coloring finite differences (thanks again for the help). 3D
> Poisson with 4th order DG or higher (35^2 blocks), gamg with default
> settings is giving textbook convergence, which is great. Of course coupled
> compressible navier-stokes is harder, and convergence is bad-to-nonexistent.
>
>
> 1) Doc says "Equations must be ordered in “vertex-major” ordering"; in my
> discretization, each "node" has 5 coupled degrees of freedom (density, 3 x
> momentum, energy). I'm ordering my unknowns:
>
> rho_i, rhou_i, rhov_i, rhow_i, Et_i, rho_i+1, rhou_i+1, ...  e.g.
> row-major matrix order if you wrote the unknowns [{rho}, {rhou}, ... ].
>
> and then setting block size to 5. Is that correct?
>

yes


> I've also tried using the actual sparsity of the matrix
>

Sorry, but using for what?


> which has larger dense blocks (e.g. [35x5]^2), but neither seemed to help.
>

Do you mean the element Jacobian matrix has 35 "vertices"? You have 5x5
dense blocks (or at least you don't care about resolving any sparsity or
you want to use block preconditioners. So your element Jacobians are
[35x5]^2 dense(ish) matrices. This is not particularly useful for the
solvers.


>
>
> 2) With default settings, and with -pc_gamg_square_graph,
> pc_gamg_sym_graph, agg_nsmooths 0 mentioned in the manual, the eigenvalue
> estimates explode on the coarsest level, which I don't see with poisson:
>
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 32 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.18994, max = 2.08935
>         eigenvalues estimate via gmres min 0.00933256, max 1.8994
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 32 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.165969, max = 1.82566
>         eigenvalues estimate via gmres min 0.0290728, max 1.65969
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 32 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.146479, max = 1.61126
>         eigenvalues estimate via gmres min 0.204673, max 1.46479
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 32 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 6.81977e+09, max = 7.50175e+10
>         eigenvalues estimate via gmres min -2.76436e+12, max 6.81977e+10
>
> What's happening here? (Full -ksp_view below)
>

This is on the full NS problem I assume. You need special solvers for this.
Scalable solvers for NS is not trivial.


>
> 3) I'm not very familiar with chebyshev smoothers, but they're only for
> SPD systems (?).
>

Yes, you should use Gauss-Seidel for non-symmetric systems. Or indefinite
systems


> Is this an inappropriate smoother for this problem?
>
> 4) With gmres, the preconditioned residual is ~10 orders larger than the
> true residual; and the preconditioned residual drops while the true
> residual rises. I'm assuming this means something very wrong?
>

Yes, your preconditioner is no good.


>
> 5) -pc_type hyper -pc_hypre_type boomeramg also works perfectly for the
> poisson case, but hits NaN on the first cycle for NS.
>
>
>
You want to use the Schur complement PCs for NS. We have some support for
PC where you give us a mass matrix. These are motivated by assuming there
is a "commutation" (that is not true). These were developed by, among
others, Andy Wathan, if you want to do a literature search. You will
probably have to read the literature to understand the options and issues
for your problem. Look at the PETSc manual and you should find a
description of PETSc support for these PCs and some discussion of them and
references. This should get you started.

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171108/74ed2874/attachment.html>