[petsc-users] GAMG parameters for ideal coarsening ratio

[Jed Brown]

Sajid Ali <sajidsyed2021 at u.northwestern.edu> writes:

> Hi PETSc-developers,
>
> As per the manual, the ideal gamg parameters are those which result in
> MatPtAP time being roughly similar to (or just slightly larger) than KSP
> solve times. The way to adjust this is by changing the threshold for
> coarsening and/or squaring the graph. I was working with a grid of size
> 2^14 by 2^14 in a linear & time-independent TS with the following params :
>
> #PETSc Option Table entries:
> -ksp_monitor
> -ksp_rtol 1e-5
> -ksp_type fgmres
> -ksp_view
> -log_view
> -mg_levels_ksp_type gmres
> -mg_levels_pc_type jacobi
> -pc_gamg_coarse_eq_limit 1000
> -pc_gamg_reuse_interpolation true
> -pc_gamg_square_graph 10
> -pc_gamg_threshold -0.04
> -pc_gamg_type agg
> -pc_gamg_use_parallel_coarse_grid_solver
> -pc_mg_monitor
> -pc_type gamg
> -prop_steps 8
> -ts_monitor
> -ts_type cn
> #End of PETSc Option Table entries
>
> With this I get a grid complexity of 1.33047, 6 multigrid levels,
> MatPtAP/KSPSolve ratio of 0.24, and the linear solve at each TS step takes
> 5 iterations (with approx one order of magnitude reduction in residual per
> step for iterations 2 through 5 and two orders for the first). The
> convergence and grid complexity look good, but the ratio of grid coarsening
> time to ksp-solve time is far from ideal. I've attached the log file from
> this set of base parameters as well.
>
> To investigate the effect of coarsening rates, I ran a parameter sweep over
> the coarsening parameters (threshold and sq. graph) and I'm confused by the
> results. For some reason either the number of gamg levels turns out to be
> too high or it is set to 1. When I try to manually set the number of levels
> to 4 (with pc_mg_levels 4 and thres. -0.04/ squaring of 10) I see
> performance much worse than the base parameters. Any advice as to what I'm
> missing in my search for a set of params where MatPtAP to KSPSolve is ~ 1 ?

Your solver looks efficient and the time to setup roughly matches the
solve time:

PCSetUp                8 1.0 1.2202e+02 1.0 4.39e+09 1.0 4.9e+05 6.5e+03 6.3e+02 36 12 19 27 21  36 12 19 27 22  9201
PCApply               40 1.0 1.1077e+02 1.0 2.63e+10 1.0 2.0e+06 3.8e+03 2.0e+03 33 72 79 65 68  33 72 79 65 68 60662

If you have a specific need to reduce setup time or reduce solve time
(e.g., if you'll do many solves with the same setup), you might be able
to adjust.  But your iteration count is pretty low so probably not a lot
of room in that direction.