[petsc-users] PETSc 3.14 to PETSc 3.20: Different (slower) convergence for classical AMG (sequential and especially in parallel)

[LEDAC Pierre]

Hello Mark,


Thanks for your answer. Indeed, I didn't see the information that classical AMG was not really supported:


 -solver2_pc_gamg_type <now classical : formerly agg>: Type of AMG method (only 'agg' supported and useful) (one of) classical geo agg (PCGAMGSetType)

We switched very recently from GAMG("agg") to GAMG("classical") for a weak scaling test up to 32000 cores, where we saw very good scalability with GAMG("classical") compared to GAMG("agg"). But it was with PETSc 3.14...

So today, we are going to upgrade to 3.20 and focus on GAMG("agg") or Hypre Classical AMG. We will see how it compares.

May I ask you what is your point of view of the current state of the GPU versions of GAMG("agg") versus Hypre AMG Classical ?

In fact, the reason of our move from 3.14 to 3.20 is to take advantage of all the progress in PETSc and Hypre on accelerated solvers/preconditioners during the last 2 years.

Greatly appreciate your help,


Pierre LEDAC
Commissariat à l’énergie atomique et aux énergies alternatives
Centre de SACLAY
DES/ISAS/DM2S/SGLS/LCAN
Bâtiment 451 – point courrier n°43
F-91191 Gif-sur-Yvette
+33 1 69 08 04 03
+33 6 83 42 05 79
________________________________
De : Mark Adams <mfadams at lbl.gov>
Envoyé : mercredi 13 décembre 2023 20:54:17
À : LEDAC Pierre
Cc : petsc-users at mcs.anl.gov; BRUNETON Adrien
Objet : Re: [petsc-users] PETSc 3.14 to PETSc 3.20: Different (slower) convergence for classical AMG (sequential and especially in parallel)

Hi Pierre,

Sorry I missed this post and your issues were brought to my attention today.

First, the classic version is not supported well. The postdoc that wrote the code is long gone and I don't know the code at all.
It is really a reference implementation that someone could build on and is not meant for production.
In 10 years you are the first user that has connected us.

The hypre package is a very good AMG solver and it uses classical AMG as the main solver.
I wrote GAMG ("agg") which is a smoothed aggregation AMG solver and is very different from classical.
I would suggest you move to hypre or '-pc_gamg_type agg'.

The coarsening was developed in this time frame and there was a lot of churn as a new strategy for aggressive coarsening did not work well for some users and I had to add the old method in and then made it the default (again).
This change missed v3.20, but you can get the old aggressive strategy with '-pc_gamg_aggressive_square_graph'.
Check with -options_left to check that it is being used.

As far as your output (nice formatting, thank you), the coarse grid is smaller in the new code.
            rows=41, cols=41   |          rows=30, cols=30
"square graph" should fix this.

You can also try not using aggressive coarsening with:
You could try '-pc_gamg_aggressive_coarsening 0'

Let me know how it goes and let's try to get you into a more sustainable state ... I really try not to change this code but sometimes need to.

Thanks,
Mark





On Mon, Oct 9, 2023 at 10:43 AM LEDAC Pierre <Pierre.LEDAC at cea.fr<mailto:Pierre.LEDAC at cea.fr>> wrote:

Hello all,


I am struggling to find the same convergence in iterations when using classical algebric multigrid in my code with PETSc 3.20 compared to PETSc 3.14.


I am using in order to solve a Poisson system:

-ksp_type cg -pc_type gamg -pc_gamg_type classical


I read the different releases notes between 3.15 and 3.20:

https://petsc.org/release/changes/317

https://petsc.org/main/manualpages/PC/PCGAMGSetThreshold/


And have a look at the archive mailing list (especially this one: https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg46688.html)

so I added some other options to try to have the same behaviour than PETSc 3.14:


-ksp_type cg -pc_type gamg -pc_gamg_type classical -mg_levels_pc_type sor -pc_gamg_threshold 0.


It improves the convergence but there still a different convergence though (26 vs 18 iterations).

On another of my test case, the number of levels is different (e.g. 6 vs 4) also, and here it is the same, but with a different coarsening according to the output from the -ksp_view option


The main point is that the convergence dramatically degrades in parallel on a third test case, so I can't upgrade to PETSc 3.20 for now unhappily.


I send you the partial report (petsc_314_vs_petsc_320.ksp_view) with -ksp_view (left PETSc 3.14, right PETSc 3.20) and the configure/command line options used (in petsc_XXX_petsc.TU files).


Could my issue related to the following 3.18 change ? I have not tried the first one.

  *   Remove PCGAMGSetSymGraph() and -pc_gamg_sym_graph. The user should now indicate symmetry and structural symmetry using MatSetOption<https://petsc.org/release/manualpages/Mat/MatSetOption/>() and GAMG will symmetrize the graph if a symmetric options is not set

  *   Change -pc_gamg_reuse_interpolation default from false to true.


Any advice would be greatly appreciated,


Pierre LEDAC
Commissariat à l’énergie atomique et aux énergies alternatives
Centre de SACLAY
DES/ISAS/DM2S/SGLS/LCAN
Bâtiment 451 – point courrier n°43
F-91191 Gif-sur-Yvette
+33 1 69 08 04 03
+33 6 83 42 05 79
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231214/3c3d307f/attachment-0001.html>