[petsc-users] Expected weak scaling behaviour for AMG libraries?

[Mark Adams]

As Matt said snes ex56 is better because it does a convergence test that
refines the grid. You need/want these two parameters to have the same arg
(eg, 2,2,1): -dm_plex_box_faces 2,2,1 -petscpartitioner_simple_process_grid
2,2,1.
This will put one cell per process.

Then you use: -max_conv_its N, to specify the N levels of refinement to do.
It will run the 2,2,1 first then a 4,4,2, etc., N times.

/src/snes/tests/ex13.c is designed for benchmarking and it uses
'-petscpartitioner_simple_node_grid 1,1,1 [default]' to give you a two
level partitioner.
You need to have dm_plex_box_faces_i =
petscpartitioner_simple_process_grid_i  *
petscpartitioner_simple_node_grid_i
Again, you should put one cell per process (NP = product of
dm_plex_box_faces args) and use -dm_refine N to get a single solve.

Mark



On Wed, Oct 30, 2024 at 11:02 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 30, 2024 at 4:13 PM Khurana, Parv <p.khurana22 at imperial.ac.uk>
> wrote:
>
>> Hello PETSc Community,
>> I am trying to understand the scaling behaviour of AMG methods in PETSc
>> (Hypre for now) and how many DOFs/Rank are needed for a performant AMG
>> solve.
>> I’m currently conducting weak scaling tests using
>> src/snes/tutorials/ex12.c in 3D, applying Dirichlet BCs with FEM at P=1.
>> The tests keep DOFs per processor constant while increasing the mesh size
>> and processor count, specifically:
>>
>>    - *20000 and 80000 DOF/RANK* configurations.
>>    - Running SNES twice, using GMRES with a tolerance of 1e-5 and
>>    preconditioning with Hypre-BoomerAMG.
>>
>> A couple of quick points  in order to make sure that there is no
> confusion:
>
> 1) Partitioner type "simple" is for the CI. It is a very bad partition,
> and should not be used for timing. The default is ParMetis which should be
> good enough.
>
> 2) You start out with 6^3 = 216 elements, distribute that, and then refine
> it. This will be _really_ bad load balance on all arrangement except the
> divisors of 216. You usually want to start out with something bigger at the
> later stages. You can use -dm_refine_pre to refine before distribution.
>
> 3) It is not clear you are using the timing for just the solver
> (SNESSolve). It could be that extraneous things are taking time. When
> asking questions like this, please always send the output of -log_view for
> timing, and at least -ksp_monitor_true_residial for convergence.
>
> 4) SNES ex56 is the example we use for GAMG scalability testing
>
>   Thanks,
>
>       Matt
>
>> Unfortunately, parallel efficiency degrades noticeably with increased
>> processor counts. Are there any insights or rules of thumb for using AMG
>> more effectively? I have been looking at this issue for a while
>> now and would love to engage in a further discussion. Please find below the
>> weak scaling results and the options I use to run the tests.
>> *#Run type*
>> -run_type full
>> -petscpartitioner_type simple
>>
>> *#Mesh settings*
>> -dm_plex_dim 3
>> -dm_plex_simplex 1
>> -dm_refine 5 #Varied this
>> -dm_plex_box_faces 6,6,6
>>
>> *#BCs and FEM space*
>> -bc_type dirichlet
>> -petscspace_degree 1
>>
>> *#Solver settings*
>> -snes_max_it 2
>> -ksp_type gmres
>> -ksp_rtol 1.0e-5
>> #Same settings as what we use for LOR
>> -pc_type hypre
>> -pc_hypre_type boomeramg
>> -pc_hypre_boomeramg_coarsen_type hmis
>> -pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi
>> -pc_hypre_boomeramg_strong_threshold 0.7
>> -pc_hypre_boomeramg_interp_type ext+i
>> -pc_hypre_boomeramg_P_max 2
>> -pc_hypre_boomeramg_truncfactor 0.3
>>
>> Best,
>> Parv
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cMOkz8-C-z7mPu1_ju5K0omHBE-cuHEm-yHUPTRQIOlOoAPQGVBhPIB-FD1VU_y4lJ1LMbJE6S3RyDw3TSCCdDg$ 
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YXR-8qKioRYS0fNOHacYGkm6WaIuKge2zoTiW1n0vLsWQUBiyLM48cg58pRLtNm0QjVigIZYftn2x09fmjiN$>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20241031/29c2a2bd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 119488 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20241031/29c2a2bd/attachment-0001.png>