[petsc-users] performance regression with GAMG

[Pierre Jolivet]

> On 11 Aug 2023, at 1:14 AM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> BTW, nice bug report ...
>> 
>> So in the first step it coarsens from 150e6 to 5.4e6 DOFs instead of to 
>> 2.6e6 DOFs.
> 
> Yes, this is the critical place to see what is different and going wrong.
> 
> My 3D tests were not that different and I see you lowered the threshold.
> Note, you can set the threshold to zero, but your test is running so much differently than mine there is something else going on.
> Note, the new, bad, coarsening rate of 30:1 is what we tend to shoot for in 3D.
> 
> So it is not clear what the problem is.  Some questions:
> 
> * do you have a picture of this mesh to show me?
> * what do you mean by Q1-Q2 elements?
> 
> It would be nice to see if the new and old codes are similar without aggressive coarsening.
> This was the intended change of the major change in this time frame as you noticed.
> If these jobs are easy to run, could you check that the old and new versions are similar with "-pc_gamg_square_graph  0 ",  ( and you only need one time step).
> All you need to do is check that the first coarse grid has about the same number of equations (large).
> 
> BTW, I am starting to think I should add the old method back as an option. I did not think this change would cause large differences.

Not op, but that would be extremely valuable, IMHO.
This is impacting codes left, right, and center (see, e.g., another research group left wondering https://github.com/feelpp/feelpp/issues/2138).

Mini-rant: as developers, we are being asked to maintain backward compatibility of the API/headers, but there is no such an enforcement for the numerics.
A breakage in the API is “easy” to fix, you get a compilation error, you either try to fix your code or stick to a lower version of PETSc.
Changes in the numerics trigger silent errors which are much more delicate to fix because users do not know whether something needs to be addressed in their code or if there is a change in PETSc.
I don’t see the point of enforcing one backward compatibility but not the other.

Thanks,
Pierre

> Thanks,
> Mark
> 
> 
>  
>> Note that we are providing the rigid body near nullspace, 
>> hence the bs=3 to bs=6.
>> We have tried different values for the gamg_threshold but it doesn't 
>> really seem to significantly alter the coarsening amount in that first step.
>> 
>> Do you have any suggestions for further things we should try/look at? 
>> Any feedback would be much appreciated
>> 
>> Best wishes
>> Stephan Kramer
>> 
>> Full logs including log_view timings available from 
>> https://github.com/stephankramer/petsc-scaling/
>> 
>> In particular:
>> 
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_5/output_2.dat
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_5/output_2.dat
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_6/output_2.dat
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_6/output_2.dat
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_7/output_2.dat
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_7/output_2.dat 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230811/cd627ffc/attachment-0001.html>