[petsc-users] performance regression with GAMG

[Mark Adams]

I will add the old method back in.

And I did touch the MIS code and graph creation, which turns a bs>1 matrix
into a bs==1 matrix/graph.
I don't think I changed the semantics but I did not verify that carefully.

In thinking more about this error I realize that this test has thin body
elements, probably, and the threshold can make a big difference here.
The threshold is used in a slightly different way in the new algorithm, and
there is no avoiding that.
Stephan did play around with the threshold but it would be nice to see if
the old and new code are still (very) different with zero or < 0 threshold
(< 0 will keep zero entries like BCs).

I am still puzzled why the new code was so different, given that there does
not seem to be some sensitivity to threshold.
I'd like to figure this out because the old method can be very slow and use
a lot of memory for full 3D problems.

Thanks,
Mark


On Fri, Aug 11, 2023 at 12:48 AM Pierre Jolivet <pierre.jolivet at lip6.fr>
wrote:

>
>
> On 11 Aug 2023, at 1:14 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
> BTW, nice bug report ...
>
>>
>> So in the first step it coarsens from 150e6 to 5.4e6 DOFs instead of to
>> 2.6e6 DOFs.
>
>
> Yes, this is the critical place to see what is different and going wrong.
>
> My 3D tests were not that different and I see you lowered the threshold.
> Note, you can set the threshold to zero, but your test is running so much
> differently than mine there is something else going on.
> Note, the new, bad, coarsening rate of 30:1 is what we tend to shoot for
> in 3D.
>
> So it is not clear what the problem is.  Some questions:
>
> * do you have a picture of this mesh to show me?
> * what do you mean by Q1-Q2 elements?
>
> It would be nice to see if the new and old codes are similar without
> aggressive coarsening.
> This was the intended change of the major change in this time frame as you
> noticed.
> If these jobs are easy to run, could you check that the old and new
> versions are similar with "-pc_gamg_square_graph  0 ",  ( and you only need
> one time step).
> All you need to do is check that the first coarse grid has about the same
> number of equations (large).
>
> BTW, I am starting to think I should add the old method back as an option.
> I did not think this change would cause large differences.
>
>
> Not op, but that would be extremely valuable, IMHO.
> This is impacting codes left, right, and center (see, e.g., another
> research group left wondering https://github.com/feelpp/feelpp/issues/2138
> ).
>
> Mini-rant: as developers, we are being asked to maintain backward
> compatibility of the API/headers, but there is no such an enforcement for
> the numerics.
> A breakage in the API is “easy” to fix, you get a compilation error, you
> either try to fix your code or stick to a lower version of PETSc.
> Changes in the numerics trigger silent errors which are much more delicate
> to fix because users do not know whether something needs to be addressed in
> their code or if there is a change in PETSc.
> I don’t see the point of enforcing one backward compatibility but not the
> other.
>
> Thanks,
> Pierre
>
> Thanks,
> Mark
>
>
>
>
>> Note that we are providing the rigid body near nullspace,
>> hence the bs=3 to bs=6.
>> We have tried different values for the gamg_threshold but it doesn't
>> really seem to significantly alter the coarsening amount in that first
>> step.
>>
>> Do you have any suggestions for further things we should try/look at?
>> Any feedback would be much appreciated
>>
>> Best wishes
>> Stephan Kramer
>>
>> Full logs including log_view timings available from
>> https://github.com/stephankramer/petsc-scaling/
>>
>> In particular:
>>
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_5/output_2.dat
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_5/output_2.dat
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_6/output_2.dat
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_6/output_2.dat
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/before/Level_7/output_2.dat
>>
>> https://github.com/stephankramer/petsc-scaling/blob/main/after/Level_7/output_2.dat
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230812/2c9cdd8b/attachment.html>