[petsc-dev] Bad scaling of GAMG in FieldSplit

[Jed Brown]

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, Jul 26, 2018 at 2:43 PM Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>>
>> > On Thu, Jul 26, 2018 at 12:56 PM Fande Kong <fdkong.jd at gmail.com> wrote:
>> >
>> >>
>> >>
>> >> On Thu, Jul 26, 2018 at 10:35 AM, Junchao Zhang <jczhang at mcs.anl.gov>
>> >> wrote:
>> >>
>> >>> On Thu, Jul 26, 2018 at 11:15 AM, Fande Kong <fdkong.jd at gmail.com>
>> wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>> On Thu, Jul 26, 2018 at 9:51 AM, Junchao Zhang <jczhang at mcs.anl.gov>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi, Pierre,
>> >>>>>   From your log_view files, I see you did strong scaling. You used 4X
>> >>>>> more cores, but the execution time only dropped from 3.9143e+04
>> >>>>> to 1.6910e+04.
>> >>>>>   From my previous analysis of a GAMG weak scaling test, it looks
>> >>>>> communication is one of the reasons that caused poor scaling.  In
>> your
>> >>>>> case,  VecScatterEnd time was doubled from 1.5575e+03 to 3.2413e+03.
>> Its
>> >>>>> time percent jumped from 1% to 17%. This time can contribute to the
>> big
>> >>>>> time ratio in MatMultAdd ant MatMultTranspose, misleading you guys
>> thinking
>> >>>>> there was load-imbalance computation-wise.
>> >>>>>   The reason is that I found in the interpolation and restriction
>> >>>>> phases of gamg, the communication pattern is very bad. Few processes
>> >>>>> communicate with hundreds of neighbors with message sizes of a few
>> bytes.
>> >>>>>
>> >>>>
>> >>>> We may need to truncate interpolation/restriction operators. Also do
>> >>>> some aggressive coarsening.  Unfortunately, GAMG currently does not
>> support.
>> >>>>
>> >>>
>> >>>  Are these gamg options the truncation you thought?
>> >>>
>> >>
>> >>> -pc_gamg_threshold[] <thresh,default=0> - Before aggregating the graph
>> >>> GAMG will remove small values from the graph on each level
>> >>> -pc_gamg_threshold_scale <scale,default=1> - Scaling of threshold on
>> each
>> >>> coarser grid if not specified
>> >>>
>> >>
>> >> Nope.  Totally different things.
>> >>
>> >
>> > Well, you could use _threshold to do more aggressive coarsening, but not
>> > for thinning out
>> > the interpolation.
>>
>> Increasing the threshold results in slower coarsening.
>>
>
> Hmm, I think we have to change the webpage then:
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>
> I read it the opposite way.

More coarse points is "better" (stronger), but higher complexity.