[petsc-dev] Bad scaling of GAMG in FieldSplit

[Pierre Jolivet]

> On 27 Jul 2018, at 3:02 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
>> 
>> Do you mean the coarse grids? GAMG reduces active processors (and repartitions the coarse grids if you ask it to) like telescope.
> 
> No I was talking about the fine grid. If I do this (telescope then GAMG),
> 
> What does Telescope do on the fine grids? Is this a structured grid and Telescope does geometric MG?

No, it’s a MatMPIAIJ. Telescope just concatenates local matrices together, makes some process idle (throughout the complete setup + solve), while the others do the work. This is just to ensure that the local workload is high enough so that the solver is still in a good regime. Of course, it also means that most of the processes are idling while GAMG is being called (thus why I’d like to overlap the two solves in my PCCOMPOSITE).

>  
> MatMultTranspose and MatMultAdd perform OK, but the smoother (MatSOR) takes more and more time as one would have guessed… it seems that there is no easy way to make this strong scale.
> 
> 
> Strong scaling will always stop at some point. The question is where. I'm not seeing any data on where it is stopping (I have not dug into the data files).
> 
> My first suspicion here was that GAMG stops coarsening and there are many coarse grids (ie, crazy behavior). I have seen this occasionally.
> 
> Also, did you check the iteration count in GAMG. I'm not sure if this is a convergence problem or a complexity problem.

Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed.

Thanks,
Pierre



>  
>> >> 2) have the sub_0_ and the sub_1_ work on two different nonoverlapping communicators of size PETSC_COMM_WORLD/2, do the solve concurrently, and then sum the solutions (only worth doing because of -pc_composite_type additive). I have no idea if this easily doable with PETSc command line arguments
>> > 
>> > 1) is the more flexible approach, as you have better control over the system sizes after 'telescoping’.
>> 
>> Right, but the advantage of 2) is that I wouldn't have one half or more of processes idling and I could overlap the solves of both subpc in the PCCOMPOSITE.
>> 
>> I’m attaching the -log_view for both runs (I trimmed some options).
>> 
>> Thanks for your help,
>> Pierre
>> 
>> 
>> > Best regards,
>> > Karli
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180727/2eb831e3/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gamg_view.log
Type: application/octet-stream
Size: 10451 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180727/2eb831e3/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180727/2eb831e3/attachment-0003.html>