<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 27 Jul 2018, at 3:02 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class="">Do you mean the coarse grids? GAMG reduces active processors (and repartitions the coarse grids if you ask it to) like telescope.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">No I was talking about the fine grid. If I do this (telescope then GAMG), </div></div></div></blockquote><div class=""><br class=""></div><div class="">What does Telescope do on the fine grids? Is this a structured grid and Telescope does geometric MG?</div></div></div></div></blockquote><div><br class=""></div><div>No, it’s a MatMPIAIJ. Telescope just concatenates local matrices together, makes some process idle (throughout the complete setup + solve), while the others do the work. This is just to ensure that the local workload is high enough so that the solver is still in a good regime. Of course, it also means that most of the processes are idling while GAMG is being called (thus why I’d like to overlap the two solves in my PCCOMPOSITE).</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><div class="">MatMultTranspose and MatMultAdd perform OK, but the smoother (MatSOR) takes more and more time as one would have guessed… it seems that there is no easy way to make this strong scale.</div><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Strong scaling will always stop at some point. The question is where. I'm not seeing any data on where it is stopping (I have not dug into the data files).</div><div class=""><br class=""></div><div class="">My first suspicion here was that GAMG stops coarsening and there are many coarse grids (ie, crazy behavior). I have seen this occasionally.</div><div class=""><br class=""></div><div class="">Also, did you check the iteration count in GAMG. I'm not sure if this is a convergence problem or a complexity problem.</div></div></div></div></blockquote><div><br class=""></div><div>Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed.</div><div><br class=""></div><div>Thanks,</div><div>Pierre</div><div><br class=""></div><div></div></div></body></html>