<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> It is essentially doing AMR each time-step, and for the given application, I don't think that is pathological. It is an app built on Randy LeVeque's Clawpack stuff. The linear solver totally dominates the time which makes users of Clawpack very hesitant to consider methods with implicit steps, even when they need them. One linear solution takes far longer than the entire explicit step including its sub-grid cycling and the adaptive changes to the grid at each time-step.<div class=""><br class=""></div><div class=""> There is sub-grid cycling where similar solves are done on the same mesh (hence same nonzero pattern) two or three times, so thanks, we will try -pc_gamg_reuse_interpolation true and it could potentially help a good amount.</div><div class=""><br class=""></div><div class=""> There could be a mode where -pc_gamg_reuse_interpolation true is on by default, and the KSP/PC monitors the performance (convergence rate) of the solve following the reuse to decide if sticking with the old interpolation is ok or if a new interpolation should be done for the next change in the matrix values. Thus not requiring user knowledge and tuning of this option which most users who just want to get on with their work would not want to mess with.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 17, 2022, at 1:43 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I don't see a problem here other than the network looks bad relative to the problem size.<div class=""><br class=""></div><div class="">All the graph methods (PCGAMGCreateG and MIS) are 2x slower.</div><div class=""> - THe symmetrization must be in PCGAMGCreateG.</div><div class=""> - MIS is pretty old code (the algorithm and original code are 25 years old)</div><div class="">RAPs are about the same.</div><div class="">KSPGMRESOrthog and MatMult are nowhere near perfect.<br class=""></div><div class=""><br class=""></div><div class="">The graph setup work gets amortized by (most) applications and benchmarkers that know how to benchmark, so it is not highly engineered like the RAP and MatMult.</div><div class="">Note, this application is building the graph work for every linear solve.</div><div class="">I am guessing they want '-pc_gamg_reuse_interpolation true' or are doing a single step/stage TS with a linear problem and AMR every time step, which would be pretty pathological.</div><div class="">I'm doing an MR right now, maybe I should change the default for -pc_gamg_reuse_interpolation?</div><div class=""><br class=""></div><div class="">Mark</div><div class=""><br class=""></div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 17, 2022 at 10:12 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> Sure, but have you ever seen such a large jump in time in going from one to two MPI ranks, and are there any algorithms to do the aggregation that would not require this very expensive parallel symmetrization?<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Sep 17, 2022, at 9:07 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Symetrix graph make a transpose and then adds them.<div class="">I imagine adding two different matrices is expensive.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 16, 2022 at 8:30 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br class="">
Mark,<br class="">
<br class="">
I have runs of GAMG on one and two ranks with -pc_gamg_symmetrize_graph because the matrix is far from symmetric and some of GAMG is taking a huge amount more time with 2 ranks than one. (While other stuff like VecNorm shows improvement with two ranks). I've attached the two files<br class="">
<br class="">
Have you seen this before, is there anything that can be done about? If going to two ranks causes almost a doubling in GAMG setup time that makes using parallelism not useful,<br class="">
<br class="">
Barry<br class="">
<br class="">
</blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</div></blockquote></div><br class=""></div></body></html>