<div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2018-09-20 12:00 GMT+02:00 Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><br><div class="gmail_quote"><span class="gmail-"><div dir="ltr">On Wed, Sep 19, 2018 at 7:44 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
Look at the code in KSPSolve_Chebyshev(). <br>
<br>
Problem 1) VERY MAJOR<br>
<br>
Once you start running the eigenestimates it always runs them, this is because the routine begins with<br>
<br>
if (cheb->kspest) {<br>
<br>
but once cheb->kspest is set it is never unset. This means, for example, that every time PCMG runs the smoother that uses Chebyshev it runs the eigenestimator (which uses GMRES) (even when it is suppose to be just smoothing since the eigenestimates were already made in the setup stage). This is totally wrong. </blockquote><div><br></div></span><div>Yikes, does this code (a few lines down) address this?</div><div><br></div><div>if (amatid != cheb->amatid || pmatid != cheb->pmatid || amatstate != cheb->amatstate || pmatstate != cheb->pmatstate) {<br></div><div><br></div><div>Maybe you could run with CG as the outer solver and check that the number of GMRES solve calls (maybe with GMRESOrtho/max_it) is equal to the number of SNES iterations * (number of levels - 1).<br></div></div></div></div></div></blockquote><div><br></div><div>If I run this</div><div><br></div><div> ./ex19 -snes_monitor -ksp_view -pc_type mg -ksp_type gcr -pc_mg_levels 2</div><div><br></div><div>then I have one Chebyshev smoother (fine grid) and 2 SNES iterations, so I'd expect 2 calls to KSPSolve_GMRES. That is what I see when I set a breakpoint in gdb for KSPSolve_GMRES.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div></div><div><div class="gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Sure enough, if I run, for example, src/snes/examples/tutorials/<wbr>ex19.c with -pc_type gamg I see in the debugger that GMRES is being called by KSPSolve_Chebyshev as it smooths. For example,<br>
<br>
0 MatSOR (mat=0x28689f0, b=0x29ee310, omega=1, flag=28, shift=0, its=1, lits=1, x=0x29f4070)<br>
at /sandbox/bsmith/petsc/src/mat/<wbr>interface/matrix.c:3913<br>
#1 0x00007f59d2e353b9 in PCApply_SOR (pc=0x29aa770, x=0x29ee310, y=0x29f4070)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/impls/sor/sor.c:31<br>
#2 0x00007f59d2fa6a7b in PCApply (pc=0x29aa770, x=0x29ee310, y=0x29f4070)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/interface/precon.c:462<br>
#3 0x00007f59d2faa6a7 in PCApplyBAorAB (pc=0x29aa770, side=PC_LEFT, x=0x29f11c0, y=0x29f4070, work=0x29ee310)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/interface/precon.c:691<br>
#4 0x00007f59d3084d46 in KSP_PCApplyBAorAB (ksp=0x29c4d30, x=0x29f11c0, y=0x29f4070, w=0x29ee310)<br>
at /sandbox/bsmith/petsc/include/<wbr>petsc/private/kspimpl.h:309<br>
#5 0x00007f59d3086874 in KSPGMRESCycle (itcount=0x7ffd3d60143c, ksp=0x29c4d30)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/impls/gmres/gmres.c:152<br>
#6 0x00007f59d3087352 in KSPSolve_GMRES (ksp=0x29c4d30) at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/impls/gmres/gmres.c:234<br>
#7 0x00007f59d30fae94 in KSPSolve (ksp=0x29c4d30, b=0x29dc900, x=0x29d9a30)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/interface/itfunc.c:780<br>
#8 0x00007f59d306a1e1 in KSPSolve_Chebyshev (ksp=0x29a9550) at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/impls/cheby/cheby.c:367<br>
#9 0x00007f59d30fae94 in KSPSolve (ksp=0x29a9550, b=0x28653d0, x=0x2906a70)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/interface/itfunc.c:780<br>
#10 0x00007f59d2f59042 in PCMGMCycle_Private (pc=0x2832fd0, mglevelsin=0x2944b88, reason=0x0)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/impls/mg/mg.c:20<br>
#11 0x00007f59d2f5e350 in PCApply_MG (pc=0x2832fd0, b=0x28653d0, x=0x2906a70)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/impls/mg/mg.c:377<br>
#12 0x00007f59d2fa6a7b in PCApply (pc=0x2832fd0, x=0x28653d0, y=0x2906a70)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>pc/interface/precon.c:462<br>
#13 0x00007f59d31242d7 in KSP_PCApply (ksp=0x27df750, x=0x28653d0, y=0x2906a70)<br>
at /sandbox/bsmith/petsc/include/<wbr>petsc/private/kspimpl.h:281<br>
#14 0x00007f59d31251ba in KSPInitialResidual (ksp=0x27df750, vsoln=0x28610d0, vt1=0x28ff7b0, vt2=0x2903450, <br>
vres=0x2906a70, vb=0x28653d0) at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/interface/itres.c:67<br>
#15 0x00007f59d30872ef in KSPSolve_GMRES (ksp=0x27df750) at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/impls/gmres/gmres.c:233<br>
#16 0x00007f59d30fae94 in KSPSolve (ksp=0x27df750, b=0x28653d0, x=0x28610d0)<br>
at /sandbox/bsmith/petsc/src/ksp/<wbr>ksp/interface/itfunc.c:780<br>
#17 0x00007f59d3291d32 in SNESSolve_NEWTONLS (snes=0x26f2550) at /sandbox/bsmith/petsc/src/<wbr>snes/impls/ls/ls.c:224<br>
#18 0x00007f59d320f7da in SNESSolve (snes=0x26f2550, b=0x0, x=0x285d440)<br>
<br></blockquote><div><br></div></div></div><div><div>Can this just be the first time it is called, so it is doing the setup?</div></div><span class="gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Not only is the code wrong but it is also a huge inefficiency in the code running all these unneeded GMRES.<br></blockquote><div><br></div></span><div>Just to be clear, this is inefficiency, but I don't see why it is (mathematically) wrong.</div><span class="gmail-"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
Problem 2) Less catastrophic <br>
<br>
When cheb->kspest is set the "regular" Chebyshev is also run (after the eigenvalues are estimated). </blockquote><div><br></div></span><div>I don't see that. I see:</div><div><br></div><div><div>static PetscErrorCode KSPSolve_Chebyshev(KSP ksp)</div><div>.....</div><div> if (cheb->kspest) {</div><div> .....</div><div> if (amatid != cheb->amatid || pmatid != cheb->pmatid || amatstate != cheb->amatstate || pmatstate != cheb->pmatstate) {</div></div><div> ... eig est</div><div> }</div><div> }</div><div> .. cheby ..</div><span class="gmail-"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">I am pretty sure this is wrong because there is no reason to run the Chebyshev when estimating the eigenvalues? The eigenvalues are stored to be used later right?</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
So what needs to be fixed? Well somehow cheb->kspest has to be turned off once the eigenvalues are computed, but turned back on each time the matrix changes. And (easier) when cheb->kspest is used then the actual Chebyshev iterations need to be turned off.<br>
<br>
I hope I am wrong and the code is correct but I'm pretty sure the code is not what it should be.<br>
<br>
Barry<br>
<br>
<br>
<br>
<br>
</blockquote></span></div></div></div></div>
</blockquote></div><br></div></div></div></div>