<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 4, 2016, at 9:26 PM, Boyce Griffith <<a href="mailto:griffith@cims.nyu.edu" class="">griffith@cims.nyu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class="Apple-interchange-newline">On Aug 4, 2016, at 9:01 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="Apple-interchange-newline">On Aug 4, 2016, at 8:51 PM, Boyce Griffith <<a href="mailto:griffith@cims.nyu.edu" class="">griffith@cims.nyu.edu</a>> wrote:<br class=""><br class=""><blockquote type="cite" class=""><br class="">On Aug 4, 2016, at 8:42 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:<br class=""><br class=""><br class="">History,<br class=""><br class="">1) I originally implemented the ASM with one subdomain per process<br class="">2) easily extended to support multiple domain per process<br class="">3) added -pc_asm_type restrict etc but it only worked for one subdomain per process because it took advantage of the fact that<span class="Apple-converted-space"> </span><br class=""> restrict etc could be achieved by simply dropping the parallel communication in the vector scatters<br class="">4) Matt didn't like the restriction to one process per subdomain so he added an additional argument to PCASMSetLocalSubdomains() that allowed passing in the overlapping and non-overlapping regions of each domain (foolishly calling the non-overlapping index set is_local even though local has nothing to do with), so that the restrict etc could be handled.<br class=""><br class="">Unfortunately IMHO Matt made a mess of things because if you use things like -pc_asm_blocks n or -pc_asm_overlap 1 etc it does not handle the -pc_asm_type restrict since it cannot track the is vs is_local. The code needs to be refactored so that things like -pc_asm_blocks and -pc_asm_overlap 1 can track the is vs is_local index sets properly when the -pc_asm_type is set. Also the name is_local needs to be changed to something meaningfully like is_nonoverlapping This refactoring would also result in easier cleaner code then is currently there.<br class=""><br class="">So basically until the PCASM is refactored properly to handle restrict etc you are stuck with being able to use the restrict etc ONLY if you specifically supply the overlapping and non overlapping domains yourself with PCASMSetLocalSubdomains and curse at Matt everyday like we all do.<br class=""></blockquote><br class="">OK, got it. The reason I’m asking is that we are using PCASM in a custom smoother, and I noticed that basic/restrict/interpolate/none all give identical results. We are using PCASMSetLocalSubdomains to set up the subdomains.<br class=""></blockquote><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;"> But are you setting different is and is_local (stupid name) and not have PETSc computing the overlap in your custom code? If you are setting them differently and not having PETSc compute overlap but getting identical convergence then something is wrong and you likely have to run in the debugger to insure that restrict etc is properly being set and used.</span><br class=""></div></blockquote><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Yes we are computing overlapping and non-overlapping IS’es.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">I just double-checked, and somehow the ASMType setting is not making it from the command line into the solver configuration — sorry, I should have checked this more carefully before emailing the list. (I thought that the command line options were being captured correctly, since I am able to control the PC type and all of the sub-KSP/sub-PC settings.)</div></div></blockquote><div><br class=""></div><div>OK, so here is what appears to be happening. These solvers are named things like “stokes_pc_level_0_”, “stokes_pc_level_1_”, … . If I use the command-line argument</div><div><br class=""></div><div><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: rgb(245, 245, 245); font-family: Menlo; font-size: 13px; background-color: rgb(0, 0, 0);" class="">-stokes_ib_pc_level_0_pc_asm_type basic</span></div><div><br class=""></div><div>then the ASM settings are used, but if I do:</div><div><br class=""></div><div><div><span class="Apple-tab-span" style="white-space: pre;"> </span><span style="color: rgb(245, 245, 245); font-family: Menlo; font-size: 13px; background-color: rgb(0, 0, 0);" class="">-stokes_ib_pc_level_pc_asm_type basic</span></div></div><div><br class=""></div>they are ignored. Any ideas? :-)</div><div><br class=""></div><div>Thanks,</div><div><br class=""></div><div>— Boyce</div><div><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">BTW, there is also this bit (which was easy to overlook in all of the repetitive convergence histories):<br class=""></blockquote><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;"> Yeah, better one question per email or we will miss them.</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;"> There is nothing that says that multiplicative will ALWAYS beat additive, though intuitively you expect it to.</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></div></blockquote><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">OK, so similar story as above: we have a custom MSM that, when used as a MG smoother, gives convergence rates that are about 2x PCASM, whereas when we use PCASM with MULTIPLICATIVE, it doesn’t seem to help.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> </div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">However, now I am questioning whether the settings are getting propagated into PCASM… I’ll need to take another look.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Thanks,</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">— Boyce</div><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;"> Barry</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Also, the MULTIPLICATIVE variant does not seem to behave as I would expect --- for this same example, if you switch from ADDITIVE to MULTIPLICATIVE, the solver converges slightly more slowly:<br class=""><br class="">$ ./ex2 -m 32 -n 32 -pc_type asm -pc_asm_blocks 8 -ksp_view -ksp_monitor_true_residual -pc_asm_local_type MULTIPLICATIVE<br class="">0 KSP preconditioned resid norm 7.467363913958e+00 true resid norm 1.166190378969e+01 ||r(i)||/||b|| 1.000000000000e+00<br class="">1 KSP preconditioned resid norm 2.878371937592e+00 true resid norm 3.646367718253e+00 ||r(i)||/||b|| 3.126734522949e-01<br class="">2 KSP preconditioned resid norm 1.666575161021e+00 true resid norm 1.940699059619e+00 ||r(i)||/||b|| 1.664135714560e-01<br class="">3 KSP preconditioned resid norm 1.086140238220e+00 true resid norm 1.191473615464e+00 ||r(i)||/||b|| 1.021680196433e-01<br class="">4 KSP preconditioned resid norm 7.939217314942e-01 true resid norm 8.059317628307e-01 ||r(i)||/||b|| 6.910807852344e-02<br class="">5 KSP preconditioned resid norm 6.265169154675e-01 true resid norm 5.942294290555e-01 ||r(i)||/||b|| 5.095475316653e-02<br class="">6 KSP preconditioned resid norm 5.164999302721e-01 true resid norm 4.585844476718e-01 ||r(i)||/||b|| 3.932329197203e-02<br class="">7 KSP preconditioned resid norm 4.472399844370e-01 true resid norm 3.884049472908e-01 ||r(i)||/||b|| 3.330544946136e-02<br class="">8 KSP preconditioned resid norm 3.445446366213e-01 true resid norm 4.008290378967e-01 ||r(i)||/||b|| 3.437080644166e-02<br class="">9 KSP preconditioned resid norm 1.987509894375e-01 true resid norm 2.619628925380e-01 ||r(i)||/||b|| 2.246313271505e-02<br class="">10 KSP preconditioned resid norm 1.084551743751e-01 true resid norm 1.354891040098e-01 ||r(i)||/||b|| 1.161809481995e-02<br class="">11 KSP preconditioned resid norm 6.108303419460e-02 true resid norm 7.252267103275e-02 ||r(i)||/||b|| 6.218767736436e-03<br class="">12 KSP preconditioned resid norm 3.641579250431e-02 true resid norm 4.069996187932e-02 ||r(i)||/||b|| 3.489992938829e-03<br class="">13 KSP preconditioned resid norm 2.424898818735e-02 true resid norm 2.469590201945e-02 ||r(i)||/||b|| 2.117656127577e-03<br class="">14 KSP preconditioned resid norm 1.792399391125e-02 true resid norm 1.622090905110e-02 ||r(i)||/||b|| 1.390931475995e-03<br class="">15 KSP preconditioned resid norm 1.320657155648e-02 true resid norm 1.336753101147e-02 ||r(i)||/||b|| 1.146256327657e-03<br class="">16 KSP preconditioned resid norm 7.398524571182e-03 true resid norm 9.747691680405e-03 ||r(i)||/||b|| 8.358576657974e-04<br class="">17 KSP preconditioned resid norm 3.043993613039e-03 true resid norm 3.848714422908e-03 ||r(i)||/||b|| 3.300245390731e-04<br class="">18 KSP preconditioned resid norm 1.767867968946e-03 true resid norm 1.736586340170e-03 ||r(i)||/||b|| 1.489110501585e-04<br class="">19 KSP preconditioned resid norm 1.088792656005e-03 true resid norm 1.307506936484e-03 ||r(i)||/||b|| 1.121177948355e-04<br class="">20 KSP preconditioned resid norm 4.622653682144e-04 true resid norm 5.718427718734e-04 ||r(i)||/||b|| 4.903511315013e-05<br class="">21 KSP preconditioned resid norm 2.591703287585e-04 true resid norm 2.690982547548e-04 ||r(i)||/||b|| 2.307498497738e-05<br class="">22 KSP preconditioned resid norm 1.596527181997e-04 true resid norm 1.715846687846e-04 ||r(i)||/||b|| 1.471326396435e-05<br class="">23 KSP preconditioned resid norm 1.006766623019e-04 true resid norm 1.044525361282e-04 ||r(i)||/||b|| 8.956731080268e-06<br class="">24 KSP preconditioned resid norm 5.349814270060e-05 true resid norm 6.598682341705e-05 ||r(i)||/||b|| 5.658323427037e-06<br class="">KSP Object: 1 MPI processes<br class="">type: gmres<br class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> GMRES: happy breakdown tolerance 1e-30<br class="">maximum iterations=10000, initial guess is zero<br class="">tolerances: relative=9.18274e-06, absolute=1e-50, divergence=10000.<br class="">left preconditioning<br class="">using PRECONDITIONED norm type for convergence test<br class="">PC Object: 1 MPI processes<br class="">type: asm<br class=""> Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1<br class=""> Additive Schwarz: restriction/interpolation type - BASIC<br class=""> Additive Schwarz: local solve composition type - MULTIPLICATIVE<br class=""> Local solve is same for all blocks, in the following KSP and PC objects:<br class=""> KSP Object: (sub_) 1 MPI processes<br class=""> type: preonly<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class=""> left preconditioning<br class=""> using NONE norm type for convergence test<br class=""> PC Object: (sub_) 1 MPI processes<br class=""> type: icc<br class=""> 0 levels of fill<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> using Manteuffel shift [POSITIVE_DEFINITE]<br class=""> matrix ordering: natural<br class=""> factor fill ratio given 1., needed 1.<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqsbaij<br class=""> rows=160, cols=160<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=443, allocated nonzeros=443<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> block size is 1<br class=""> linear system matrix = precond matrix:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=160, cols=160<br class=""> total: nonzeros=726, allocated nonzeros=726<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class="">linear system matrix = precond matrix:<br class="">Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=1024, cols=1024<br class=""> total: nonzeros=4992, allocated nonzeros=5120<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class="">Norm of error 0.000292304 iterations 24<br class=""><br class="">Thanks,<br class=""><br class="">-- Boyce</blockquote></blockquote></blockquote></div></blockquote></div></blockquote></div><br class=""></body></html>