<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I tried to use these options with GAMG:<div class=""><br class=""><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class=""><div class="">-mg_levels_ksp_type richardson</div></div></div></blockquote><blockquote type="cite" class=""><div dir="ltr" class=""><div class=""><div class="">-mg_levels_pc_type sor</div></div></div></blockquote><div class=""><br class=""></div>but also in this case i don’t obtain a convergence.</div><div class=""><br class=""></div><div class="">I used also direct solvers (MUMPS ad SuperLU) and they works fine, but slower (solve time) then iterative method with ML and hypre.</div><div class=""><br class=""></div><div class="">If i check with -ksp_monitor_true_residual, is it advisable to continue using the iterative method?</div><div class=""><br class=""></div><div class="">Pierpaolo</div><div class=""><br class=""></div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">Il giorno 03 ago 2018, alle ore 15:16, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">So this is a complex valued indefinite Helmholtz operator (very hard to solve scalably) with axisymmetric coordinates. ML, hypre and GAMG all performed about the same, with a big jump in residual initially and essentially not solving it. You scaled it and this fixed ML and hypre but not GAMG.<div class=""><br class=""></div><div class="">From this output I can see that the eigenvalue estimates are strange. Your equations look fine so I have to assume that the complex values are the problem. If this is symmetric the CG is a much better solver and eigen estimator. But this is not a big deal, especially since you have two options that work. I would suggest not using cheby smoother, it uses these bad eigen estimates, and it is basically not smoothing on some levels. You can use this instead:</div><div class=""><br class=""></div><div class=""><div class="">-mg_levels_ksp_type richardson</div><div class="">-mg_levels_pc_type sor</div></div><div class=""><br class=""></div><div class="">Note, if you have a large shift these equations are very hard to solve iteratively and you should just use a direct solver. Direct solvers in 2D are not bad,</div><div class=""><br class=""></div><div class="">Mark</div><div class=""><br class=""></div></div><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div dir="ltr" class="">On Fri, Aug 3, 2018 at 3:02 AM Pierpaolo Minelli <<a href="mailto:pierpaolo.minelli@cnr.it" class="">pierpaolo.minelli@cnr.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""><div class="">In this simulation I'm solving two equations in a two-dimensional domain (z,r) at each time step. The first is an equation derived from the Maxwell equation. Taking advantage of the fact that the domain is axialsymmetric and applying a temporal harmonic approximation, the equation I am solving is the following:</div><div class=""><br class=""></div><div class=""><p class="MsoNormal"><span style="font-size: 12pt; font-family: Calibri, sans-serif;" class=""><img id="m_3908295385861418691B9E1FC4B-AB70-49FA-A0D9-7343BFEA458A" src="cid:clip_image002.png" class=""></span><span style="font-size: 16pt;" class=""><u class=""></u><u class=""></u></span></p></div><div class=""><br class=""></div><div class="">The second equation is a Poisson’s equation in cylindrical coordinates (z,r) in the field of real numbers.</div><div class=""><br class=""></div></div>This is the output obtained using these options (Note that at this moment of development I am only using a processor):<div class=""><br class=""></div><div class=""><b class="">-pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-7 -ksp_max_it 30 -ksp_monitor_true_residual -info | grep GAMG</b></div><div class=""><b class=""><br class=""></b></div><div class="">[0] PCSetUp_GAMG(): level 0) N=321201, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 4.97011 nnz ave. (N=321201)<br class="">[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square<br class="">[0] PCGAMGProlongator_AGG(): New grid 45754 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.972526e+00 min=3.461411e-03 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 1) N=45754, n data cols=1, nnz/row (ave)=10, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 10.7695 nnz ave. (N=45754)<br class="">[0] PCGAMGProlongator_AGG(): New grid 7893 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.686837e+00 min=5.062501e-01 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 2) N=7893, n data cols=1, nnz/row (ave)=23, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 23.2179 nnz ave. (N=7893)<br class="">[0] PCGAMGProlongator_AGG(): New grid 752 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.504451e+01 min=2.124898e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 3) N=752, n data cols=1, nnz/row (ave)=30, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 30.7367 nnz ave. (N=752)<br class="">[0] PCGAMGProlongator_AGG(): New grid 56 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=7.781296e+00 min=2.212257e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 4) N=56, n data cols=1, nnz/row (ave)=22, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 22.9643 nnz ave. (N=56)<br class="">[0] PCGAMGProlongator_AGG(): New grid 6 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.525086e+00 min=1.375043e-01 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 5) N=6, n data cols=1, nnz/row (ave)=6, 1 active pes<br class="">[0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.43876<br class="">[0] PCSetUp_GAMG(): level 0) N=321201, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 4.97011 nnz ave. (N=321201)<br class="">[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square<br class="">[0] PCGAMGProlongator_AGG(): New grid 45754 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.972526e+00 min=3.461411e-03 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 1) N=45754, n data cols=1, nnz/row (ave)=10, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 10.7695 nnz ave. (N=45754)<br class="">[0] PCGAMGProlongator_AGG(): New grid 7893 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.686837e+00 min=5.062501e-01 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 2) N=7893, n data cols=1, nnz/row (ave)=23, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 23.2179 nnz ave. (N=7893)<br class="">[0] PCGAMGProlongator_AGG(): New grid 752 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.504451e+01 min=2.124898e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 3) N=752, n data cols=1, nnz/row (ave)=30, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 30.7367 nnz ave. (N=752)<br class="">[0] PCGAMGProlongator_AGG(): New grid 56 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=7.781296e+00 min=2.212257e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 4) N=56, n data cols=1, nnz/row (ave)=22, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 22.9643 nnz ave. (N=56)<br class="">[0] PCGAMGProlongator_AGG(): New grid 6 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.525086e+00 min=1.375043e-01 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 5) N=6, n data cols=1, nnz/row (ave)=6, 1 active pes<br class="">[0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.43876<br class="">[0] PCSetUp_GAMG(): level 0) N=271201, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 4.97455 nnz ave. (N=271201)<br class="">[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square<br class="">[0] PCGAMGProlongator_AGG(): New grid 38501 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.933798e+00 min=4.684075e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 1) N=38501, n data cols=1, nnz/row (ave)=10, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 10.7732 nnz ave. (N=38501)<br class="">[0] PCGAMGProlongator_AGG(): New grid 6664 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.623029e+00 min=1.250957e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 2) N=6664, n data cols=1, nnz/row (ave)=23, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 23.2098 nnz ave. (N=6664)<br class="">[0] PCGAMGProlongator_AGG(): New grid 620 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.763329e+00 min=1.611776e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 3) N=620, n data cols=1, nnz/row (ave)=29, 1 active pes<br class="">[0] PCGAMGFilterGraph(): <span class="m_3908295385861418691Apple-tab-span" style="white-space: pre-wrap;"> </span> 100.% nnz after filtering, with threshold 0., 29.6129 nnz ave. (N=620)<br class="">[0] PCGAMGProlongator_AGG(): New grid 46 nodes<br class="">[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.497611e+00 min=2.630403e-02 PC=jacobi<br class="">[0] PCSetUp_GAMG(): 4) N=46, n data cols=1, nnz/row (ave)=20, 1 active pes<br class="">[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.43639</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">Il giorno 02 ago 2018, alle ore 17:39, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> ha scritto:</div><br class="m_3908295385861418691Apple-interchange-newline"><div class=""><div dir="ltr" class="">It looks like ML and hypre are working well now. If you want to debug GAMG you can run with -info, which is very noisy, and grep on GAMG or send the whole output.<div class=""><br class=""></div><div class="">BTW, what equations are you solving?</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Thu, Aug 2, 2018 at 5:12 AM Pierpaolo Minelli <<a href="mailto:pierpaolo.minelli@cnr.it" target="_blank" class="">pierpaolo.minelli@cnr.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class="">Thank you very much for the correction.</div><div class="">By rebalancing the matrix coefficients, making them dimensionless, in my complex problem i managed to obtain a better result with both ML and HYPRE. GAMG instead seems to be unable to converge and in fact I get unexpected results. I report the outputs of the three simulations again (i tried to use also -pc_gamg_square_graph 20 without any improvement).</div><div class=""><br class=""></div><div class="">Pierpaolo</div><div class=""><br class=""></div><div class=""><b class="">-pc_type hypre -ksp_rtol 1.e-7 -ksp_monitor_true_residual</b></div><div class=""><b class=""><br class=""></b></div><div class=""> <span class="Apple-converted-space"> </span>0 KSP preconditioned resid norm 1.984853668904e-02 true resid norm 2.979865703850e-03 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP preconditioned resid norm 1.924446712661e-04 true resid norm 1.204260204811e-04 ||r(i)||/||b|| 4.041323752460e-02<br class=""> 2 KSP preconditioned resid norm 5.161509100765e-06 true resid norm 2.810809726926e-06 ||r(i)||/||b|| 9.432672496933e-04<br class=""> 3 KSP preconditioned resid norm 9.297326931238e-08 true resid norm 4.474617977876e-08 ||r(i)||/||b|| 1.501617328625e-05<br class=""> 4 KSP preconditioned resid norm 1.910271882670e-09 true resid norm 9.637470658283e-10 ||r(i)||/||b|| 3.234196308186e-07<br class=""> 0 KSP preconditioned resid norm 2.157687745805e+04 true resid norm 3.182001523188e+03 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP preconditioned resid norm 1.949268476386e+02 true resid norm 1.243419788627e+02 ||r(i)||/||b|| 3.907665598415e-02<br class=""> 2 KSP preconditioned resid norm 5.078054475792e+00 true resid norm 2.745355604400e+00 ||r(i)||/||b|| 8.627763325675e-04<br class=""> 3 KSP preconditioned resid norm 8.663802743529e-02 true resid norm 4.254290979292e-02 ||r(i)||/||b|| 1.336985839979e-05<br class=""> 4 KSP preconditioned resid norm 1.795605563039e-03 true resid norm 9.040507428245e-04 ||r(i)||/||b|| 2.841138623714e-07<br class=""> 0 KSP preconditioned resid norm 6.728304961395e+02 true resid norm 1.879478105170e+02 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP preconditioned resid norm 2.190497539532e+01 true resid norm 4.630095820203e+02 ||r(i)||/||b|| 2.463500802413e+00<br class=""> 2 KSP preconditioned resid norm 8.425561564252e-01 true resid norm 7.012565302251e+01 ||r(i)||/||b|| 3.731123700223e-01<br class=""> 3 KSP preconditioned resid norm 3.029848345705e-02 true resid norm 4.379018464663e+00 ||r(i)||/||b|| 2.329911932795e-02<br class=""> 4 KSP preconditioned resid norm 7.374025528575e-04 true resid norm 1.337183702137e-01 ||r(i)||/||b|| 7.114654320570e-04<br class=""> 5 KSP preconditioned resid norm 3.009400175162e-05 true resid norm 7.731135032616e-03 ||r(i)||/||b|| 4.113447776459e-05</div><div class=""><br class=""></div><div class=""><b class="">-pc_type ml -ksp_rtol 1.e-7 -ksp_monitor_true_residual</b></div><div class=""><b class=""><br class=""></b></div><div class=""> <span class="Apple-converted-space"> </span>0 KSP preconditioned resid norm 1.825767020538e-02 true resid norm 2.979865703850e-03 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP preconditioned resid norm 6.495628259383e-04 true resid norm 3.739440526742e-04 ||r(i)||/||b|| 1.254902367550e-01<br class=""> 2 KSP preconditioned resid norm 4.971875712015e-05 true resid norm 2.118856024328e-05 ||r(i)||/||b|| 7.110575559127e-03<br class=""> 3 KSP preconditioned resid norm 3.726806462912e-06 true resid norm 1.370355844514e-06 ||r(i)||/||b|| 4.598716790303e-04<br class=""> 4 KSP preconditioned resid norm 2.496898447120e-07 true resid norm 9.494701893753</div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div><span id="cid:164ffe4b08a8a3bbf411"><clip_image002.png></span><span id="cid:%3C%3E"><clip_image002.png></span></div></blockquote></div><br class=""></div></div></body></html>