<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 25, 2021, at 4:14 PM, Lionel CHENG <<a href="mailto:cheng@cerfacs.fr" class="">cheng@cerfacs.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt;" class=""><div class="">The number of iterations at initialization (with rtol=1e-10) for both cg is 11 for gmres and 12 for cg so roughly the same. Switching to bcgs the number of iterations goes down to 6. So CG does not seem to fake it. </div></div></div></div></blockquote><div><br class=""></div> Yes, this sounds reasonable enough, maybe a touch of loss of orthogonality with the CG.<br class=""><blockquote type="cite" class=""><div class=""><div class=""><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt;" class=""><div class=""><br data-mce-bogus="1" class=""></div><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none; float: none; display: inline !important;" data-mce-style="caret-color: #000000; color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration: none; display: inline !important; float: none;" class="">Going back to the number of multi grid levels: how should I choose the number of multi grid level depending on the problem at hand? How does GAMG evaluate the number of grid levels required?</span></div></div></div></div></blockquote><div><br class=""></div> Usually we just stick with the heuristic that GAMG comes up with. Maybe Mark has some better advice.</div><div><br class=""></div><div> Barry</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt;" class=""><div class=""><br data-mce-bogus="1" class=""></div><div class="">Lionel</div><div class=""><br class=""></div><hr id="zwchr" data-marker="__DIVIDER__" class=""><div data-marker="__HEADERS__" class=""><b class="">De: </b>"Barry Smith" <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class=""><b class="">À: </b>"cheng" <<a href="mailto:cheng@cerfacs.fr" class="">cheng@cerfacs.fr</a>><br class=""><b class="">Cc: </b>"petsc-users" <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Envoyé: </b>Lundi 25 Octobre 2021 19:52:26<br class=""><b class="">Objet: </b>Re: [petsc-users] Convergence on Axisymmetric Poisson matrix<br class=""></div><div class=""><br class=""></div><div data-marker="__QUOTED_TEXT__" class=""><br class=""><div class=""><br class=""><blockquote class=""><div class="">On Oct 25, 2021, at 1:01 PM, Lionel CHENG <<a href="mailto:cheng@cerfacs.fr" target="_blank" rel="nofollow noopener noreferrer" class="">cheng@cerfacs.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt" class=""><div class="">We are running with the <span style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb( 255 , 255 , 255 );text-decoration:none;float:none;display:inline !important" class=""><span style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb( 255 , 255 , 255 );text-decoration:none;float:none;display:inline !important" class="">-ksp_norm_type unpreconditioned so the convergence is done with the true residual for all the previous tests. I have a case with 800 000 nodes that I have run for 200 iterations on 36 CPU cor</span></span>es (Intel Xeon Gold 6140 - Skylake) and the Poisson solver gives me<div class=""><br class=""></div></div><div class=""><div class="">| Krylov Solver | Poisson running time [s] |</div><div class="">| `cg` | 3.9150E+00 |</div><div class="">| `gmres` | 4.6527E+00 |</div><div class="">| `bcgs` | 5.4416E+00 |</div></div><div class=""><br class=""></div><div class="">Only the ksp_type has been changed in the following line:</div><div class=""><div class="">mpirun -np $nb_cpu $exec -ksp_initial_guess_nonzero true \</div><div class=""> -ksp_type bcgs -ksp_norm_type unpreconditioned \</div><div class=""> -ksp_rtol 1e-10 \</div><div class=""> -pc_type gamg -mg_levels_pc_type sor -mg_levels_ksp_type richardson \</div><div class=""><br class=""></div><div class="">So CG is better than gmres (I have included the BiCGStab runs as well as I have talked about them earlier). </div></div></div></div></div></blockquote><div class=""><br class=""></div><div class=""> I was not interested in the runtime, I was interested in the convergence behavior of CG vs GMRES for this problem. If CG is "faking it" then one would see the GMRES converging faster (its residual would get smaller with fewer iterations).</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><br class=""><blockquote class=""><div class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt" class=""><div class=""><div class="">I find it really weird that it behaves well with the preconditioner gamg I can't really find an explanation why, it is really against my intuition. </div><div class=""><br class=""></div><div class="">Apart from that I have also played around with the number of multi-grid levels (-pc_mg_levels):</div><div class=""><br class=""></div><div class=""><div class=""><div class="">| Number of MG levels | Poisson running time [s] |</div><div class="">| ------------------------------- | ------------------------ |</div><div class="">| 2 | 1.0385E+01 |</div><div class="">| 3 | 5.0015E+00 |</div><div class="">| 4 | 3.9150E+00 |</div><div class="">| 5 | 4.5015E+00 |</div><div class="">| 6 (default petsc for this case) | 4.5510E+00 |</div><div class=""><br class=""></div><div class="">So that I find an optimum for 4 and not 6 as in the default PETSc configuration and not specifying anything. How should I choose the number of multi grid level depending on my problem? How does GAMG evaluate the number of grid levels required?</div><div class=""><br class=""></div><div class="">Lionel</div></div></div></div><div class=""><br class=""></div><hr id="zwchr" class=""><div class=""><b class="">De: </b>"Barry Smith" <<a href="mailto:bsmith@petsc.dev" target="_blank" rel="nofollow noopener noreferrer" class="">bsmith@petsc.dev</a>><br class=""><b class="">À: </b>"cheng" <<a href="mailto:cheng@cerfacs.fr" target="_blank" rel="nofollow noopener noreferrer" class="">cheng@cerfacs.fr</a>><br class=""><b class="">Cc: </b>"petsc-users" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="nofollow noopener noreferrer" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Envoyé: </b>Lundi 25 Octobre 2021 15:33:50<br class=""><b class="">Objet: </b>Re: [petsc-users] Convergence on Axisymmetric Poisson matrix<br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div> Are you running with -ksp_monitor_true_residual to track the b - A*x residual instead of just the preconditioned residual? <div class=""><br class=""></div><div class=""> GAMG definitely does not symmetrize the system but it is possible the preconditioner results in the solve "not seeing" the unsymmetry during the solution process and hence CG still converging; it would be dangerous to rely on this in general I think. You could also run this case with GMRES to see if that is better than the CG iterations.<br class=""><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><blockquote class=""><div class="">On Oct 24, 2021, at 7:00 PM, Lionel CHENG <<a href="mailto:cheng@cerfacs.fr" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank" class="">cheng@cerfacs.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt" class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Hello everyone,</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">I have some questions regarding a linear system that I am solving in my plasma simulations. We have in this case a strongly non-symmetric matrix due to the cylindrical coordinates for which the Laplacian cell is given by Fig. 2 for two kinds of triangles. The different unstructured grids have from 300 000 nodes to 7 000 000 nodes.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">To my understanding, CG should not work properly on this matrix but BiCGStab(1) should. When using SOR preconditioner it is indeed the case: -ksp_type cg -pc_type sor yields solutions in 10 to 20 times more iterations than -ksp_type bcgs -pc_type sor.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">However, when switching to -ksp_type cg -pc_type gamg the convergence is great and even slightly better than -ksp_type bcgs. I do not understand how CG is able to make the system converge when using GAMG although the matrix is non-symmetric ? Is GAMG able to somehow symmetrize the system? I have the impression that when using -pc_type gamg the Krylov solver is actually the Pre-relaxation and post-relaxation of the initial grid, is that right?</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">For GAMG since the matrix is non-symmetric -mg_levels_pc_type sor for and -mg_levels_ksp_type richardson have been used and yields better results than the original chebychev solver.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Sincerely yours,</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Lionel Cheng</div></div></div></div><span id="cid:419DA849-7F57-4955-A86C-C0DB83BBBC79" class=""><main.pdf></span></div></blockquote></div><br class=""></div></div><br class=""></div></div></div></div></blockquote></div><br class=""><br class=""></div></div></div></div></blockquote></div><br class=""></body></html>