<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>We are running with the <span style="caret-color: #000000; color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration: none; display: inline !important; float: none;" data-mce-style="caret-color: #000000; color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration: none; display: inline !important; float: none;"><span style="caret-color: #000000; color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration: none; display: inline !important; float: none;" data-mce-style="caret-color: #000000; color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration: none; display: inline !important; float: none;">-ksp_norm_type unpreconditioned so the convergence is done with the true residual for all the previous tests. I have a case with 800 000 nodes that I have run for 200 iterations on 36 CPU cor</span></span>es (Intel Xeon Gold 6140 - Skylake) and the Poisson solver gives me<div style="clear: both;" data-mce-style="clear: both;"><br></div></div><div><div>| Krylov Solver | Poisson running time [s] |</div><div>| `cg` | 3.9150E+00 |</div><div>| `gmres` | 4.6527E+00 |</div><div>| `bcgs` | 5.4416E+00 |</div></div><div><br data-mce-bogus="1"></div><div>Only the ksp_type has been changed in the following line:</div><div><div>mpirun -np $nb_cpu $exec -ksp_initial_guess_nonzero true \</div><div> -ksp_type bcgs -ksp_norm_type unpreconditioned \</div><div> -ksp_rtol 1e-10 \</div><div> -pc_type gamg -mg_levels_pc_type sor -mg_levels_ksp_type richardson \</div><div><br data-mce-bogus="1"></div><div>So CG is better than gmres (I have included the BiCGStab runs as well as I have talked about them earlier). I find it really weird that it behaves well with the preconditioner gamg I can't really find an explanation why, it is really against my intuition. </div><div><br data-mce-bogus="1"></div><div>Apart from that I have also played around with the number of multi-grid levels (-pc_mg_levels):</div><div><br data-mce-bogus="1"></div><div><div><div>| Number of MG levels | Poisson running time [s] |</div><div>| ------------------------------- | ------------------------ |</div><div>| 2 | 1.0385E+01 |</div><div>| 3 | 5.0015E+00 |</div><div>| 4 | 3.9150E+00 |</div><div>| 5 | 4.5015E+00 |</div><div>| 6 (default petsc for this case) | 4.5510E+00 |</div><div><br data-mce-bogus="1"></div><div>So that I find an optimum for 4 and not 6 as in the default PETSc configuration and not specifying anything. How should I choose the number of multi grid level depending on my problem? How does GAMG evaluate the number of grid levels required?</div><div><br data-mce-bogus="1"></div><div>Lionel</div></div></div></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>De: </b>"Barry Smith" <bsmith@petsc.dev><br><b>À: </b>"cheng" <cheng@cerfacs.fr><br><b>Cc: </b>"petsc-users" <petsc-users@mcs.anl.gov><br><b>Envoyé: </b>Lundi 25 Octobre 2021 15:33:50<br><b>Objet: </b>Re: [petsc-users] Convergence on Axisymmetric Poisson matrix<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div class=""><br class=""></div> Are you running with -ksp_monitor_true_residual to track the b - A*x residual instead of just the preconditioned residual? <div class=""><br class=""></div><div class=""> GAMG definitely does not symmetrize the system but it is possible the preconditioner results in the solve "not seeing" the unsymmetry during the solution process and hence CG still converging; it would be dangerous to rely on this in general I think. You could also run this case with GMRES to see if that is better than the CG iterations.<br class=""><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><blockquote class=""><div class="">On Oct 24, 2021, at 7:00 PM, Lionel CHENG <<a href="mailto:cheng@cerfacs.fr" target="_blank" rel="nofollow noopener noreferrer">cheng@cerfacs.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt" class=""><div class=""><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Hello everyone,</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">I have some questions regarding a linear system that I am solving in my plasma simulations. We have in this case a strongly non-symmetric matrix due to the cylindrical coordinates for which the Laplacian cell is given by Fig. 2 for two kinds of triangles. The different unstructured grids have from 300 000 nodes to 7 000 000 nodes.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">To my understanding, CG should not work properly on this matrix but BiCGStab(1) should. When using SOR preconditioner it is indeed the case: -ksp_type cg -pc_type sor yields solutions in 10 to 20 times more iterations than -ksp_type bcgs -pc_type sor.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">However, when switching to -ksp_type cg -pc_type gamg the convergence is great and even slightly better than -ksp_type bcgs. I do not understand how CG is able to make the system converge when using GAMG although the matrix is non-symmetric ? Is GAMG able to somehow symmetrize the system? I have the impression that when using -pc_type gamg the Krylov solver is actually the Pre-relaxation and post-relaxation of the initial grid, is that right?</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">For GAMG since the matrix is non-symmetric -mg_levels_pc_type sor for and -mg_levels_ksp_type richardson have been used and yields better results than the original chebychev solver.</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Sincerely yours,</div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:16px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">Lionel Cheng</div></div></div></div><span id="cid:419DA849-7F57-4955-A86C-C0DB83BBBC79"><main.pdf></span></div></blockquote></div><br class=""></div></div><br></div></div></body></html>