<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 2 Sep 2021, at 2:31 PM, Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="Apple-interchange-newline"><br class=""><blockquote type="cite" class=""><div class="">On 2 Sep 2021, at 2:07 PM, Viktor Nazdrachev <<a href="mailto:numbersixvs@gmail.com" class="">numbersixvs@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-family: "Courier New"; font-size: 10pt;">Hello, Pierre!</span><br class=""></p><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Thank you for your response!</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">I attached log files (txt files with convergence behavior and RAM usage log in separate txt files) and resulting table with convergence investigation data(xls). Data for main non-regular grid with 500K cells and heterogeneous properties are in 500K folder, whereas data for simple uniform 125K cells grid with constant properties are in 125K folder. <span class="Apple-converted-space"> </span></span></div><div class="" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;"> </span><br class="webkit-block-placeholder"></div><blockquote class="" style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">>Dear Viktor,</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">> </span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">>><i class=""><span class="Apple-converted-space"> </span>On 1 Sep 2021, at 10:42 AM,<span class="Apple-converted-space"> </span></i></span><i class=""><span class="" style="font-size: 10pt; font-family: "Courier New";">Наздрачёв</span></i><i class=""><span class="" style="font-size: 10pt; font-family: "Courier New";"><span class="Apple-converted-space"> </span></span></i><i class=""><span class="" style="font-size: 10pt; font-family: "Courier New";">Виктор</span></i><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"><span class="Apple-converted-space"> </span><</span></i><i class=""><span class="" style="font-size: 10pt; font-family: "Courier New";"><a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" class=""><span lang="EN-US" class="">numbersixvs at gmail.com</span></a></span></i><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><span class="Apple-converted-space"> </span><a name="_Hlk81487196" class="">></a>wrote:</span></i></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""><span class="Apple-converted-space"> </span>Dear all,</i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""><span class="Apple-converted-space"> </span>I have a 3D elasticity problem with heterogeneous properties. There is unstructured grid with aspect ratio varied from 4 to 25. Zero Dirichlet BCs are imposed on bottom face of mesh. Also, Neumann (traction) BCs are imposed on side faces. Gravity load is also accounted for. The grid I use consists of 500k cells (which is approximately 1.6M of DOFs).</i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""><span class="Apple-converted-space"> </span>The best performance and memory usage for single MPI process was obtained with HPDDM(BFBCG) solver</i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Block Krylov solvers are (most often) only useful if you have multiple right-hand sides, e.g., in the context of elasticity, multiple loadings.</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Is that really the case? If not, you may as well stick to “standard” CG instead of the breakdown-free block (BFB) variant.</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">> </span></i></div></blockquote><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><div class="" style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">In that case only single right-hand side is utilized, so I switched to “standard” cg solver (-ksp_hpddm_type cg), but I noticed the interesting convergence behavior. For non-regular grid with 500K cells and heterogeneous properties CG solver converged with 1 iteration (log_hpddm(cg)_gamg_nearnullspace_1_mpi.txt), but for more simple uniform grid with 125K cells and homogeneous properties CG solves linear system successfully(log_hpddm(cg)_gamg_nearnullspace_1_mpi.txt).</span></div><div class="" style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">BFBCG solver works properly for both grids.</span></div></div></div></blockquote><div class=""><br class=""></div><div class="">Just stick to -ksp_type cg or maybe -ksp_type gmres -ksp_gmres_modifiedgramschmidt (even if the problem is SPD).</div><div class="">Sorry if I repeat myself, but KSPHPDDM methods are mostly useful for either blocking or recycling.</div><div class="">If you use something as simple as CG, you’ll get better diagnostics and error handling if you use the native PETSc implementation (KSPCG) instead of the external implementation (-ksp_hpddm_type cg).</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="" style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-family: "Courier New"; font-size: 10pt;"> </span><span class="" style="font-family: "Courier New"; font-size: 10pt;"> </span></div><blockquote class="" style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""><span class="Apple-converted-space"> </span>and bjacobian + ICC (1) in subdomains as preconditioner, it took 1 m 45 s and RAM 5.0 GB. Parallel computation with 4 MPI processes took 2 m 46 s when using 5.6 GB of RAM. This because of number of iterations required to achieve the same tolerance is significantly increased.</i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class=""><span class="Apple-converted-space"> </span>I`ve also tried PCGAMG (agg) preconditioner with IC</i></span><i class=""><span class="" style="font-size: 10pt; font-family: "Courier New";">С</span></i><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"><span class="Apple-converted-space"> </span>(1) sub-precondtioner. For single MPI process, the calculation took 10 min and 3.4 GB of RAM. To improve the convergence rate, the nullspace was attached using MatNullSpaceCreateRigidBody and MatSetNearNullSpace subroutines. This has reduced calculation time to 3 m 58 s when using 4.3 GB of RAM. Also, there is peak memory usage with 14.1 GB, which appears just before the start of the iterations. Parallel computation with 4 MPI processes took 2 m 53 s when using 8.4 GB of RAM. In that case the peak memory usage is about 22 GB.</span></i></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">><i class="">><span class="Apple-converted-space"> </span></i></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">I’m surprised that GAMG is converging so slowly. What do you mean by "ICC(1) sub-preconditioner"? Do you use that as a smoother or as a coarse level solver?</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></div></blockquote><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"><br class=""></span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Sorry for misleading, ICC is used only for BJACOBI preconditioner, no ICC for GAMG.</span></div><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><blockquote class="" style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">How many iterations are required to reach convergence?</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Could you please maybe run the solver with -ksp_view -log_view and send us the output?</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></div></blockquote><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><div class="" style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">For case with 4 MPI processes and attached nullspace it is required 177 iterations to reach convergence (you may see detailed log in log_hpddm(bfbcg)_gamg_nearnullspace_4_mpi.txt and memory usage log in RAM_log_hpddm(bfbcg)_gamg_nearnullspace_4_mpi.txt). For comparison, 90 iterations are required for sequential run(log_hpddm(bfbcg)_gamg_nearnullspace_1_mpi.txt).</span></div><div class="" style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"><br class=""><br class=""></span></div><blockquote class="" style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Most of the default parameters of GAMG should be good enough for 3D elasticity, provided that your MatNullSpace is correct.</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></div></blockquote><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">How can I be sure that nullspace is attached correctly? Is there any way for self-checking (Well perhaps calculate some parameters using matrix and solution vector)?<span class="Apple-converted-space"> </span></span></div><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><blockquote class="" style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">></span></i><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">One parameter that may need some adjustments though is the aggregation threshold -pc_gamg_threshold (you could try values in the [0.01; 0.1] range, that’s what I always use for elasticity problems).</span></div><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><i class=""><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">> </span></i></div></blockquote><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; line-height: 14.266666412353516px; font-family: "Courier New";">Tried to find optimal value of this option, set -pc_gamg_threshold 0.01 and -</span><span lang="EN-US" class="" style="font-size: 10.5pt; line-height: 14.979999542236328px; font-family: "Courier New";">pc_gamg_threshold_scale 2, but I didn't notice any significant changes (Need more time for experiments ) </span></p></div></blockquote><div class="">I don’t see anything too crazy in your logs at first sight. In addition to maybe trying GMRES with a more robust orthogonalization scheme, here is what I would do:</div><div class="">1) MatSetBlockSize(Pmat, 6), it seems to be missing right now, cf.</div></div></div></blockquote><div><br class=""></div><div>Sorry for the noise, but this should read 3, not 6…</div><div><br class=""></div><div>Thanks,</div><div>Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""><div class=""> linear system matrix = precond matrix:</div><div class=""> Mat Object: 4 MPI processes</div><div class=""> type: mpiaij</div><div class=""> rows=1600200, cols=1600200</div><div class=""> total: nonzeros=124439742, allocated nonzeros=259232400</div><div class=""> total number of mallocs used during MatSetValues calls=0</div><div class=""> has attached near null space</div></div>2) -mg_coarse_pc_type redundant -mg_coarse_redundant_pc_type lu</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">3) more playing around with the threshold, this can be critical for hard problems</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">If you can share your matrix/nullspace/RHS, we could have a crack at it as well.</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Thanks,</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Pierre </div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Kind regards,</span></div><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">Viktor Nazdrachev</span></div><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><div class="" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";">R&D senior researcher</span></div><p class="MsoNormal" style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10pt; font-family: "Courier New";"> </span></p><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 10.5pt; line-height: 14.979999542236328px; font-family: "Courier New";"><span class="" style="font-size: 10pt;">Geosteering Technologies LLC</span> </span></p></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">ср, 1 сент. 2021 г. в 12:01, Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="overflow-wrap: break-word;">Dear Viktor,<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Sep 2021, at 10:42 AM, Наздрачёв Виктор <<a href="mailto:numbersixvs@gmail.com" target="_blank" class="">numbersixvs@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Dear all,</span></p><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">I have a 3D elasticity problem with heterogeneous properties. There is unstructured grid with aspect ratio varied from 4 to 25. Zero Dirichlet BCs are imposed on bottom face of mesh. Also, Neumann (traction) BCs are imposed on side faces. Gravity load is also accounted for. The grid I use consists of 500k cells (which is approximately 1.6M of DOFs).</span></p><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">The best performance and memory usage for single MPI process was obtained with HPDDM(BFBCG) solver</span></p></div></div></blockquote><div class="">Block Krylov solvers are (most often) only useful if you have multiple right-hand sides, e.g., in the context of elasticity, multiple loadings.</div><div class="">Is that really the case? If not, you may as well stick to “standard” CG instead of the breakdown-free block (BFB) variant.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">and bjacobian + ICC (1) in subdomains as preconditioner, it took 1 m 45 s and RAM 5.0 GB. Parallel computation with 4 MPI processes took 2 m 46 s when using 5.6 GB of RAM. This because of number of iterations required to achieve the same tolerance is significantly increased.</span></p><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">I`ve also tried PCGAMG (agg) preconditioner with IC</span><span class="" style="font-size: 14pt; line-height: 19.97333335876465px;">С</span><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;"><span class="Apple-converted-space"> </span>(1) sub-precondtioner. For single MPI process, the calculation took 10 min and 3.4 GB of RAM. To improve the convergence rate, the nullspace was attached using MatNullSpaceCreateRigidBody and MatSetNearNullSpace subroutines. This has reduced calculation time to 3 m 58 s when using 4.3 GB of RAM. Also, there is peak memory usage with 14.1 GB, which appears just before the start of the iterations. Parallel computation with 4 MPI processes took 2 m 53 s when using 8.4 GB of RAM. In that case the peak memory usage is about 22 GB.</span></p></div></div></blockquote><div class="">I’m surprised that GAMG is converging so slowly. What do you mean by "ICC(1) sub-preconditioner"? Do you use that as a smoother or as a coarse level solver?</div><div class="">How many iterations are required to reach convergence?</div><div class="">Could you please maybe run the solver with -ksp_view -log_view and send us the output?</div><div class="">Most of the default parameters of GAMG should be good enough for 3D elasticity, provided that your MatNullSpace is correct.</div><div class="">One parameter that may need some adjustments though is the aggregation threshold -pc_gamg_threshold (you could try values in the [0.01; 0.1] range, that’s what I always use for elasticity problems).</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Are there ways to avoid decreasing of the convergence rate for bjacobi precondtioner in parallel mode? Does it make sense to use hierarchical or nested krylov methods with a local gmres solver (sub_pc_type gmres) and some sub-precondtioner (for example, sub_pc_type bjacobi)?</span></p><div class="" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;"> </span><br class=""></div><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Is this peak memory usage expected for gamg preconditioner? is there any way to reduce it?</span></p><div class="" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;"> </span><br class=""></div><p class="MsoNormal" style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">What advice would you give to improve the convergence rate with multiple MPI processes, but keep memory consumption reasonable?</span></p><div class="" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;"> </span><br class=""></div><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Kind regards,</span></p><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Viktor Nazdrachev</span></p><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">R&D senior researcher</span></p><p class="MsoNormal" style="margin: 0cm 0cm 8pt; line-height: 15.693333625793457px; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US" class="" style="font-size: 14pt; line-height: 19.97333335876465px;">Geosteering Technologies LLC</span></p></div></div></blockquote></div><br class=""></div></blockquote></div><span id="cid:f_kt2vv2p60" class=""><logs.rar></span></blockquote></div></div></blockquote></div><br class=""></body></html>