<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 2 Sep 2021, at 2:07 PM, Viktor Nazdrachev <<a href="mailto:numbersixvs@gmail.com" class="">numbersixvs@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family: "Courier New"; font-size: 10pt;" class="">Hello, Pierre!</span><br class=""></p><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">
Thank you for your response!</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">I attached log files (txt files with convergence
behavior and RAM usage log in separate txt files) and resulting table with
convergence investigation data(xls). Data for main non-regular grid with 500K cells
and heterogeneous properties are in 500K folder, whereas data for simple
uniform 125K cells grid with constant properties are in 125K folder. </span></div><div style="text-align: justify; margin: 0cm 0cm 8pt; line-height: 107%; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 14pt; line-height: 107%;" class=""> </span><br class="webkit-block-placeholder"></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">>Dear Viktor,</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">> </span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">>><i class=""> On 1 Sep 2021, at 10:42 AM, </i></span><i class=""><span style="font-size: 10pt; font-family: "Courier New";" class="">Наздрачёв</span></i><i class=""><span style="font-size: 10pt; font-family: "Courier New";" class=""> </span></i><i class=""><span style="font-size: 10pt; font-family: "Courier New";" class="">Виктор</span></i><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> <</span></i><i class=""><span style="font-size: 10pt; font-family: "Courier New";" class=""><a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" class=""><span lang="EN-US" style="" class="">numbersixvs at
gmail.com</span></a></span></i><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">> <a name="_Hlk81487196" class="">></a>wrote:</span></i></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class="">
Dear all,</i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> I
have a 3D elasticity problem with heterogeneous properties. There is
unstructured grid with aspect ratio varied from 4 to 25. Zero Dirichlet
BCs are imposed on bottom face of mesh.
Also, Neumann (traction) BCs are imposed on side faces. Gravity load is also
accounted for. The grid I use consists of 500k cells (which is approximately
1.6M of DOFs).</i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class="">
The best performance and memory usage for single MPI process was obtained with
HPDDM(BFBCG) solver</i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Block
Krylov solvers are (most often) only useful if you have multiple right-hand
sides, e.g., in the context of elasticity, multiple loadings.</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Is that really the case? If not, you may as well stick
to “standard” CG instead of the breakdown-free block (BFB) variant.</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">> </span></i></div></blockquote><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><div style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">In that case only single right-hand side is utilized,
so I switched to “standard” cg solver (-ksp_hpddm_type cg), but I noticed the interesting
convergence behavior. For non-regular grid with 500K cells and heterogeneous
properties CG solver converged with 1
iteration (log_hpddm(cg)_gamg_nearnullspace_1_mpi.txt), but for more simple
uniform grid with 125K cells and homogeneous properties CG solves linear system
successfully(log_hpddm(cg)_gamg_nearnullspace_1_mpi.txt).</span></div><div style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">BFBCG solver works properly for both grids.</span></div></div></div></blockquote><div><br class=""></div><div>Just stick to -ksp_type cg or maybe -ksp_type gmres -ksp_gmres_modifiedgramschmidt (even if the problem is SPD).</div><div>Sorry if I repeat myself, but KSPHPDDM methods are mostly useful for either blocking or recycling.</div><div>If you use something as simple as CG, you’ll get better diagnostics and error handling if you use the native PETSc implementation (KSPCG) instead of the external implementation (-ksp_hpddm_type cg).</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: "Courier New"; font-size: 10pt;" class=""> </span><span style="font-family: "Courier New"; font-size: 10pt;" class=""> </span></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class="">
and bjacobian + ICC (1) in subdomains as preconditioner, it took 1 m 45 s and
RAM 5.0 GB. Parallel computation with 4 MPI processes took 2 m 46 s when using
5.6 GB of RAM. This because of number of iterations required to achieve the
same tolerance is significantly increased.</i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class=""> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class="">
I`ve also tried PCGAMG (agg) preconditioner with IC</i></span><i class=""><span style="font-size: 10pt; font-family: "Courier New";" class="">С</span></i><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> (1) sub-precondtioner. For single MPI process, the
calculation took 10 min and 3.4 GB of RAM. To improve the convergence rate, the
nullspace was attached using MatNullSpaceCreateRigidBody and
MatSetNearNullSpace subroutines. This
has reduced calculation time to 3 m 58 s when using 4.3 GB of RAM. Also, there
is peak memory usage with 14.1 GB, which appears just before the start of the
iterations. Parallel computation with 4 MPI processes took 2 m 53 s when using
8.4 GB of RAM. In that case the peak memory usage is about 22 GB.</span></i></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">><i class="">> </i></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">I’m
surprised that GAMG is converging so slowly. What do you mean by "ICC(1)
sub-preconditioner"? Do you use that as a smoother or as a coarse level
solver?</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></div></blockquote><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""><br class=""></span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Sorry for misleading, ICC is used only for BJACOBI preconditioner,
no ICC for GAMG.</span></div><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">How many
iterations are required to reach convergence?</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Could you
please maybe run the solver with -ksp_view -log_view and send us the output?</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></div></blockquote><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><div style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">For case with 4 MPI processes and attached nullspace
it is required 177 iterations to reach convergence (you may see detailed log in
log_hpddm(bfbcg)_gamg_nearnullspace_4_mpi.txt and memory usage log in RAM_log_hpddm(bfbcg)_gamg_nearnullspace_4_mpi.txt).
For comparison, 90 iterations are required for sequential run(log_hpddm(bfbcg)_gamg_nearnullspace_1_mpi.txt).
</span></div><div style="margin: 0cm; text-align: justify; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""><br class="">
<br class="">
</span></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Most of
the default parameters of GAMG should be good enough for 3D elasticity,
provided that your MatNullSpace is correct.</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></div></blockquote><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">How can I be sure that nullspace is attached
correctly? Is there any way for self-checking (Well perhaps calculate some
parameters using matrix and solution vector)? </span></div><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">></span></i><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">One
parameter that may need some adjustments though is the aggregation threshold
-pc_gamg_threshold (you could try values in the [0.01; 0.1] range, that’s what
I always use for elasticity problems).</span></div><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">> </span></i></div></blockquote><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; line-height: 107%; font-family: "Courier New";" class="">Tried to
find optimal value of this option, set -pc_gamg_threshold 0.01 and -</span><span lang="EN-US" style="font-size: 10.5pt; line-height: 107%; font-family: "Courier New";" class="">pc_gamg_threshold_scale 2, but
I didn't notice any significant changes (Need more time for experiments ) </span></p></div></blockquote><div>I don’t see anything too crazy in your logs at first sight. In addition to maybe trying GMRES with a more robust orthogonalization scheme, here is what I would do:</div><div>1) MatSetBlockSize(Pmat, 6), it seems to be missing right now, cf.</div><div><div> linear system matrix = precond matrix:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=1600200, cols=1600200</div><div> total: nonzeros=124439742, allocated nonzeros=259232400</div><div> total number of mallocs used during MatSetValues calls=0</div><div> has attached near null space</div></div>2) -mg_coarse_pc_type redundant -mg_coarse_redundant_pc_type lu</div><div>3) more playing around with the threshold, this can be critical for hard problems</div><div>If you can share your matrix/nullspace/RHS, we could have a crack at it as well.</div><div><br class=""></div><div>Thanks,</div><div>Pierre </div><div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Kind regards,</span></div><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">Viktor Nazdrachev</span></div><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><div style="margin: 0cm; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class="">R&D senior researcher</span></div><p class="MsoNormal" style="margin:0cm;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10pt; font-family: "Courier New";" class=""> </span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size: 10.5pt; line-height: 107%; font-family: "Courier New";" class=""><span style="font-size:10pt" class="">Geosteering Technologies LLC</span> </span></p></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">ср, 1 сент. 2021 г. в 12:01, Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>>:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Dear Viktor,<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Sep 2021, at 10:42 AM, Наздрачёв Виктор <<a href="mailto:numbersixvs@gmail.com" target="_blank" class="">numbersixvs@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Dear all,</span></p><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">I have a 3D
elasticity problem with heterogeneous properties. There is unstructured grid
with aspect ratio varied from 4 to 25. Zero Dirichlet BCs are imposed on bottom face of mesh. Also,
Neumann (traction) BCs are imposed on side faces. Gravity load is also
accounted for. The grid I use consists of 500k cells (which is approximately
1.6M of DOFs).</span></p><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">The best
performance and memory usage for single MPI process was obtained with
HPDDM(BFBCG) solver</span></p></div></div></blockquote><div class="">Block Krylov solvers are (most often) only useful if you have multiple right-hand sides, e.g., in the context of elasticity, multiple loadings.</div><div class="">Is that really the case? If not, you may as well stick to “standard” CG instead of the breakdown-free block (BFB) variant.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class=""> and bjacobian + ICC (1) in subdomains as preconditioner, it
took 1 m 45 s and RAM 5.0 GB. Parallel computation with 4 MPI processes took 2
m 46 s when using 5.6 GB of RAM. This because of number of iterations required
to achieve the same tolerance is significantly increased.</span></p><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">I`ve also
tried PCGAMG (agg) preconditioner with IC</span><span style="font-size:14pt;line-height:107%" class="">С</span><span lang="EN-US" style="font-size:14pt;line-height:107%" class=""> (1) sub-precondtioner. For single MPI process, the
calculation took 10 min and 3.4 GB of RAM. To improve the convergence rate, the
nullspace was attached using MatNullSpaceCreateRigidBody and MatSetNearNullSpace
subroutines. This has reduced
calculation time to 3 m 58 s when using 4.3 GB of RAM. Also, there is peak
memory usage with 14.1 GB, which appears just before the start of the
iterations. Parallel computation with 4 MPI processes took 2 m 53 s when using
8.4 GB of RAM. In that case the peak memory usage is about 22 GB.</span></p></div></div></blockquote><div class="">I’m surprised that GAMG is converging so slowly. What do you mean by "ICC(1) sub-preconditioner"? Do you use that as a smoother or as a coarse level solver?</div><div class="">How many iterations are required to reach convergence?</div><div class="">Could you please maybe run the solver with -ksp_view -log_view and send us the output?</div><div class="">Most of the default parameters of GAMG should be good enough for 3D elasticity, provided that your MatNullSpace is correct.</div><div class="">One parameter that may need some adjustments though is the aggregation threshold -pc_gamg_threshold (you could try values in the [0.01; 0.1] range, that’s what I always use for elasticity problems).</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Are there
ways to avoid decreasing of the convergence rate for bjacobi precondtioner in
parallel mode? Does it make sense to use hierarchical or nested krylov methods
with a local gmres solver (sub_pc_type gmres) and some sub-precondtioner (for
example, sub_pc_type bjacobi)?</span></p><div style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif" class=""><span lang="EN-US" style="font-size:14pt;line-height:107%" class=""> </span><br class=""></div><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Is this peak
memory usage expected for gamg preconditioner? is there any way to reduce it?</span></p><div style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif" class=""><span lang="EN-US" style="font-size:14pt;line-height:107%" class=""> </span><br class=""></div><p class="MsoNormal" style="text-align:justify;margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">What advice
would you give to improve the convergence rate with multiple MPI processes, but
keep memory consumption reasonable?</span></p><div style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif" class=""><span lang="EN-US" style="font-size:14pt;line-height:107%" class=""> </span><br class=""></div><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Kind regards,</span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Viktor Nazdrachev</span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">R&D senior researcher</span></p><p class="MsoNormal" style="margin:0cm 0cm 8pt;line-height:107%;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:14pt;line-height:107%" class="">Geosteering Technologies LLC</span></p></div>
</div></blockquote></div><br class=""></div></blockquote></div>
<span id="cid:f_kt2vv2p60"><logs.rar></span></blockquote></div><br class=""></body></html>