<div dir="ltr"><div dir="ltr">On Thu, May 12, 2022 at 9:09 AM Karabelas, Elias (<a href="mailto:elias.karabelas@uni-graz.at">elias.karabelas@uni-graz.at</a>) <<a href="mailto:elias.karabelas@uni-graz.at">elias.karabelas@uni-graz.at</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><font size="4"><font face="monospace">Dear Team,</font></font></p>
<p><font size="4"><font face="monospace">I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on the ARCHER2 cluster.<br>
</font></font></p>
<p><font size="4"><font face="monospace">These are my options for solving a Navier-Stokes-like system and it ran fine on other clusters.<br>
</font></font></p>
<p><font size="4"><font face="monospace">-ksp_type fgmres<br>
-ksp_pc_side right<br>
-ksp_initial_guess_nonzero 0<br>
-ksp_lag_norm<br>
-ksp_gmres_restart 100<br>
-ksp_monitor :block_system_ksp_its.log<br>
-pc_type fieldsplit<br>
-pc_fieldsplit_type schur<br>
-pc_fieldsplit_schur_fact_type full<br>
-pc_fieldsplit_schur_precondition selfp<br>
-fieldsplit_0_ksp_type preonly<br>
-fieldsplit_0_pc_type hypre<br>
-fieldsplit_0_pc_hypre_type boomeramg<br>
-fieldsplit_0_pc_hypre_boomeramg_max_iter 1<br>
-fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS<br>
-fieldsplit_0_pc_hypre_boomeramg_tol 0.0<br>
-fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i<br>
-fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7<br>
-fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel<br>
-fieldsplit_1_mat_schur_complement_ainv_type blockdiag<br>
-fieldsplit_1_ksp_type gmres<br>
-fieldsplit_1_pc_type sor<br>
-fieldsplit_1_ksp_max_it 30<br>
-fieldsplit_1_ksp_rtol 1e-1<br>
-fieldsplit_1_ksp_atol 0.0<br>
-fieldsplit_1_ksp_pc_side right</font></font></p>
<p><font size="4"><font face="monospace"><br>
</font></font></p>
<p><font size="4"><font face="monospace">However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit my jobs. I did some digging with valgrind I found out that in PCApply when BoomerAMG is called and HYPRE is compiled with the option "--enable-persistent
1" there are a lot of calloc's on HYPRE's side that are not free'd even after the PC gets destroyed from Petsc side. This seems to be the case on the vendor-shipped Petsc on ARCHER2. Building my own Petsc and Hypre on the cluster resolves this issue with the
downside of loosing around 15% of speed.</font></font></p></div></blockquote><div>Hi Elias,</div><div><br></div><div>Is your self-compiled PETSc a newer version? If so, it sounds like this can be resolved by getting the admins on ARCHER to update the PETSc. They have been pretty responsive for the Firedrake people, so I don't think this will be hard.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p><span style="font-family:monospace;font-size:large">I thought the developers might be interested in this bug, but I do not know who is to be adressed here, the hypre developers, the petsc developers, or the ARCHER2 admins.</span></p>
<p><font size="4"><font face="monospace">Best regards</font></font></p>
<p><font size="4"><font face="monospace">Elias<br>
</font></font></p>
<pre cols="72">--
Dr. Elias Karabelas
Research Associate
University of Graz
Institute of Mathematics and Scientific Computing
Heinrichstraße 36
A-8010 Graz
Austria
Phone: +43 316 380 8546
Email: <a href="mailto:elias.karabelas@uni-graz.at" target="_blank">elias.karabelas@uni-graz.at</a>
Web: <a href="https://ccl.medunigraz.at/" target="_blank">https://ccl.medunigraz.at/</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>