<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p><font size="4"><font face="monospace">Dear Team,</font></font></p>
<p><font size="4"><font face="monospace">I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on the ARCHER2 cluster.<br>
</font></font></p>
<p><font size="4"><font face="monospace">These are my options for solving a Navier-Stokes-like system and it ran fine on other clusters.<br>
</font></font></p>
<p><font size="4"><font face="monospace">-ksp_type fgmres<br>
-ksp_pc_side right<br>
-ksp_initial_guess_nonzero 0<br>
-ksp_lag_norm<br>
-ksp_gmres_restart 100<br>
-ksp_monitor :block_system_ksp_its.log<br>
-pc_type fieldsplit<br>
-pc_fieldsplit_type schur<br>
-pc_fieldsplit_schur_fact_type full<br>
-pc_fieldsplit_schur_precondition selfp<br>
-fieldsplit_0_ksp_type preonly<br>
-fieldsplit_0_pc_type hypre<br>
-fieldsplit_0_pc_hypre_type boomeramg<br>
-fieldsplit_0_pc_hypre_boomeramg_max_iter 1<br>
-fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS<br>
-fieldsplit_0_pc_hypre_boomeramg_tol 0.0<br>
-fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i<br>
-fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7<br>
-fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel<br>
-fieldsplit_1_mat_schur_complement_ainv_type blockdiag<br>
-fieldsplit_1_ksp_type gmres<br>
-fieldsplit_1_pc_type sor<br>
-fieldsplit_1_ksp_max_it 30<br>
-fieldsplit_1_ksp_rtol 1e-1<br>
-fieldsplit_1_ksp_atol 0.0<br>
-fieldsplit_1_ksp_pc_side right</font></font></p>
<p><font size="4"><font face="monospace"><br>
</font></font></p>
<p><font size="4"><font face="monospace">However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit my jobs. I did some digging with valgrind I found out that in PCApply when BoomerAMG is called and HYPRE is compiled with the option "--enable-persistent
1" there are a lot of calloc's on HYPRE's side that are not free'd even after the PC gets destroyed from Petsc side. This seems to be the case on the vendor-shipped Petsc on ARCHER2. Building my own Petsc and Hypre on the cluster resolves this issue with the
downside of loosing around 15% of speed.</font></font></p>
<p><font size="4"><font face="monospace">I thought the developers might be interested in this bug, but I do not know who is to be adressed here, the hypre developers, the petsc developers, or the ARCHER2 admins.</font></font></p>
<p><font size="4"><font face="monospace">Best regards</font></font></p>
<p><font size="4"><font face="monospace">Elias<br>
</font></font></p>
<pre class="moz-signature" cols="72">--
Dr. Elias Karabelas
Research Associate
University of Graz
Institute of Mathematics and Scientific Computing
Heinrichstraße 36
A-8010 Graz
Austria
Phone: +43 316 380 8546
Email: <a class="moz-txt-link-abbreviated" href="mailto:elias.karabelas@uni-graz.at">elias.karabelas@uni-graz.at</a>
Web: <a class="moz-txt-link-freetext" href="https://ccl.medunigraz.at/">https://ccl.medunigraz.at/</a></pre>
</body>
</html>