[petsc-users] [MEMORY LEAK, INTERFACE BUG] Petsc + Boomeramg + Fieldsplit on ARCHER2

Karabelas, Elias (elias.karabelas@uni-graz.at) elias.karabelas at uni-graz.at
Thu May 12 02:13:10 CDT 2022

Dear Team,

I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on the ARCHER2 cluster.

These are my options for solving a Navier-Stokes-like system and it ran fine on other clusters.

-ksp_type fgmres
-ksp_pc_side right
-ksp_initial_guess_nonzero 0
-ksp_gmres_restart 100
-ksp_monitor :block_system_ksp_its.log
-pc_type fieldsplit
-pc_fieldsplit_type schur
-pc_fieldsplit_schur_fact_type full
-pc_fieldsplit_schur_precondition selfp
-fieldsplit_0_ksp_type preonly
-fieldsplit_0_pc_type hypre
-fieldsplit_0_pc_hypre_type boomeramg
-fieldsplit_0_pc_hypre_boomeramg_max_iter 1
-fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS
-fieldsplit_0_pc_hypre_boomeramg_tol 0.0
-fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i
-fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7
-fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel
-fieldsplit_1_mat_schur_complement_ainv_type blockdiag
-fieldsplit_1_ksp_type gmres
-fieldsplit_1_pc_type sor
-fieldsplit_1_ksp_max_it 30
-fieldsplit_1_ksp_rtol 1e-1
-fieldsplit_1_ksp_atol 0.0
-fieldsplit_1_ksp_pc_side right

However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit my jobs. I did some digging with valgrind I found out that in PCApply when BoomerAMG is called and HYPRE is compiled with the option "--enable-persistent 1" there are a lot of calloc's on HYPRE's side that are not free'd even after the PC gets destroyed from Petsc side. This seems to be the case on the vendor-shipped Petsc on ARCHER2. Building my own Petsc and Hypre on the cluster resolves this issue with the downside of loosing around 15% of speed.

I thought the developers might be interested in this bug, but I do not know who is to be adressed here, the hypre developers, the petsc developers, or the ARCHER2 admins.

Best regards


Dr. Elias Karabelas
Research Associate
University of Graz
Institute of Mathematics and Scientific Computing
Heinrichstraße 36
A-8010 Graz

Phone: +43 316 380 8546
Email: elias.karabelas at uni-graz.at<mailto:elias.karabelas at uni-graz.at>
Web:  https://ccl.medunigraz.at/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220512/db6e6ae7/attachment.html>

More information about the petsc-users mailing list