[petsc-users] [MEMORY LEAK, INTERFACE BUG] Petsc + Boomeramg + Fieldsplit on ARCHER2

[Karabelas, Elias (elias.karabelas@uni-graz.at)]

Dear Team,

I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on the ARCHER2 cluster.

These are my options for solving a Navier-Stokes-like system and it ran fine on other clusters.

-ksp_type fgmres
-ksp_pc_side right
-ksp_initial_guess_nonzero 0
-ksp_lag_norm
-ksp_gmres_restart 100
-ksp_monitor :block_system_ksp_its.log
-pc_type fieldsplit
-pc_fieldsplit_type schur
-pc_fieldsplit_schur_fact_type full
-pc_fieldsplit_schur_precondition selfp
-fieldsplit_0_ksp_type preonly
-fieldsplit_0_pc_type hypre
-fieldsplit_0_pc_hypre_type boomeramg
-fieldsplit_0_pc_hypre_boomeramg_max_iter 1
-fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS
-fieldsplit_0_pc_hypre_boomeramg_tol 0.0
-fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i
-fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7
-fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel
-fieldsplit_1_mat_schur_complement_ainv_type blockdiag
-fieldsplit_1_ksp_type gmres
-fieldsplit_1_pc_type sor
-fieldsplit_1_ksp_max_it 30
-fieldsplit_1_ksp_rtol 1e-1
-fieldsplit_1_ksp_atol 0.0
-fieldsplit_1_ksp_pc_side right


However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit my jobs. I did some digging with valgrind I found out that in PCApply when BoomerAMG is called and HYPRE is compiled with the option "--enable-persistent 1" there are a lot of calloc's on HYPRE's side that are not free'd even after the PC gets destroyed from Petsc side. This seems to be the case on the vendor-shipped Petsc on ARCHER2. Building my own Petsc and Hypre on the cluster resolves this issue with the downside of loosing around 15% of speed.

I thought the developers might be interested in this bug, but I do not know who is to be adressed here, the hypre developers, the petsc developers, or the ARCHER2 admins.

Best regards

Elias

--
Dr. Elias Karabelas
Research Associate
University of Graz
Institute of Mathematics and Scientific Computing
Heinrichstraße 36
A-8010 Graz
Austria

Phone: +43 316 380 8546
Email: elias.karabelas at uni-graz.at<mailto:elias.karabelas at uni-graz.at>
Web:  https://ccl.medunigraz.at/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220512/db6e6ae7/attachment.html>