[petsc-users] [MEMORY LEAK, INTERFACE BUG] Petsc + Boomeramg + Fieldsplit on ARCHER2

[Matthew Knepley]

On Thu, May 12, 2022 at 9:09 AM Karabelas, Elias (
elias.karabelas at uni-graz.at) <elias.karabelas at uni-graz.at> wrote:

> Dear Team,
>
> I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on
> the ARCHER2 cluster.
>
> These are my options for solving a Navier-Stokes-like system and it ran
> fine on other clusters.
>
> -ksp_type fgmres
> -ksp_pc_side right
> -ksp_initial_guess_nonzero 0
> -ksp_lag_norm
> -ksp_gmres_restart 100
> -ksp_monitor :block_system_ksp_its.log
> -pc_type fieldsplit
> -pc_fieldsplit_type schur
> -pc_fieldsplit_schur_fact_type full
> -pc_fieldsplit_schur_precondition selfp
> -fieldsplit_0_ksp_type preonly
> -fieldsplit_0_pc_type hypre
> -fieldsplit_0_pc_hypre_type boomeramg
> -fieldsplit_0_pc_hypre_boomeramg_max_iter 1
> -fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS
> -fieldsplit_0_pc_hypre_boomeramg_tol 0.0
> -fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i
> -fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7
> -fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel
> -fieldsplit_1_mat_schur_complement_ainv_type blockdiag
> -fieldsplit_1_ksp_type gmres
> -fieldsplit_1_pc_type sor
> -fieldsplit_1_ksp_max_it 30
> -fieldsplit_1_ksp_rtol 1e-1
> -fieldsplit_1_ksp_atol 0.0
> -fieldsplit_1_ksp_pc_side right
>
>
> However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit
> my jobs. I did some digging with valgrind I found out that in PCApply when
> BoomerAMG is called and HYPRE is compiled with the option
> "--enable-persistent 1" there are a lot of calloc's on HYPRE's side that
> are not free'd even after the PC gets destroyed from Petsc side. This seems
> to be the case on the vendor-shipped Petsc on ARCHER2. Building my own
> Petsc and Hypre on the cluster resolves this issue with the downside of
> loosing around 15% of speed.
>
Hi Elias,

Is your self-compiled PETSc a newer version? If so, it sounds like this can
be resolved by getting the admins on ARCHER to update the PETSc. They have
been pretty responsive for the Firedrake people, so I don't think this will
be hard.

  Thanks,

    Matt


> I thought the developers might be interested in this bug, but I do not
> know who is to be adressed here, the hypre developers, the petsc
> developers, or the ARCHER2 admins.
>
> Best regards
>
> Elias
>
> --
> Dr. Elias Karabelas
> Research Associate
> University of Graz
> Institute of Mathematics and Scientific Computing
> Heinrichstraße 36
> A-8010 Graz
> Austria
>
> Phone: +43 316 380 8546
> Email: elias.karabelas at uni-graz.at
> Web:  https://ccl.medunigraz.at/
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220512/6657c0b4/attachment.html>