<div dir="ltr"><div>Hello everyone,</div><div><span></span>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I seem to have a
problem of increasing memory size with the linear solver. I double
check every single part of my code and I isolated the problem to
KSPSolve command.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">A bit of
context/background regarding my algorithm.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I am writing a
finite volume numerical solver for incompressible two phase flow. I
am mainly solving liquid sloshing and waves impact problems. Until
recently most of simulations were 2D with max cell count of 100K
cells but now I am considering solving 3D problems with cell counts
between 6M~10M cells.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">The algorithm is as
follows:</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">setup ksp</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">setup pc</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">loop 1 → number of
timesteps</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 1- // do some work</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 2- create the
pressure poisson matrix</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 3 – loop: iloop 1
to 4</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 4-1- solve the
matrix if ( iloop == 1) update the precond</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 4-2- // do some
work</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 4-3- update rhs(no
matrix or precond update)</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 4- do the next
timestep and go back to step 1</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><i><u><b>From many
online posts many people recommended KSPCG with PCHYPRE with the
following options</b></u></i></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> // PC Setup</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PCCreate(PETSC_COMM_WORLD,
&preconditioner);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PCSetType(preconditioner,
PCHYPRE);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PCHYPRESetType(preconditioner,
"boomeramg");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_strong_threshold", "0.4");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_agg_nl", "3");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_agg_num_paths", "4");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> </p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_interp_type", "ext+i");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_truncfactor", "0.3");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_max_iter", "1");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_relax_type_all",</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> "SOR/Jacobi");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_grid_sweeps_down", "1");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_max_iter", "1");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PetscOptionsSetValue(NULL,
"-pc_hypre_boomeramg_rtol", "0");</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetPC(iterSolver,
preconditioner);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetOperators(iterSolver,
A, A);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> // KSP Setup</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetType(iterSolver,
KSPCG);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetInitialGuessNonzero(iterSolver,
PETSC_TRUE);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetNormType(iterSolver,
KSP_NORM_UNPRECONDITIONED);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetFromOptions(iterSolver);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> PCSetUp(preconditioner);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetUp(iterSolver);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetTolerances(iterSolver,
1e-18 /*reltol*/, 1e-12 /*abstol*/,</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> 1e5
/*divergencetol*/, 50);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><b>When solving the
linear system for the first time during the timestep (iloop =1), the
following commands are used so that the preconditioner is
reconstructed based on the new matrix values.</b></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetReusePreconditioner(iterSolver,
PETSC_FALSE);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetOperators(iterSolver,
A, A); // set the new matrix</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSolve(iterSolver,
*RHS, *x);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><i><u>Otherwise</u></i></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSetReusePreconditioner(iterSolver,
PETSC_TRUE);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"> KSPSolve(iterSolver,
*RHS, *x);</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><b>It should be
noted that matrix non-zero structure remains the same throught the
simulation but the values change every timestep.</b></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">Now My problem is
that memory size keep changing throught the simulation utill my
workstation evetually runs out of memory.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I tracked down the
memory increase to the step which solve the linear system of
equations.
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><i> KSPSetReusePreconditioner(iterSolver,
PETSC_FALSE);</i></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><i> KSPSetOperators(iterSolver,
A, A); // set the new matrix</i></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><i> KSPSolve(iterSolver,
*RHS, *x);</i></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">My current problem
is a sloshing problem with cell count of 1.2M cells running on
workstation with 40 Cores and 94GB of RAM.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">The memory increase I
observed did not occur every single time step but more like every 3-10 time step. The increase each time is somewhere from 0.51MB~3.0MB per
mpi process and not not all process show memory change. For example
in time step id 14 , proc0, proc6, proc13, proc24 show change but the
others don’t. On average 10~15 processes show memory change while
the others do not.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">Over 1000 time steps
the solver use up additional 10GB of memory and still increasing
until the system runs out of memory after nearly 10K time steps.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I wanted to try
another preconditioner PCJACOBI which ultimately failed due to max
number of iterations but at least no memory change was observed over
50 time steps. <br></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">Since I am using
multigrid I expected the increase in memory required by the linear
solver, I just assumed that after enough number of timesteps it would
settle on some value. But it just kept increasing util finally the
system ran out of memory.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I monitored the
memory usage of other parts/components of the code and no memory
change occurs at all except at code section above.</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I configure PETSC
with following options</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"><font color="#000000"><span style="background:rgb(255,255,255) none repeat scroll 0% 0%">PETSC_ARCH=release3
-with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native"
CXXOPTFLAGS="-O3 -march=native -mtune=native"
FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc
--w</span></font><br>
ith-cxx=mpicxx --with-fc=mpif90
--download-metis --download-hypre<br>
</font><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">I tried configuring
it in debug and compiling my code also in debug mode and running as
the FAQ suggests but the code ended up being too slow so I shut it
down after running it for 50mins and not even one timestep was
completed.</p><p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%">So to track down the
memory leak/increase I used<font face="monospace"> </font>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"> PetscMemoryGetCurrentUsage(&mem1);</font></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"> //
some code </font>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"> PetscMemoryGetCurrentUsage(&mem2);</font></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"> memChnge
= mem2 – mem1;</font></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><br>
</p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace">I
believe it could be one of the following possibilities:</font></p>
<ol><li><p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace">A
memory leak </font><font face="monospace">(</font><font face="monospace">bug</font><font face="monospace">)</font><font face="monospace">
in the petsc library ( unlikely otherwise someone would have found
it by now ) </font>
</p>
</li><li><p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace">Some
thing to do with setting of the boomerAMG Hypre preconditioner which
make</font><font face="monospace">s</font><font face="monospace"> it </font><font face="monospace">unnecessarily add</font><font face="monospace">
additional levels. The settings I am using, I got from an online
post I found so I don’t really understand it.</font></p>
<p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace"><a href="https://mooseframework.inl.gov/application_development/hypre.html">https://mooseframework.inl.gov/application_development/hypre.html</a></font></p>
</li><li><p style="margin-bottom:0in;line-height:100%;background:transparent none repeat scroll 0% 0%"><font face="monospace">S</font><font face="monospace">ome
problem with the sequence of commands I am using for the solution of
linear system of eqns that makes allocate additional memory with
freeing the old one.</font></p></li></ol><div><font face="monospace">Thanks</font></div><div><font face="monospace"></font><br></div>
</div></div>