[petsc-users] Strong scaling issue cg solver with HYPRE preconditioner (BoomerAMG preconditioning)

[Mark Adams]

On Mon, May 6, 2019 at 7:53 PM Raphael Egan via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear Petsc developer(s),
>
> I am assessing the strong scalability of our incompressible Navier-Stokes
> solver on distributed Octree grids developed at the University of
> California, Santa Barbara.
>
> I intend to show satisfactory strong scaling behavior, even on very large
> numbers of cores for computational problems that involve a mesh that is big
> enough. My scaling test case involves 10 time steps of our solver, applied
> to the three-dimensional flow past a sphere on an extremely fine (but
> non-uniform) mesh made of about 270,000,000 computational cells. We use the
> p4est library for grid management and Petsc solvers for all linear algebra
> problems. I am working on Stampede 2 supercomputer's KNL nodes, using 64
> cores per node.
>
> The most elementary usage of the solver can be summarized as follows:
> - "viscosity step": velocity components are decoupled from each other and
> solved for successively, leading to three successive Poisson problems with
> an added positive diagonal term;
> - "projection step": a projection step is solved to enforce the velocity
> field to be divergence-free.
> The solution of the "projection step" updates the appropriate boundary
> conditions for the "viscosity step" and the process may thus be repeated
> until a desired convergence threshold is reached for the current time step.
>
> Given the discretization that we use for the "viscosity step", the three
> corresponding linear systems are slightly nonsymmetric but have a dominant
> diagonal. We use BiCGStab with PCSOR as a preconditioner invariably for
> those problems (only a few iterations are required for those problems in
> this case). On the contrary, the linear system to be solved for the
> "projection step" is symmetric and positive definite, so we use a conjugate
> gradient solver. PCSOR or PCHYPRE can be chosen by the user as a
> preconditioner for that task. If PCHYPRE is chosen,
> "-pc_hypre_boomeramg_strong_threshold" is set to "0.5",
> "-pc_hypre_boomeramg_coarsen_type" is set to "Falgout" and
> "-pc_hypre_boomeramg_truncfactor" is set to "0.1". Tests on my local
> machine (on a much smaller problem) revealed that PCHYPRE outperforms PCSOR
> (both in terms of execution time and number of iterations) and I expected
> the conclusion to hold for (much) bigger problems on Stampede 2, as well.
>
> However that was wrong: PCSOR actually outperforms PCHYPRE quite
> significantly when using large numbers of cores for my scaling test
> problem. Worse than that, strong scaling of the projection step is actually
> very poor when using PCHYPRE as preconditioner: the time spent on that part
> of the problem grows with the number of cores. See the results here below:
>
> NB: the scaling test that is considered here below consists of 2 time
> steps of the solver starting from an initial state read from disk, of
> around 270,000,000 computational cells. Each time step involves two
> sub-iterations,* i.e.*, every time step solves a first viscosity step (3
> calls to KSPSolve of type bcgs with zero initial guesses), then a first
> projection step (1 call to KSPSolve of type cg with zero initial guess)
> then a second viscosity step (3 more calls to KSPSolve of type bcgs with
> nonzero initial guess), and a second projection step (1 call to KSPSolve of
> type cg with nonzero initial guess). This makes a total of 16 calls to
> KSPSolves, 12 of them are for the viscosity step (not under investigation
> here) and 4 others are for the projection steps
>
> *Mean execution time for the projection step (only):*
> Number of cores: 4096    -->  mean execution time for projection step
> using PCSOR: 125.7 s   ||||||  mean exectution time for projection step
> using PCHYPRE: 192.7 s
> Number of cores: 8192    -->  mean execution time for projection step
> using PCSOR: 81.58 s   ||||||  mean exectution time for projection step
> using PCHYPRE: 281.8 s
> Number of cores: 16384  -->  mean execution time for projection step using
> PCSOR:   48.2 s   ||||||  mean exectution time for projection step using
> PCHYPRE: 311.5 s
>
> The tests were all run with "-ksp_monitor", "-ksp_view" and "-log_view"
> for gathering more information about the processes (see files attached).
> While it is clear that the number of iterations is dramatically reduced
> when using PCHYPRE, it looks like a *very* long time is spent in PCSetUp
> and PCApply when using PCHYPRE for the projection step. When using PCHYPRE,
> the time spent in PCSetUp, PCApply and KSPSolve grows with the number of
> cores that are used. In particular, the time spent in PCSetUp alone when
> using PCHYPRE is larger than the total KSPSolve time when using PCSOR
> (although thousands of iterations are required in that case)...
>
> I am very surprised by these results and I don't quite understand them, I
> used PCHYPRE with similar options and option values in other elliptic
> solver contexts previously with very satisfactory results, even on large
> problems.
>

I assume you mean different hardware. The number of equations that you are
using per core (MPI process presumably) are not crazy and using 64 cores on
a KNL (ie, not 256 hardware threads) is also reasonable.


> I can't figure out why we loose strong scaling in this case?
>
> Would you have some advice about a better preconditioner to use and or a
> better set of options and/or option values to set in order to recover
> strong scaling when using PCHYPRE?
>

We can not performance debug hypre, We just provide and interface to hypre.
So you would have to talk to them.

PETSc has a native AMG solver that we can performance debug (-pc_type
gamg). You could try that if you like.

If you do, run it with -ksp_view and send the output.

The next step in debugging this, if you are not happy, is run with -info
and grep on GAMG and send that output. KNL is a challenging architecture
and it is not surprising to run into problems in going from say haswell to
KNL.

Also, you should start with smaller tests. 4K nodes is not trivial and we
don't know if this is a network (inter-node) or a local (intra-node/socket)
problem. KNL is different on the node. The network is similar.

I would start with looking at one socket and scaling up to 64 cores (MPI
processes) and debug that. You should see good scaling from 1 to 32
processes if the processes get put on each tile, which I have found it does
by default. You should see some less than perfect strong scaling to 64
cores because you are now sharing an L2, you can then go to 128 processes
(sharing L1), etc. if you like. Once you understand what is going on on a
socket you can start scaling onto the network.

Mark


>
> --
> Raphael M. Egan
>
> Department of Mechanical Engineering
> Engineering II Building, Office 2317
> University of California, Santa Barbara
> Santa Barbara, CA 93106
> Email: raphaelegan at ucsb.edu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190506/a9a09334/attachment.html>