[petsc-users] Slow convergence while parallel computations.

[Mark Adams]

as far as GAMG:

* Pierre is right, start with the defaults. AMG does take tuning. 2D and 3D
are very different, among other things. You can run with '-info :pc', which
is very noisy and grep on "GAMG" and send me the result. (Oh Lawrence
recommend this, just send it)
  - ICC is not good because it has to scale the diagonal to avoid negative
pivots (even for SPD matrices that are not M matrices at least). This is
probably a problem.
  - As Lawrence indicates, jumps in coefficients can be hard for generic
AMG.
  - And yes, -pc_gamg_threshold is an important parameter for
homogeneous problems and can be additionally important for inhomogeneous
problems to get the AMG method to "see" your jumps.

* The memory problems are from squaring the graph, among other things,
which you usually need to do for elasticity unless you have high order
elements, maybe.

* You can try  PCBDDC, DD methods are nice for elasticity.

* You can try hypre. Good solver but 3D elasticity is not its strength.

* As far as poor scaling, you have large subdomains, I assume the load
balancing is decent, and the network is not crazy. This might be a lot of
setup cost. Run with -log_view and look at the KSPSolve and MatPtAP...
 - The solver will call the setup (MatPtAP), if it has not been done yet,
so that it gets folded in. You can call KSPSetup() before KSPSolve() to get
the timings separated. I you are using the solver (eg, not full Newton)
then the setup gets amortized.

Mark

On Wed, Sep 1, 2021 at 5:02 AM Lawrence Mitchell <wence at gmx.li> wrote:

>
>
> > On 1 Sep 2021, at 09:42, Наздрачёв Виктор <numbersixvs at gmail.com> wrote:
> >
> > I have a 3D elasticity problem with heterogeneous properties.
>
> What does your coefficient variation look like? How large is the contrast?
>
> > There is unstructured grid with aspect ratio varied from 4 to 25. Zero
> Dirichlet BCs  are imposed on bottom face of mesh. Also, Neumann (traction)
> BCs are imposed on side faces. Gravity load is also accounted for. The grid
> I use consists of 500k cells (which is approximately 1.6M of DOFs).
> >
> > The best performance and memory usage for single MPI process was
> obtained with HPDDM(BFBCG) solver and bjacobian + ICC (1) in subdomains as
> preconditioner, it took 1 m 45 s and RAM 5.0 GB. Parallel computation with
> 4 MPI processes took 2 m 46 s when using 5.6 GB of RAM. This because of
> number of iterations required to achieve the same tolerance is
> significantly increased.
>
> How many iterations do you have in serial (and then in parallel)?
>
> > I`ve also tried PCGAMG (agg) preconditioner with ICС (1)
> sub-precondtioner. For single MPI process, the calculation took 10 min and
> 3.4 GB of RAM. To improve the convergence rate, the nullspace was attached
> using MatNullSpaceCreateRigidBody and MatSetNearNullSpace subroutines.
> This has reduced calculation time to 3 m 58 s when using 4.3 GB of RAM.
> Also, there is peak memory usage with 14.1 GB, which appears just before
> the start of the iterations. Parallel computation with 4 MPI processes took
> 2 m 53 s when using 8.4 GB of RAM. In that case the peak memory usage is
> about 22 GB.
>
> Does the number of iterates increase in parallel? Again, how many
> iterations do you have?
>
> > Are there ways to avoid decreasing of the convergence rate for bjacobi
> precondtioner in parallel mode? Does it make sense to use hierarchical or
> nested krylov methods with a local gmres solver (sub_pc_type gmres) and
> some sub-precondtioner (for example, sub_pc_type bjacobi)?
>
> bjacobi is only a one-level method, so you would not expect
> process-independent convergence rate for this kind of problem. If the
> coefficient variation is not too extreme, then I would expect GAMG (or some
> other smoothed aggregation package, perhaps -pc_type ml (you need
> --download-ml)) would work well with some tuning.
>
> If you have extremely high contrast coefficients you might need something
> with stronger coarse grids. If you can assemble so-called Neumann matrices (
> https://petsc.org/release/docs/manualpages/Mat/MATIS.html#MATIS) then you
> could try the geneo scheme offered by PCHPDDM.
>
> > Is this peak memory usage expected for gamg preconditioner? is there any
> way to reduce it?
>
> I think that peak memory usage comes from building the coarse grids. Can
> you run with `-info` and grep for GAMG, this will provide some output that
> more expert GAMG users can interpret.
>
> Lawrence
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210901/66e53edc/attachment.html>