[petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

[Brian Van Straalen]

So Baky and I have been at the Brookhaven GPU Hackathon now for three days,
talking to everyone.  We have also been emailing with people who will
respond to us from the hypre team and the PETSc team, as well as reading
every blog post and mail archive and message board and from what we can
tell, a distributed AMG preconditioner will not be available for us on a
Summit platform for the foreseeable future.

There is a hypre build for CUDA, but it has a problem with it's use of
CUSP, and nobody seems to be working on it.

PETSc has some .cu cuda files for the SpMV and Vector operations but the
preconditioners are limited to point Jacobi and similar simple operations
and a version of ILU.  Neither works for our stiff projection in the
embedded boundary algorithms.   We built it and ran it and PETSc takes
several hundred iterations to get the residual down by a factor of 6.  We
need to get down to more like 10e-11 for this solve.

The AMG being worked on by the NVIDIA team is not targeted for multi-node
solving, and I haven't heard back from them in months.

  We are left with two options as I see it to meet our ECP Milestones:

1. Build yet another interface, this time to see if there is a distributed
GPU AMG preconditioner in Trilinos

 2. Implement our own special-purpose EB-GMG solver written in Chombo.

I would love to be wrong about all this.

Brian

-- 
Brian Van Straalen         Lawrence Berkeley Lab
BVStraalen at lbl.gov        Computational Research
(510) 486-4976            Division (crd.lbl.gov)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180919/661f2cb2/attachment.html>