[petsc-dev] Current status: GPUs for PETSc

[Lawrence Mitchell]

On 05/11/2012 14:00, Matthew Knepley wrote:
> 
> On Mon, Nov 5, 2012 at 6:27 AM, Lawrence Mitchell
> <lawrence.mitchell at ed.ac.uk <mailto:lawrence.mitchell at ed.ac.uk>> wrote:
> 
>     Hi Karli, and others,
> 
>     On 05/11/2012 01:51, Karl Rupp wrote:
> 
>     ...
> 
>     > That's it for now, after some more refining I'll start with a careful
>     > migration of the code/concepts into PETSc. Comments are, of course,
>     > always welcome.
> 
>     So we're working on FE assembly + solve on GPUs using fenics kernels
>     (github.com/OP2/PyOP2 <http://github.com/OP2/PyOP2>).  For the GPU
>     solve, it would be nice if we could
>     backdoor assembled matrices straight on to the GPU.  That is, create a Mat
>     saying "this is the sparsity pattern" and then, rather than calling
>     MatSetValues on the host, just pass a pointer to the device data.
> 
> 
> Yes, I was doing the same thing. GPU assembly is not really any faster than CPU,
> so I did not publish it, but not having to transfer the values back is very
> nice.

Our experience is that we can outperform Dolfin's CPU assembly by a
reasonable amount, so for us it does seem worthwhile.  But given that we're
also able to generate CPU assembly+solve, it would be nice to use PETSc in
all cases.  And of course, even in the case where GPU and CPU assembly are
the same speed, if the GPU solve goes faster it's nice to be able to avoid
all the host-device transfers that CPU assembly necessitates for non-linear
systems.

> In Karl's scheme its easy, since you just go in and get the device pointer
> from the
> handle. We can talk about a user level interface after you verify that this
> works and
> is what you want.

That should be enough I think.  In the MPI case I guess it means providing
the relevant data structure for the two MatSeq (diagonal and off-diagonal) bits.

Lawrence

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.