[petsc-dev] Controlling matrix type on different levels of multigrid hierarchy? (Motivation is GPUs)

[Matthew Knepley]

On Wed, Jun 12, 2019 at 3:41 PM Smith, Barry F. via petsc-dev <
petsc-dev at mcs.anl.gov> wrote:

>
>  You'll never benefit from having coarser levels on the CPU but I guess
> you need a general mechanism to try and see that for yourself.
>
>   I think it should be a property of the DM and never let the PCMG see it.
> So something like DMSetUseGPUWhen(dm, gpuvectype, gpumattype,localsize)
> the command line option -dm_use_gpu_when
> [gpumattype,gpuvectype],[localsize] where the localsize defaults to what
> turns out to be generally the best in our experiments (which may depend on
> the machine). Then instead of using dm->mattype and dm->vectype in the code
> we would always inquire with
>
>   DMGetGetType(dm, *type) (same for vectors)
>     if !dm->usegpu *type = dm->mattype;
>     else if localsize > dm->gpulocalsize *type = dm->gpumattype
>     else *type = dm->mat type;
>
>   Note that the current model will also continue work so someone can still
> do -dm_vec_type cuda
>

This is much much worse than the previous mechanism. You have some formula
for making a decision, which is not obvious
to the user, and means that you cannot make different size decisions for
different solves.


>   Alternative model would be to use the GPU vector and matrix types
> everywhere and simply pin the coarser matrices and vecs to the CPU so have
> DMPinToCPUWhen(dm,localsize) and then
>
> DMCreateMatrix_YYY(dm,&mat) would look like
>    MatCreate()
>    MatSetType(*mat,dm->mat type);
>    if localsize < dm->localsize MatPinToCPU(*mat,PETSC_TRUE);
>
> I actually like the second model better. Note that pinning means that no
> space is used on the mat/vec on the GPU unless unpinned. But the mat/vec
> can be unpinned at anytime. With the first model you are stuck with it
> always what you chose initially  stuck on the CPU. In fact you could pin at
> say 1000 localsize, run for a while, and then the code could decide to pin
> or unpin more to find an "optimal" value. Thus I really like pinning.
>

But how will you control it in these dynamically created objects without
giving prefixes, and if you give prefixes how is
this any different than just directly telling them where to go (except that
it is more complicated)?

  Matt


> Also we don't want to hardwire codes/commend lines to particular matrix
> types but allow any GPU that is available so have things like MATGPU that
> default to what is available. So I can do -dm_mat_type gpu -dm_use_gpu_when
> 500 and it uses cuda if that is available and ViennaCL if it is available.
>
>   Better ideas, even more general?
>
>   Note that if dm->localsize which is a property of the dm and its
> partition is different on different processes that is fine, on some process
> you can have a VecGPU and others not or pinned on some and not on others
> for the same vector.
>
>   If you play with either of these models please do it off of my branch,
> not master.
>
>
>
>
>
>
>
>
>
>
>
> > On Jun 12, 2019, at 11:38 AM, Mills, Richard Tran via petsc-dev <
> petsc-dev at mcs.anl.gov> wrote:
> >
> > Colleagues,
> >
> > I think we ought to have a way to control which levels of a PETSc
> multigrid solve happen on the GPU vs. the CPU, as I'd like to keep coarse
> levels on the CPU, but run the calculations for finer levels on the GPU.
> Currently, for a code that is using a DM to manage its grid, one can use
> GPUs inside the application of PCMG by doing putting something like
> >
> >   -dm_mat_type aijcusparse -dm_vec_type cuda
> >
> > on the command line. What I'd like to be able to do is to also control
> which levels get plain AIJ matrices and which get a GPU type, maybe via
> something like
> >
> >   -mg_levels_N_dm_mat_type aijcusparse -mg_levels_N_dm_mat_type cuda
> >
> > for level N. (Being able to specify a range of levels would be even
> nicer, but let's start simple.)
> >
> > Maybe doing the above is as simple as making sure that
> DMSetFromOptions() gets called for the DM for each level. But I think I may
> be not understanding some sort of additional complications. Can someone who
> knows the PCMG framework better chime in? Or do others have ideas for a
> more elegant way of giving this sort of control to the user?
> >
> > Best regards,
> > Richard
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190612/10aacf6e/attachment.html>