[petsc-dev] Controlling matrix type on different levels of multigrid hierarchy? (Motivation is GPUs)

[Smith, Barry F.]

 You'll never benefit from having coarser levels on the CPU but I guess you need a general mechanism to try and see that for yourself.

  I think it should be a property of the DM and never let the PCMG see it. So something like DMSetUseGPUWhen(dm, gpuvectype, gpumattype,localsize)  the command line option -dm_use_gpu_when [gpumattype,gpuvectype],[localsize] where the localsize defaults to what turns out to be generally the best in our experiments (which may depend on the machine). Then instead of using dm->mattype and dm->vectype in the code we would always inquire with 

  DMGetGetType(dm, *type) (same for vectors) 
    if !dm->usegpu *type = dm->mattype;
    else if localsize > dm->gpulocalsize *type = dm->gpumattype 
    else *type = dm->mat type;

  Note that the current model will also continue work so someone can still do -dm_vec_type cuda 

  Alternative model would be to use the GPU vector and matrix types everywhere and simply pin the coarser matrices and vecs to the CPU so have
DMPinToCPUWhen(dm,localsize) and then

DMCreateMatrix_YYY(dm,&mat) would look like
   MatCreate()
   MatSetType(*mat,dm->mat type);
   if localsize < dm->localsize MatPinToCPU(*mat,PETSC_TRUE);

I actually like the second model better. Note that pinning means that no space is used on the mat/vec on the GPU unless unpinned. But the mat/vec can be unpinned at anytime. With the first model you are stuck with it always what you chose initially  stuck on the CPU. In fact you could pin at say 1000 localsize, run for a while, and then the code could decide to pin or unpin more to find an "optimal" value. Thus I really like pinning.

Also we don't want to hardwire codes/commend lines to particular matrix types but allow any GPU that is available so have things like MATGPU that default to what is available. So I can do -dm_mat_type gpu -dm_use_gpu_when 500 and it uses cuda if that is available and ViennaCL if it is available.

  Better ideas, even more general? 

  Note that if dm->localsize which is a property of the dm and its partition is different on different processes that is fine, on some process you can have a VecGPU and others not or pinned on some and not on others for the same vector.

  If you play with either of these models please do it off of my branch, not master.




 






> On Jun 12, 2019, at 11:38 AM, Mills, Richard Tran via petsc-dev <petsc-dev at mcs.anl.gov> wrote:
> 
> Colleagues,
> 
> I think we ought to have a way to control which levels of a PETSc multigrid solve happen on the GPU vs. the CPU, as I'd like to keep coarse levels on the CPU, but run the calculations for finer levels on the GPU. Currently, for a code that is using a DM to manage its grid, one can use GPUs inside the application of PCMG by doing putting something like
> 
>   -dm_mat_type aijcusparse -dm_vec_type cuda
> 
> on the command line. What I'd like to be able to do is to also control which levels get plain AIJ matrices and which get a GPU type, maybe via something like
> 
>   -mg_levels_N_dm_mat_type aijcusparse -mg_levels_N_dm_mat_type cuda
> 
> for level N. (Being able to specify a range of levels would be even nicer, but let's start simple.)
> 
> Maybe doing the above is as simple as making sure that DMSetFromOptions() gets called for the DM for each level. But I think I may be not understanding some sort of additional complications. Can someone who knows the PCMG framework better chime in? Or do others have ideas for a more elegant way of giving this sort of control to the user?
> 
> Best regards,
> Richard