[petsc-dev] Controlling matrix type on different levels of multigrid hierarchy? (Motivation is GPUs)

[Matthew Knepley]

On Wed, Jun 12, 2019 at 12:47 PM Dave May via petsc-dev <
petsc-dev at mcs.anl.gov> wrote:

> Hi Richard,
>
> On Wed, 12 Jun 2019 at 18:38, Mills, Richard Tran via petsc-dev <
> petsc-dev at mcs.anl.gov> wrote:
>
>> Colleagues,
>>
>> I think we ought to have a way to control which levels of a PETSc
>> multigrid solve happen on the GPU vs. the CPU, as I'd like to keep coarse
>> levels on the CPU, but run the calculations for finer levels on the GPU.
>> Currently, for a code that is using a DM to manage its grid, one can use
>> GPUs inside the application of PCMG by doing putting something like
>>
>>   -dm_mat_type aijcusparse -dm_vec_type cuda
>>
>> on the command line. What I'd like to be able to do is to also control
>> which levels get plain AIJ matrices and which get a GPU type, maybe via
>> something like
>>
>>   -mg_levels_N_dm_mat_type aijcusparse -mg_levels_N_dm_mat_type cuda
>>
>> for level N. (Being able to specify a range of levels would be even
>> nicer, but let's start simple.)
>>
>> Maybe doing the above is as simple as making sure that DMSetFromOptions()
>> gets called for the DM for each level.
>>
>
> I think you'd need really need a unique options prefix for each DM. Either
> set by you if you created the hierarchy or internally defined / set of the
> dms are constructed during PCSetUp_MG
>

Yes, the coarse DMs should have the same prefix as the coarse solvers.
However, note that this would break the existing usage for GPUs.


> But I think I may be not understanding some sort of additional
>> complications. Can someone who knows the PCMG framework better chime in? Or
>> do others have ideas for a more elegant way of giving this sort of control
>> to the user?
>>
>
> Are you building the DM hierarchy yourself, or is PETSc generation the
> coarse DMs for you. If the former, then you can always choose the Mat type
> via DMSetMatType().
>

He is using DMDA, so they are created automatically.

  Matt


> Cheers
> Dave
>
>
>>
>> Best regards,
>> Richard
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190612/4d12922f/attachment.html>