<div dir="ltr"><div dir="ltr">On Wed, Jun 12, 2019 at 3:41 PM Smith, Barry F. via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
You'll never benefit from having coarser levels on the CPU but I guess you need a general mechanism to try and see that for yourself.<br>
<br>
I think it should be a property of the DM and never let the PCMG see it. So something like DMSetUseGPUWhen(dm, gpuvectype, gpumattype,localsize) the command line option -dm_use_gpu_when [gpumattype,gpuvectype],[localsize] where the localsize defaults to what turns out to be generally the best in our experiments (which may depend on the machine). Then instead of using dm->mattype and dm->vectype in the code we would always inquire with <br>
<br>
DMGetGetType(dm, *type) (same for vectors) <br>
if !dm->usegpu *type = dm->mattype;<br>
else if localsize > dm->gpulocalsize *type = dm->gpumattype <br>
else *type = dm->mat type;<br>
<br>
Note that the current model will also continue work so someone can still do -dm_vec_type cuda <br></blockquote><div><br></div><div>This is much much worse than the previous mechanism. You have some formula for making a decision, which is not obvious</div><div>to the user, and means that you cannot make different size decisions for different solves.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Alternative model would be to use the GPU vector and matrix types everywhere and simply pin the coarser matrices and vecs to the CPU so have<br>
DMPinToCPUWhen(dm,localsize) and then<br>
<br>
DMCreateMatrix_YYY(dm,&mat) would look like<br>
MatCreate()<br>
MatSetType(*mat,dm->mat type);<br>
if localsize < dm->localsize MatPinToCPU(*mat,PETSC_TRUE);<br>
<br>
I actually like the second model better. Note that pinning means that no space is used on the mat/vec on the GPU unless unpinned. But the mat/vec can be unpinned at anytime. With the first model you are stuck with it always what you chose initially stuck on the CPU. In fact you could pin at say 1000 localsize, run for a while, and then the code could decide to pin or unpin more to find an "optimal" value. Thus I really like pinning.<br></blockquote><div><br></div><div>But how will you control it in these dynamically created objects without giving prefixes, and if you give prefixes how is</div><div>this any different than just directly telling them where to go (except that it is more complicated)?</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Also we don't want to hardwire codes/commend lines to particular matrix types but allow any GPU that is available so have things like MATGPU that default to what is available. So I can do -dm_mat_type gpu -dm_use_gpu_when 500 and it uses cuda if that is available and ViennaCL if it is available.<br>
<br>
Better ideas, even more general? <br>
<br>
Note that if dm->localsize which is a property of the dm and its partition is different on different processes that is fine, on some process you can have a VecGPU and others not or pinned on some and not on others for the same vector.<br>
<br>
If you play with either of these models please do it off of my branch, not master.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> On Jun 12, 2019, at 11:38 AM, Mills, Richard Tran via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br>
> <br>
> Colleagues,<br>
> <br>
> I think we ought to have a way to control which levels of a PETSc multigrid solve happen on the GPU vs. the CPU, as I'd like to keep coarse levels on the CPU, but run the calculations for finer levels on the GPU. Currently, for a code that is using a DM to manage its grid, one can use GPUs inside the application of PCMG by doing putting something like<br>
> <br>
> -dm_mat_type aijcusparse -dm_vec_type cuda<br>
> <br>
> on the command line. What I'd like to be able to do is to also control which levels get plain AIJ matrices and which get a GPU type, maybe via something like<br>
> <br>
> -mg_levels_N_dm_mat_type aijcusparse -mg_levels_N_dm_mat_type cuda<br>
> <br>
> for level N. (Being able to specify a range of levels would be even nicer, but let's start simple.)<br>
> <br>
> Maybe doing the above is as simple as making sure that DMSetFromOptions() gets called for the DM for each level. But I think I may be not understanding some sort of additional complications. Can someone who knows the PCMG framework better chime in? Or do others have ideas for a more elegant way of giving this sort of control to the user?<br>
> <br>
> Best regards,<br>
> Richard<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>