<div dir="ltr">* snes/ex56 runs a convergence study and confusingly sets the options manually, thus erasing your -ex56_dm_refine.<div><br></div><div>* To refine, use -max_conv_its N <3>, this sets the number of steps of refinement. That is, the length of the convergence study</div><div><br></div><div>* You can adjust where it starts from with -cells i,j,k <1,1,1></div><div><div>You do want to set this if you have multiple MPI processes so that the size of this mesh is the number of processes. That way it starts with one cell per process and refines from there.<br></div><div><br></div><div>* GPU speedup is all about subdomain size. AMG has lots of kernel launches and you need to overcome this before you get net gain.</div><div>Very rough numbers: I see a speedup of about 5-10x with a few million equations per GPU.</div><div>As Matt said the assembly is on the CPU and ex56 gets really slow on larger problems. Be prepared to run the largest case for close to an hour.</div><div>This setup is not measured in KSP[SNES]Solve in the -log_view output so look at that.</div><div>When you do this convergence study there will be a new stage created for each refinement, so one run will give you a range of problem size data.</div><div>Each refinement step increases the problem size by 8x so when the solve times increase by ~8x then that tells you you are past the latency dominated regime. You want to get into that to see gain.</div><div><br></div><div>* The end of the source file has example parameters that you should use (the gamg one)</div><div><br></div><div>* src/snes/tests/ex13.c is designed to be a benchmark test and it partitions the problem better in parallel and has modern Plex usage. If you are doing large scale parallelism then you should use this. </div><div>(It is a little hard to understand. Not well documented.)</div><div><br></div><div>Hope that helps,</div><div>Mark</div><div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 6, 2021 at 9:05 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks, Matt,<div><br></div><div>Sorry, I still have more questions on this example. How to refine mesh to make the problem larger? </div><div><br></div><div>I tried the following options, and none of them worked. I might do something wrong.</div><div><br></div><div>-ex56_dm_refine 9<br></div><div><br></div><div>and </div><div><br></div><div>-dm_refine 4</div></div></blockquote><div><br></div><div>The mesh handling in this example does not conform to the others, but it appears that</div><div><br></div><div> -ex56_dm_refine <k></div><div><br></div><div>should take effect at</div><div><br></div><div> <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381" target="_blank">https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381</a></div><div><br></div></div></div></blockquote><div><br></div><div>I was puzzled about this because DMSetFromOptions does not seem to trigger -ex56_dm_refine.</div><div><br></div><div>I did a search, and could not find where we call "<span> </span>-ex56_dm_refine" in PETSc. </div><div><br></div><div>I got the same result by running the following two combinations:</div><div><br></div><div>1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10<br><br>2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0<br></div><div><br></div><div>Thanks,</div><div><br></div><div>Fande</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div>unless you are setting max_conv_its to 0 somehow.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div><br></div><div>Fande</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks, Matt</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear PETSc team,<div><br><div>I am interested in a careful evaluation of PETSc GPU performance in our INL cluster. </div><div><br></div><div>Any example in PETSc that can show GPU speedup with solving a nonlinear equation? </div><div><br></div><div>I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I tried that, but I could not find any speedup using the GPU. I could attach some results of "log_view" later if we would like to see that. </div></div></div></blockquote><div><br></div><div>We should note that you will only see speedup in the solver, so that problem has to be pretty large. I believe Mark has good results with it.</div><div>The assembly is still all on the CPU. I am working on this over break, and hope to have a CEED version of it by the new year.</div></div></div></blockquote><div><br></div><div>Are both function and matrix assmelies on CPU? Or just the matrix assembly?</div></div></div></blockquote><div><br></div><div>There is no GPU assembly right now.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>OK, I will try to check the solver part </div><div><br></div><div>Thanks, again </div><div><br></div><div>Fande </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Appreciate any instructions/comments about running a simple PETSc GPU example to get a speedup. </div><div><br></div><div>Thanks,</div><div><br></div><div>Fande</div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div>