<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.<div><br><blockquote type="cite"><div>On Jun 30, 2023, at 9:30 AM, Mark Adams <mfadams@lbl.gov> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr">PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:<br><div><br></div><div>a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Mark,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?</div></div></div></blockquote><div><br></div><div>PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Cho<br>
</div>
<div id="m_-2363626647532502450m_5514417947815199577appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-2363626647532502450m_5514417947815199577divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 8:32 PM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Mark,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Cho<br>
</div>
<div id="m_-2363626647532502450m_5514417947815199577x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-2363626647532502450m_5514417947815199577x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 5:55 PM<br>
<b>To:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div dir="ltr">Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
<div><br>
</div>
<div>The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.</div>
<div><br>
</div>
<div>The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
I installed PETSc on Perlmutter using "spack install <span style="background:rgb(255,255,255)">
petsc+cuda+zoltan</span><span style="background-color:rgb(255,255,255)">" </span>
and used it by "<span style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; background: rgb(255, 255, 255);">spack load petsc/fwge6pf</span>". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that
I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to
use the GPU backend?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Thanks,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">
Cho<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</div></blockquote></div><br></body></html>