<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Barry, Mark and Matt,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thank you all for the suggestions. I will modify the code so we can pass runtime options.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Cho<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Friday, June 30, 2023 7:01 AM<br>
<b>To:</b> Mark Adams <mfadams@lbl.gov><br>
<b>Cc:</b> Matthew Knepley <knepley@gmail.com>; Ng, Cho-Kuen <cho@slac.stanford.edu>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div style="line-break:after-white-space">
<div><br>
</div>
Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because
Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
<div><br>
<blockquote type="cite">
<div>On Jun 30, 2023, at 9:30 AM, Mark Adams <mfadams@lbl.gov> wrote:</div>
<br class="x_Apple-interchange-newline">
<div>
<div dir="ltr">PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:<br>
<div><br>
</div>
<div>a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
<div id="x_m_-2363626647532502450m_5514417947815199577appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-2363626647532502450m_5514417947815199577divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 8:32 PM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read
in the options?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
<div id="x_m_-2363626647532502450m_5514417947815199577x_appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-2363626647532502450m_5514417947815199577x_divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 5:55 PM<br>
<b>To:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div dir="ltr">Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
<div><br>
</div>
<div>The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.</div>
<div><br>
</div>
<div>The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I installed PETSc on Perlmutter using "spack install
<span style="background:rgb(255,255,255)">petsc+cuda+zoltan</span><span style="background-color:rgb(255,255,255)">"
</span>and used it by "<span style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; background:rgb(255,255,255)">spack load petsc/fwge6pf</span>". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can
get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use
the GPU backend?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="x_gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>