<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Matt,</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
After inserting 2 lines in the code:</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof ContentPasted0" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); <br>
<div class="ContentPasted0"> ierr = MatSetFromOptions(A);CHKERRQ(ierr);</div>
<div class="ContentPasted0"> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,</div>
<div class="ContentPasted0"> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);</div>
<div class="ContentPasted0"><br>
</div>
<div class="ContentPasted0 ContentPasted1">"There are no unused options." However, there is no improvement on the GPU performance.</div>
<div class="ContentPasted0 ContentPasted1"><br>
</div>
<div class="ContentPasted0 ContentPasted1">Thanks,</div>
<div class="ContentPasted0 ContentPasted1">Cho<br>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b> Friday, July 14, 2023 5:57 PM<br>
<b>To:</b> Ng, Cho-Kuen <cho@slac.stanford.edu><br>
<b>Cc:</b> Barry Smith <bsmith@petsc.dev>; Mark Adams <mfadams@lbl.gov>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" data-loopstyle="linkonly">cho@slac.stanford.edu</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-8026140834471843556">
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I managed to pass the following options to PETSc using a GPU node on Perlmutter.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-mat_type aijcusparse -vec_type cuda -log_view -options_left</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Below is a summary of the test using 4 MPI tasks and 1 GPU per task.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
o #PETSc Option Table entries:
<div><span> </span>-log_view</div>
<div><span> </span>-mat_type aijcusparse</div>
<div> -options_left</div>
<div> -vec_type cuda</div>
<div> #End of PETSc Option Table entries</div>
<div> WARNING! There are options you set that were not used!</div>
<div> WARNING! could be spelling mistake, etc!</div>
<div> There is one unused database option. It is:</div>
<div> Option left: name:-mat_type value: aijcusparse</div>
<div><br>
</div>
<div>The -mat_type option has not been used. In the application code, we use</div>
<div><br>
</div>
<div> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
<div> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);</div>
</div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-8026140834471843556">
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
o The percent flops on the GPU for KSPSolve is 17%.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cho<br>
</div>
<div id="x_m_-8026140834471843556appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-8026140834471843556divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" data-loopstyle="linkonly">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Friday, June 30, 2023 7:57 AM<br>
<b>To:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" data-loopstyle="linkonly">bsmith@petsc.dev</a>>; Mark Adams <<a href="mailto:mfadams@lbl.gov" data-loopstyle="linkonly">mfadams@lbl.gov</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" data-loopstyle="linkonly">knepley@gmail.com</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Barry, Mark and Matt,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you all for the suggestions. I will modify the code so we can pass runtime options.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cho<br>
</div>
<div id="x_m_-8026140834471843556x_appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-8026140834471843556x_divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" data-loopstyle="linkonly">bsmith@petsc.dev</a>><br>
<b>Sent:</b> Friday, June 30, 2023 7:01 AM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" data-loopstyle="linkonly">mfadams@lbl.gov</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" data-loopstyle="linkonly">knepley@gmail.com</a>>; Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" data-loopstyle="linkonly">cho@slac.stanford.edu</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div><br>
</div>
Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because
Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
<div><br>
<blockquote type="cite">
<div>On Jun 30, 2023, at 9:30 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" data-loopstyle="linkonly">mfadams@lbl.gov</a>> wrote:</div>
<br>
<div>
<div dir="ltr">PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:<br>
<div><br>
</div>
<div>a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" data-loopstyle="linkonly">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
<div id="x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577appendonsend">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" face="Calibri, sans-serif"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" data-loopstyle="linkonly">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 8:32 PM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" data-loopstyle="linkonly">mfadams@lbl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read
in the options?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
<div id="x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_appendonsend">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" face="Calibri, sans-serif"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" data-loopstyle="linkonly">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 5:55 PM<br>
<b>To:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" data-loopstyle="linkonly">cho@slac.stanford.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div dir="ltr">Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
<div><br>
</div>
<div>The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.</div>
<div><br>
</div>
<div>The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" data-loopstyle="linkonly">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I installed PETSc on Perlmutter using "spack install
<span style="background: rgb(255, 255, 255);">petsc+cuda+zoltan</span><span style="background-color: rgb(255, 255, 255);">"
</span>and used it by "<span style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; background: rgb(255, 255, 255);">spack load petsc/fwge6pf</span>". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that
I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to
use the GPU backend?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Cho<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable" data-loopstyle="linkonly">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="x_gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable" data-loopstyle="linkonly">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>