<div dir="ltr"><div dir="ltr">On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu">cho@slac.stanford.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-8026140834471843556">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I managed to pass the following options to PETSc using a GPU node on Perlmutter.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-mat_type aijcusparse -vec_type cuda -log_view -options_left</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Below is a summary of the test using 4 MPI tasks and 1 GPU per task.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
o #PETSc Option Table entries:
<div><span> </span>-log_view</div>
<div><span> </span>-mat_type aijcusparse</div>
<div> -options_left</div>
<div> -vec_type cuda</div>
<div> #End of PETSc Option Table entries</div>
<div> WARNING! There are options you set that were not used!</div>
<div> WARNING! could be spelling mistake, etc!</div>
<div> There is one unused database option. It is:</div>
<div> Option left: name:-mat_type value: aijcusparse</div>
<div><br>
</div>
<div>The -mat_type option has not been used. In the application code, we use</div>
<div><br>
</div>
<div> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
<div> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);</div>
</div></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br></div></div></div></blockquote><div><br></div><div>If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-8026140834471843556"><div dir="ltr"><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
o The percent flops on the GPU for KSPSolve is 17%.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Cho<br>
</div>
<div id="m_-8026140834471843556appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-8026140834471843556divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Friday, June 30, 2023 7:57 AM<br>
<b>To:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>; Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Barry, Mark and Matt,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you all for the suggestions. I will modify the code so we can pass runtime options.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Cho<br>
</div>
<div id="m_-8026140834471843556x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-8026140834471843556x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br>
<b>Sent:</b> Friday, June 30, 2023 7:01 AM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div><br>
</div>
Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because
Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
<div><br>
<blockquote type="cite">
<div>On Jun 30, 2023, at 9:30 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div>
<br>
<div>
<div dir="ltr">PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:<br>
<div><br>
</div>
<div>a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Cho<br>
</div>
<div id="m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 8:32 PM<br>
<b>To:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Mark,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read
in the options?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Cho<br>
</div>
<div id="m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Thursday, June 29, 2023 5:55 PM<br>
<b>To:</b> Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Using PETSc GPU backend</font>
<div> </div>
</div>
<div>
<div dir="ltr">Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
<div><br>
</div>
<div>The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.</div>
<div><br>
</div>
<div>The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">I installed PETSc on Perlmutter using "spack install
<span style="background:rgb(255,255,255)">petsc+cuda+zoltan</span><span style="background-color:rgb(255,255,255)">"
</span>and used it by "<span style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;background:rgb(255,255,255)">spack load petsc/fwge6pf</span>". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can
get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use
the GPU backend?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt">Cho<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>