<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main.
I've put them in a single google drive folder linked here:</span></div>
<div class="elementToProof"><span style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);"><a href="https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link" id="LPNoLPOWALinkPreview" class="OWAAutoLink" contenteditable="true" style="pointer-events: auto;">https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link</a></span></div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Have a happy holiday weekend!</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,<br>
</div>
<div class="elementToProof" style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size: 11pt;"><b>Philip Fackler<br>
</b></span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Advanced Computing Systems Research Section</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Oak Ridge National Laboratory</b></span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Monday, October 16, 2023 15:24<br>
<b>To:</b> Fackler, Philip <facklerpw@ornl.gov><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov>; xolotl-psi-development@lists.sourceforge.net <xolotl-psi-development@lists.sourceforge.net><br>
<b>Subject:</b> Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi, Philip,
<div> That branch was merged to petsc/main today. Let me know once you have new profiling results.</div>
<div><br>
</div>
<div> Thanks.<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <<a href="mailto:facklerpw@ornl.gov">facklerpw@ornl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg3053484003997947275">
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
Junchao,</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I
assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon.</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
Thanks,<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_3053484003997947275Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="x_m_3053484003997947275appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_3053484003997947275divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Fackler, Philip via Xolotl-psi-development <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>><br>
<b>Sent:</b> Wednesday, October 11, 2023 11:31<br>
<b>To:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>;
<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>><br>
<b>Subject:</b> Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
I'm on it.<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_3053484003997947275x_Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="x_m_3053484003997947275x_appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_3053484003997947275x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Wednesday, October 11, 2023 10:14<br>
<b>To:</b> Fackler, Philip <<a href="mailto:facklerpw@ornl.gov" target="_blank">facklerpw@ornl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>;
<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>>;
Blondel, Sophie <<a href="mailto:sblondel@utk.edu" target="_blank">sblondel@utk.edu</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi, Philip,
<div> Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ?</div>
<div><br>
</div>
<div> Thanks.<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <<a href="mailto:facklerpw@ornl.gov" target="_blank">facklerpw@ornl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
Aha! That makes sense. Thank you.<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_3053484003997947275x_x_m_972965047420409840Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="x_m_3053484003997947275x_x_m_972965047420409840appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_3053484003997947275x_x_m_972965047420409840divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Thursday, October 5, 2023 17:29<br>
<b>To:</b> Fackler, Philip <<a href="mailto:facklerpw@ornl.gov" target="_blank">facklerpw@ornl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>;
<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>>;
Blondel, Sophie <<a href="mailto:sblondel@utk.edu" target="_blank">sblondel@utk.edu</a>><br>
<b>Subject:</b> [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Wait a moment, it seems it was because we do not have a GPU implementation of MatShift...
<div>Let me see how to add it.
<div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>Hi, Philip,</div>
<div> I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive.</div>
<div> Do you have profiling results with COO enabled?</div>
<div><br>
</div>
<img alt="Screenshot 2023-10-05 at 10.55.29 AM.png" style="margin-right:0px"><br>
<div> </div>
<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div>
<div dir="ltr">On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi, Philip,
<div> I will look into the tarballs and get back to you.</div>
<div> Thanks.<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div><span style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version
(using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration).</span></div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the
blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible.<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for
the GPUs is much smaller. We've still got more optimization to do on this.<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance
here. Any help looking into this would be appreciated.<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
The tarballs linked <a href="https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=" title="https://drive.google.com/file/d/19X_L3SVkGBM9YUzXnRR_kVWFG0JFwqZ3/view?usp=drive_link" id="x_m_3053484003997947275x_x_m_972965047420409840x_m_4560154858951847936m_2345933986412643521m_3364327590849106712LPNoLPOWALinkPreview" target="_blank">
here</a> and <a href="https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=" title="https://drive.google.com/file/d/15yDBN7-YlO1g6RJNPYNImzr611i1Ffhv/view?usp=drive_link" id="x_m_3053484003997947275x_x_m_972965047420409840x_m_4560154858951847936m_2345933986412643521m_3364327590849106712LPNoLPOWALinkPreview_1" target="_blank">
here</a> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction.</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<span style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">Thanks for your help,</span>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_3053484003997947275x_x_m_972965047420409840x_m_4560154858951847936m_2345933986412643521m_3364327590849106712Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>