<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerb3kad56 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonb3kad56:hover, .pfptPrimaryButtonb3kad56:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonb3kad56:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
[keep on list] I have little experience with running hypre on GPUs but others might have more. 1M dogs/node is not a lot and NVIDIA has larger L1 cache and more mature compilers, etc. so it is not surprising that NVIDIA is faster. I suspect
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerb3kad56" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerb3kad56" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerb3kad56" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerb3kad56" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">[keep on list]<div><br></div><div>I have little experience with running hypre on GPUs but others might have more.</div><div><br></div><div>1M dogs/node is not a lot and NVIDIA has larger L1 cache and more mature compilers, etc. so it is not surprising that NVIDIA is faster.</div><div>I suspect the gap would narrow with a larger problem.</div><div><br></div><div>Also, why are you using Kokkos? It should not make a difference but you could check easily. Just use -vec_type hip with your current code.</div><div><br></div><div>You could also test with GAMG, -pc_type gamg</div><div><br></div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 19, 2024 at 4:12 PM Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov">marcos.vanella@nist.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg141370437975451340">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Mark, I run a canonical test we have to time our code. It is a propane fire on a burner within a box with around 1 million cells.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I split the problem in 4 GPUS, single node, both in Polaris and Frontier. I compiled PETSc with gnu and HYPRE being downloaded and the following configure options:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<ul style="margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Polaris:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
$./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" CUDAOPTFLAGS="-O3" --with-debugging=0 --download-suitesparse --download-hypre --with-cuda --with-cc=cc --with-cxx=CC --with-fc=ftn --with-cudac=nvcc --with-cuda-arch=80 --download-cmake</div>
</li></ul>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<ul style="margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Frontier:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
$./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0 --with-cc=cc --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc --LIBS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}"
--download-kokkos --download-kokkos-kernels --download-suitesparse --download-hypre --download-cmake</div>
</li></ul>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Our code was compiled also with gnu compilers and -O3 flag. I used latest (from this week) PETSc repo update. These are the timings for the test case:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<ul style="margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">8 meshes + 1Million cells case, 8 MPI processes, 4 GPUS, 2 MPI Procs per GPU, 1 sec run time (~580 time steps, ~1160
Poisson solves):</span></li></ul>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
System Poisson Solver GPU Implementation Poisson Wall time (sec) Total Wall time (sec)</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Polaris CG + HYPRE PC CUDA 80 287</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Frontier CG + HYPRE PC Kokkos + HIP 158 401</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It is interesting to see that the Poisson solves take twice the time in Frontier than in Polaris. </div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Do you have experience on running HYPRE AMG on these machines? Is this difference between the CUDA implementation and Kokkos-kernels to be expected?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I can run the case in both computers with the log flags you suggest. Might give more information on where the differences are.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you for your time,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Marcos</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_141370437975451340appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_141370437975451340divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Tuesday, March 5, 2024 2:41 PM<br>
<b>To:</b> Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov" target="_blank">marcos.vanella@nist.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Running CG with HYPRE AMG preconditioner in AMD GPUs</font>
<div> </div>
</div>
<div>
<div dir="ltr">You can run with -log_view_gpu_time to get rid of the nans and get more data.
<div><br>
</div>
<div>You can run with -ksp_view to get more info on the solver and send that output.</div>
<div><br>
</div>
<div>-options_left is also good to use so we can see what parameters you used.</div>
<div><br>
</div>
<div>The last 100 in this row:</div>
<div><br>
</div>
<div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">KSPSolve 1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04 3.1e+04 12 100 100 100 49 12 100 100 100 98 2503 -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<br>
</div>
<div>tells us that all the flops were logged on GPUs.</div>
<div><br>
</div>
<div>You do need at least 100K equations per GPU to see speedup, so don't worry about small problems.</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Mar 5, 2024 at 12:52 PM Vanella, Marcos (Fed) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos and hip options: ./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0
</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_141370437975451340x_m_3917504900481874410pfptBannerbz5k6xw" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_141370437975451340x_m_3917504900481874410pfptBannerbz5k6xw" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_141370437975451340x_m_3917504900481874410pfptBannerbz5k6xw" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender </div>
<div id="m_141370437975451340x_m_3917504900481874410pfptBannerbz5k6xw" style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px">
</div>
</div>
<u></u>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos and hip options:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0 --with-cc=cc
--with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc --LIBS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}" --download-kokkos --download-kokkos-kernels --download-suitesparse --download-hypre --download-cmake</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">and have started testing our code solving a Poisson linear system with CG + HYPRE preconditioner. Timings look rather
high compared to compilations done on other machines that have NVIDIA cards. They are also not changing when using more than one GPU for the simple test I doing.</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Does anyone happen to know if HYPRE has an hip GPU implementation for Boomer AMG and is it compiled when configuring PETSc?</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Thanks!</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Marcos</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">PS: This is what I see on the log file (-log_view) when running the case with 2 GPUs in the node:</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">/ccs/home/vanellam/Firemodels_fork/fds/Build/mpich_gnu_frontier/fds_mpich_gnu_frontier on a arch-linux-frontier-opt-gcc named frontier04119 with 4 processors, by vanellam
Tue Mar 5 12:42:29 2024</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using Petsc Development GIT revision: v3.20.5-713-gabdf6bc0fcf GIT Date: 2024-03-05 01:04:54 +0000</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Max Max/Min Avg Total</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Time (sec): 8.368e+02 1.000 8.368e+02</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Objects: 0.000e+00 0.000 0.000e+00</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Flops: 2.546e+11 0.000 1.270e+11 5.079e+11</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Flops/sec: 3.043e+08 0.000 1.518e+08 6.070e+08</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MPI Msg Count: 1.950e+04 0.000 9.748e+03 3.899e+04</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MPI Msg Len (bytes): 1.560e+09 0.000 7.999e+04 3.119e+09</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MPI Reductions: 6.331e+04 2877.545</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> e.g., VecAXPY() for real vectors of length N --> 2N flops</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> and VecAXPY() for complex vectors of length N --> 8N flops</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Avg %Total Avg %Total Count %Total Avg %Total Count %Total</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> 0: Main Stage: 8.3676e+02 100.0% 5.0792e+11 100.0% 3.899e+04 100.0% 7.999e+04 100.0% 3.164e+04 50.0%</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Phase summary info:</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Count: number of times phase was executed</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Time and Flop: Max - maximum over all processors</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Ratio - ratio of maximum to minimum over all processors</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Mess: number of messages sent</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> AvgLen: average message length (bytes)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Reduct: number of global reductions</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Global: entire computation</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> %T - percent time in this phase %F - percent flop in this phase</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> %M - percent messages in this phase %L - percent message lengths in this phase</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> %R - percent reductions in this phase</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> CpuToGpu Count: total number of CPU to GPU copies per processor</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> GpuToCpu Count: total number of GPU to CPU copies per processor</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> GPU %F: percent flops on GPU in this event</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">--- Event Stage 0: Main Stage</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">BuildTwoSided 1201 0.0 nan nan 0.00e+00 0.0 2.0e+00 4.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">BuildTwoSidedF 1200 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MatMult 19494 0.0 nan nan 1.35e+11 0.0 3.9e+04 8.0e+04 0.0e+00 7 53 100 100 0 7 53 100 100 0 -nan -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MatConvert 3 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MatAssemblyBegin 2 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">MatAssemblyEnd 2 0.0 nan nan 0.00e+00 0.0 4.0e+00 2.0e+04 3.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecTDot 41382 0.0 nan nan 4.14e+10 0.0 0.0e+00 0.0e+00 2.1e+04 0 16 0 0 33 0 16 0 0 65 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecNorm 20691 0.0 nan nan 2.07e+10 0.0 0.0e+00 0.0e+00 1.0e+04 0 8 0 0 16 0 8 0 0 33 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecCopy 2394 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecSet 21888 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecAXPY 38988 0.0 nan nan 3.90e+10 0.0 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecAYPX 18297 0.0 nan nan 1.83e+10 0.0 0.0e+00 0.0e+00 0.0e+00 0 7 0 0 0 0 7 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecAssemblyBegin 1197 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecAssemblyEnd 1197 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecScatterBegin 19494 0.0 nan nan 0.00e+00 0.0 3.9e+04 8.0e+04 0.0e+00 0 0 100 100 0 0 0 100 100 0 -nan -nan 0 1.80e-05 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">VecScatterEnd 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">SFSetGraph 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">SFSetUp 1 0.0 nan nan 0.00e+00 0.0 4.0e+00 2.0e+04 5.0e-01 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">SFPack 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 1.80e-05 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">SFUnpack 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">KSPSetUp 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">KSPSolve 1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04 3.1e+04 12 100 100 100 49 12 100 100 100 98 2503 -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">PCSetUp 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">PCApply 20691 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Object Type Creations Destructions. Reports information only for process 0.</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">--- Event Stage 0: Main Stage</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Matrix 7 3</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Vector 7 1</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Index Set 2 2</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Star Forest Graph 1 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Krylov Solver 1 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"> Preconditioner 1 0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">========================================================================================================================</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Average time to get PetscTime(): 3.01e-08</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Average time for MPI_Barrier(): 3.8054e-06</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Average time for zero size MPI_Send(): 7.101e-06</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">#PETSc Option Table entries:</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-log_view # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-mat_type mpiaijkokkos # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-vec_type kokkos # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">#End of PETSc Option Table entries</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Compiled without FORTRAN kernels</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Compiled with full precision matrices (default)</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Configure options: COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 FCOPTFLAGS=-O3 HIPOPTFLAGS=-O3 --with-debugging=0 --with-cc=cc --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc
--LIBS="-L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib -lmpi -L/opt/cray/pe/mpich/8.1.23/gtl/lib -lmpi_gtl_hsa" --download-kokkos --download-kokkos-kernels --download-suitesparse --download-hypre --download-cmake</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Libraries compiled on 2024-03-05 17:04:36 on login08</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Machine characteristics: Linux-5.14.21-150400.24.46_12.0.83-cray_shasta_c-x86_64-with-glibc2.3.4</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using PETSc directory: /autofs/nccs-svm1_home1/vanellam/Software/petsc</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using PETSc arch: arch-linux-frontier-opt-gcc</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using C compiler: cc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -O3</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using Fortran compiler: ftn -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using include paths: -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/include -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include/suitesparse
-I/opt/rocm-5.4.0/include</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using C linker: cc</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using Fortran linker: ftn</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">Using libraries: -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
-lpetsc -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -Wl,-rpath,/opt/rocm-5.4.0/lib -L/opt/rocm-5.4.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib
-L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/gtl/lib -L/opt/cray/pe/mpich/8.1.23/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/<a href="https://urldefense.us/v3/__http://22.12.1.1/GNU/9.1/x86_64/lib__;!!G_uCfscf7eWS!dXzM_BfDdOUC8vEtn0ErlzrpWbsUp87b1eF-G0jiLOYsWSjCOrlSIxcyCldStycCNU7UJsV2jZKKe8p8KXNno6s$" target="_blank">22.12.1.1/GNU/9.1/x86_64/lib</a>
-L/opt/cray/pe/libsci/<a href="https://urldefense.us/v3/__http://22.12.1.1/GNU/9.1/x86_64/lib__;!!G_uCfscf7eWS!dXzM_BfDdOUC8vEtn0ErlzrpWbsUp87b1eF-G0jiLOYsWSjCOrlSIxcyCldStycCNU7UJsV2jZKKe8p8KXNno6s$" target="_blank">22.12.1.1/GNU/9.1/x86_64/lib</a>
-Wl,-rpath,/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib -L/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib
-L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.1.8/lib -L/opt/cray/pe/pmi/6.1.8/lib -Wl,-rpath,/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64 -L/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0
-L/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib64 -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -Wl,-rpath,/opt/rocm-5.4.0/llvm/lib -L/opt/rocm-5.4.0/llvm/lib -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib -L/opt/cray/pe/gcc/12.2.0/snos/lib
-lHYPRE -lspqr -lumfpack -lklu -lcholmod -lamd -lkokkoskernels -lkokkoscontainers -lkokkoscore -lkokkossimd -lhipsparse -lhipblas -lhipsolver -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lmpi -lmpi_gtl_hsa -ldarshan -lz -ldl -lxpmem -lgfortran -lm
-lmpifort_gnu_91 -lmpi_gnu_91 -lsci_gnu_82_mpi -lsci_gnu_82 -ldsmml -lpmi -lpmi2 -lgfortran -lquadmath -lpthread -lm -lgcc_s -lstdc++ -lquadmath -lmpi -lmpi_gtl_hsa</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">-----------------------------------------</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div></blockquote></div>