<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Thank you Mark, I'll try the options you suggest to get more info. I'm also building PETSc and the code with the cray compiler suite to test. The test I'm running has 1 million unknowns. I was able to see good scaling up to 4 gpus on this case
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerinb2eoq" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerinb2eoq" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerinb2eoq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerinb2eoq" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerinb2eoq { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtoninb2eoq:hover, .pfptPrimaryButtoninb2eoq:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtoninb2eoq:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you Mark, I'll try the options you suggest to get more info. I'm also building PETSc and the code with the cray compiler suite to test.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The test I'm running has 1 million unknowns. I was able to see good scaling up to 4 gpus on this case in Polaris.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Talk soon,<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Marcos</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mark Adams <mfadams@lbl.gov><br>
<b>Sent:</b> Tuesday, March 5, 2024 2:41 PM<br>
<b>To:</b> Vanella, Marcos (Fed) <marcos.vanella@nist.gov><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Running CG with HYPRE AMG preconditioner in AMD GPUs</font>
<div> </div>
</div>
<div>
<div dir="ltr">You can run with -log_view_gpu_time to get rid of the nans and get more data.
<div><br>
</div>
<div>You can run with -ksp_view to get more info on the solver and send that output.</div>
<div><br>
</div>
<div>-options_left is also good to use so we can see what parameters you used.</div>
<div><br>
</div>
<div>The last 100 in this row:</div>
<div><br>
</div>
<div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">KSPSolve 1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04 3.1e+04 12 100 100 100 49 12 100 100 100 98 2503 -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<br class="x_gmail-Apple-interchange-newline">
</div>
<div>tells us that all the flops were logged on GPUs.</div>
<div><br>
</div>
<div>You do need at least 100K equations per GPU to see speedup, so don't worry about small problems.</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Mar 5, 2024 at 12:52 PM Vanella, Marcos (Fed) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg3917504900481874410">
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos and hip options: ./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0
</div>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="x_m_3917504900481874410pfptBannerbz5k6xw" style="display:block; text-align:left; margin:16px 0px; padding:8px 16px; border-radius:4px; min-width:200px; background-color:rgb(208,216,220); border-top:4px solid rgb(144,164,174)">
<div id="x_m_3917504900481874410pfptBannerbz5k6xw" style="float:left; display:block; margin:0px 0px 1px; max-width:600px">
<div id="x_m_3917504900481874410pfptBannerbz5k6xw" style="display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-weight:bold; font-size:14px; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_m_3917504900481874410pfptBannerbz5k6xw" style="font-weight:normal; display:block; background-color:rgb(208,216,220); color:rgb(0,0,0); font-family:Arial,sans-serif; font-size:12px; line-height:18px; margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px; clear:both; display:block; line-height:0; font-size:0.01px">
</div>
</div>
<u></u>
<div style="font-size:1px; color:rgb(255,255,255); line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden; display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos and hip options:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0 --with-cc=cc
--with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc --LIBS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}" --download-kokkos --download-kokkos-kernels --download-suitesparse --download-hypre --download-cmake</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">and have started testing our code solving a Poisson linear system with CG + HYPRE preconditioner. Timings look rather
high compared to compilations done on other machines that have NVIDIA cards. They are also not changing when using more than one GPU for the simple test I doing.</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Does anyone happen to know if HYPRE has an hip GPU implementation for Boomer AMG and is it compiled when configuring PETSc?</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Thanks!</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Marcos</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">PS: This is what I see on the log file (-log_view) when running the case with 2 GPUs in the node:</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">/ccs/home/vanellam/Firemodels_fork/fds/Build/mpich_gnu_frontier/fds_mpich_gnu_frontier on a arch-linux-frontier-opt-gcc named frontier04119 with 4 processors, by vanellam
Tue Mar 5 12:42:29 2024</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using Petsc Development GIT revision: v3.20.5-713-gabdf6bc0fcf GIT Date: 2024-03-05 01:04:54 +0000</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Max Max/Min Avg Total</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Time (sec): 8.368e+02 1.000 8.368e+02</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Objects: 0.000e+00 0.000 0.000e+00</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Flops: 2.546e+11 0.000 1.270e+11 5.079e+11</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Flops/sec: 3.043e+08 0.000 1.518e+08 6.070e+08</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MPI Msg Count: 1.950e+04 0.000 9.748e+03 3.899e+04</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MPI Msg Len (bytes): 1.560e+09 0.000 7.999e+04 3.119e+09</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MPI Reductions: 6.331e+04 2877.545</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> e.g., VecAXPY() for real vectors of length N --> 2N flops</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> and VecAXPY() for complex vectors of length N --> 8N flops</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Avg %Total Avg %Total Count %Total Avg %Total Count %Total</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> 0: Main Stage: 8.3676e+02 100.0% 5.0792e+11 100.0% 3.899e+04 100.0% 7.999e+04 100.0% 3.164e+04 50.0%</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Phase summary info:</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Count: number of times phase was executed</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Time and Flop: Max - maximum over all processors</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Ratio - ratio of maximum to minimum over all processors</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Mess: number of messages sent</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> AvgLen: average message length (bytes)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Reduct: number of global reductions</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Global: entire computation</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> %T - percent time in this phase %F - percent flop in this phase</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> %M - percent messages in this phase %L - percent message lengths in this phase</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> %R - percent reductions in this phase</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> CpuToGpu Count: total number of CPU to GPU copies per processor</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> GpuToCpu Count: total number of GPU to CPU copies per processor</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> GPU %F: percent flops on GPU in this event</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">--- Event Stage 0: Main Stage</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">BuildTwoSided 1201 0.0 nan nan 0.00e+00 0.0 2.0e+00 4.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">BuildTwoSidedF 1200 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MatMult 19494 0.0 nan nan 1.35e+11 0.0 3.9e+04 8.0e+04 0.0e+00 7 53 100 100 0 7 53 100 100 0 -nan -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MatConvert 3 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MatAssemblyBegin 2 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">MatAssemblyEnd 2 0.0 nan nan 0.00e+00 0.0 4.0e+00 2.0e+04 3.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecTDot 41382 0.0 nan nan 4.14e+10 0.0 0.0e+00 0.0e+00 2.1e+04 0 16 0 0 33 0 16 0 0 65 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecNorm 20691 0.0 nan nan 2.07e+10 0.0 0.0e+00 0.0e+00 1.0e+04 0 8 0 0 16 0 8 0 0 33 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecCopy 2394 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecSet 21888 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecAXPY 38988 0.0 nan nan 3.90e+10 0.0 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecAYPX 18297 0.0 nan nan 1.83e+10 0.0 0.0e+00 0.0e+00 0.0e+00 0 7 0 0 0 0 7 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecAssemblyBegin 1197 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+02 0 0 0 0 1 0 0 0 0 2 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecAssemblyEnd 1197 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecScatterBegin 19494 0.0 nan nan 0.00e+00 0.0 3.9e+04 8.0e+04 0.0e+00 0 0 100 100 0 0 0 100 100 0 -nan -nan 0 1.80e-05 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">VecScatterEnd 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">SFSetGraph 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">SFSetUp 1 0.0 nan nan 0.00e+00 0.0 4.0e+00 2.0e+04 5.0e-01 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">SFPack 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 1.80e-05 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">SFUnpack 19494 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">KSPSetUp 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">KSPSolve 1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04 3.1e+04 12 100 100 100 49 12 100 100 100 98 2503 -nan 0 1.80e-05 0 0.00e+00 100</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">PCSetUp 1 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">PCApply 20691 0.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Object Type Creations Destructions. Reports information only for process 0.</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">--- Event Stage 0: Main Stage</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Matrix 7 3</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Vector 7 1</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Index Set 2 2</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Star Forest Graph 1 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Krylov Solver 1 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"> Preconditioner 1 0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">========================================================================================================================</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Average time to get PetscTime(): 3.01e-08</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Average time for MPI_Barrier(): 3.8054e-06</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Average time for zero size MPI_Send(): 7.101e-06</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">#PETSc Option Table entries:</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-log_view # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-mat_type mpiaijkokkos # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-vec_type kokkos # (source: command line)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">#End of PETSc Option Table entries</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Compiled without FORTRAN kernels</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Compiled with full precision matrices (default)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Configure options: COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 FCOPTFLAGS=-O3 HIPOPTFLAGS=-O3 --with-debugging=0 --with-cc=cc --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc
--LIBS="-L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib -lmpi -L/opt/cray/pe/mpich/8.1.23/gtl/lib -lmpi_gtl_hsa" --download-kokkos --download-kokkos-kernels --download-suitesparse --download-hypre --download-cmake</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Libraries compiled on 2024-03-05 17:04:36 on login08</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Machine characteristics: Linux-5.14.21-150400.24.46_12.0.83-cray_shasta_c-x86_64-with-glibc2.3.4</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using PETSc directory: /autofs/nccs-svm1_home1/vanellam/Software/petsc</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using PETSc arch: arch-linux-frontier-opt-gcc</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using C compiler: cc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -O3</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using Fortran compiler: ftn -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using include paths: -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/include -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include/suitesparse
-I/opt/rocm-5.4.0/include</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-----------------------------------------</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using C linker: cc</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using Fortran linker: ftn</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">Using libraries: -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
-lpetsc -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib -Wl,-rpath,/opt/rocm-5.4.0/lib -L/opt/rocm-5.4.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib
-L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/gtl/lib -L/opt/cray/pe/mpich/8.1.23/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/<a href="https://urldefense.us/v3/__http://22.12.1.1/GNU/9.1/x86_64/lib__;!!G_uCfscf7eWS!YrJBJyEc5KGMbySL6LqDZiKMfHvLbhX0jp0hNVluceLhr3Iruk1PQI6Cwh_gd-q3khrOSQNE6i1O45T6gv_0mjN6AvBhNTSL$" originalsrc="http://22.12.1.1/GNU/9.1/x86_64/lib" shash="P7labfpcztGTFuefpilpSxIvNrUobMipr/69UxrzXMY6yZNV4FOy35qqdzi3pzCVofkPTZPgkaIvsZHdUqNkYRKXEwHtXQ6gFs5FB1/wTPfbdGvRCJhl2XV0eoi8uSYhm4mE+pLu79SPt0lzv1KQ5kStV4ayzFlEBIFISIMgkzA=" target="_blank">22.12.1.1/GNU/9.1/x86_64/lib</a>
-L/opt/cray/pe/libsci/<a href="https://urldefense.us/v3/__http://22.12.1.1/GNU/9.1/x86_64/lib__;!!G_uCfscf7eWS!YrJBJyEc5KGMbySL6LqDZiKMfHvLbhX0jp0hNVluceLhr3Iruk1PQI6Cwh_gd-q3khrOSQNE6i1O45T6gv_0mjN6AvBhNTSL$" originalsrc="http://22.12.1.1/GNU/9.1/x86_64/lib" shash="gu5OqD1qb50pa+G5GFS7eePKe4nqwORZQR/Apvoc85JOJgor8AnCHj/p5ZH1gCOOyaPz6OtSgRH5+XyUo/e784rdv6LLRE738eb1E4bGW9DGPWOVeCGMxBBqXHdPT//EUOttXO7+20EMGqSoMNjdajlv+w/7OQb5wswlxcpxfPM=" target="_blank">22.12.1.1/GNU/9.1/x86_64/lib</a>
-Wl,-rpath,/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib -L/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib
-L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.1.8/lib -L/opt/cray/pe/pmi/6.1.8/lib -Wl,-rpath,/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64 -L/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0
-L/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0 -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib64 -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -Wl,-rpath,/opt/rocm-5.4.0/llvm/lib -L/opt/rocm-5.4.0/llvm/lib -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib -L/opt/cray/pe/gcc/12.2.0/snos/lib
-lHYPRE -lspqr -lumfpack -lklu -lcholmod -lamd -lkokkoskernels -lkokkoscontainers -lkokkoscore -lkokkossimd -lhipsparse -lhipblas -lhipsolver -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lmpi -lmpi_gtl_hsa -ldarshan -lz -ldl -lxpmem -lgfortran -lm
-lmpifort_gnu_91 -lmpi_gnu_91 -lsci_gnu_82_mpi -lsci_gnu_82 -ldsmml -lpmi -lpmi2 -lgfortran -lquadmath -lpthread -lm -lgcc_s -lstdc++ -lquadmath -lmpi -lmpi_gtl_hsa</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">-----------------------------------------</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>