<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerqun10tp { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonqun10tp:hover, .pfptPrimaryButtonqun10tp:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonqun10tp:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi, Sophie, I tried various modules and compilers on Summit and failed to find one that works with gpu aware mpi. The one that could build petsc and kokkos was "module load cuda/11. 7. 1 gcc/9. 3. 0-compiler_only spectrum-mpi essl netlib-lapack".
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerqun10tp" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerqun10tp" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerqun10tp" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerqun10tp" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi, Sophie, <div> I tried various modules and compilers on Summit and failed to find one that works with gpu aware mpi.<br></div><div> The one that could build petsc and kokkos was "module load cuda/11.7.1 gcc/9.3.0-compiler_only spectrum-mpi essl netlib-lapack". But it only worked with "-use_gpu_aware_mpi 0". Without it, I saw code crashes.<br> From what I can see, the gpu-aware mpi on Summit is an unusable and unmaintained state. <div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 1, 2024 at 3:58 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It is weird, with<div> <span style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:16px">jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-</span><span style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:16px">shared/dependencies/petsc-</span><span style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:16px">kokkos/src/ksp/ksp/tutorials/</span><span style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:16px">bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 1</span></div><div><font color="#000000" face="Calibri, Helvetica, sans-serif"><span style="font-size:16px"><br></span></font></div><div><font color="#000000" face="Calibri, Helvetica, sans-serif"><span style="font-size:16px">petsc tried to test if the MPI is gpu aware (by doing an MPI_Allreduce on device buffers). It tried and found it was not, so it threw out the complaint in the error message.</span></font></div><div><font color="#000000" face="Calibri, Helvetica, sans-serif"><span style="font-size:16px"><br></span></font></div><div><font color="#000000" face="Calibri, Helvetica, sans-serif"><span style="font-size:16px">From </span></font><a href="https://urldefense.us/v3/__https://docs.olcf.ornl.gov/systems/summit_user_guide.html*cuda-aware-mpi__;Iw!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstbOGRUZc$" target="_blank">https://docs.olcf.ornl.gov/systems/summit_user_guide.html#cuda-aware-mpi</a>, I think your flags were right.</div><div><font color="#000000" face="Calibri, Helvetica, sans-serif"><span style="font-size:16px"><br></span></font></div><div>I just got my Summit account reactivated today. I will give it a try. </div><div><br></div><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 1, 2024 at 3:32 PM Blondel, Sophie <<a href="mailto:sblondel@utk.edu" target="_blank">sblondel@utk.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I have been using --smpiargs "-gpu".</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I tried the benchmark with "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve
-mat_type aijkokkos -use_gpu_aware_mpi 0" and it seems to work:</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Fri Mar 1 16:27:14 EST 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">===========================================</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Test: KSP performance - Poisson</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Input matrix: 27-pt finite difference stencil</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> -n 100</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> DoFs = 1000000</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Number of nonzeros = 26463592</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Step1 - creating Vecs and Mat...</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Step2 - running KSPSolve()...</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Step3 - calculating error norm...</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Error norm: 5.591e-02</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">KSP iters: 63 </span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">KSPSolve: 3.16646 seconds</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">FOM: 3.158e+05 DoFs/sec</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">===========================================</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">------------------------------------------------------------</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Sender: LSF System <lsfadmin@batch3></span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Subject: Job 3322694: <xolotlTest> in cluster <summit> Done</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Job <xolotlTest> was submitted from host <login2> by user <bqo> in cluster <summit> at Fri Mar 1 16:26:58 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Job was executed on host(s) <1*batch3>, in queue <debug>, as user <bqo> in cluster <summit> at Fri Mar 1 16:27:00 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <42*a35n05></span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></ccs/home/bqo> was used as the home directory.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></gpfs/alpine2/mat267/scratch/bqo/test> was used as the working directory.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Started at Fri Mar 1 16:27:00 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Terminated at Fri Mar 1 16:27:26 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Results reported at Fri Mar 1 16:27:26 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">The output (if any) is above this job summary.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">If I switch to "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve
-mat_type aijkokkos -use_gpu_aware_mpi 1" it complains:</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Fri Mar 1 16:25:02 EST 2024</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">===========================================</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Test: KSP performance - Poisson</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Input matrix: 27-pt finite difference stencil</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> -n 100</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> DoFs = 1000000</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Number of nonzeros = 26463592</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Step1 - creating Vecs and Mat...</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not GPU-aware. For better performance, please use a GPU-aware MPI.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To not see the message again, add the option to your .petscrc, OR add it to the env var
PETSC_OPTIONS.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you may need jsrun --smpiargs=-gpu.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: For Open MPI, you need to configure it --with-cuda (<a href="https://urldefense.us/v3/__https://www.open-mpi.org/faq/?category=buildcuda__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstegRM8hj$" target="_blank">https://www.open-mpi.org/faq/?category=buildcuda</a>)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 (<a href="https://urldefense.us/v3/__http://mvapich.cse.ohio-state.edu/userguide/gdr/__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstf6tc3Lv$" target="_blank">http://mvapich.cse.ohio-state.edu/userguide/gdr/</a>)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[5]PETSC ERROR: For Cray-MPICH, you need to set MPICH_GPU_SUPPORT_ENABLED=1 (man mpi to see manual of cray-mpich)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">--------------------------------------------------------------------------</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">with errorcode 76.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Best,</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Sophie</span></div>
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Thursday, February 29, 2024 17:09<br>
<b>To:</b> Blondel, Sophie <<a href="mailto:sblondel@utk.edu" target="_blank">sblondel@utk.edu</a>><br>
<b>Cc:</b> <a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] PAMI error on Summit</font>
<div> </div>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0px;display:table;width:100%;table-layout:fixed;float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding:7px 2px;background-color:rgb(166,166,166)">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width:100%;background-color:rgb(234,234,234);padding:7px 5px 7px 15px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
<div>You don't often get email from <a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>. <a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstYnzecTK$" target="_blank">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width:75px;background-color:rgb(234,234,234);padding:7px 5px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
</td>
</tr>
</tbody>
</table>
<div>
<div dir="ltr">Could you try a petsc example to see if the environment is good? For example, <br>
<div><br>
</div>
<div>cd src/ksp/ksp/tutorials<br>
make bench_kspsolve<br>
mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 or 1}
<div><br>
</div>
<div>BTW, I remember to use gpu-aware mpi on Summit, one needs to pass --smpiargs "-gpu" to jsrun</div>
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly
</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192pfptBannerozt91gd" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192pfptBannerozt91gd" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192pfptBannerozt91gd" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender </div>
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192pfptBannerozt91gd" style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization. </div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px">
</div>
</div>
<u></u>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I still get the same error when deactivating GPU-aware MPI.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly turning it off):</div>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">0 TS dt 1e-12 time 0.</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] ------------------------------------------------------------------------</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] Try option -start_in_debugger or -on_error_attach_debugger</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] or see
<a href="https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$" id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192OWA55be8502-8d13-b445-887a-e3475b7e8ed4" rel="noopener noreferrer" style="margin-top:0px;margin-bottom:0px" target="_blank">
https://petsc.org/release/faq/#valgrind</a> and <a href="https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$" id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192OWA32fe1662-a183-8125-494e-edc4729a491b" rel="noopener noreferrer" style="margin-top:0px;margin-bottom:0px" target="_blank">
https://petsc.org/release/faq/</a></span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] or try
<a href="https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$" id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192OWAebfba962-cde8-2a72-2805-9166031fa676" rel="noopener noreferrer" style="margin-top:0px;margin-bottom:0px" target="_blank">
https://docs.nvidia.com/cuda/cuda-memcheck/index.html</a> on NVIDIA CUDA systems to find memory corruption errors</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] configure using --with-debugging=yes, recompile, link, and run </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] to get more information on the crash.</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] [0]PETSC ERROR: </span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">[ERROR] Run with -malloc_debug to check if memory corruption is causing the crash.</span></p>
<p style="margin:0px"><span style="font-family:Calibri,sans-serif;font-size:12pt;color:black">--------------------------------------------------------------------------</span></p>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Best,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sophie</div>
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_30272906136386278m_-8989069677642689449m_7489002335580220352x_m_-5695560179332709192divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Blondel, Sophie via Xolotl-psi-development <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>><br>
<b>Sent:</b> Thursday, February 29, 2024 10:17<br>
<b>To:</b> <a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">
xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> [Xolotl-psi-development] PAMI error on Summit</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error:</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">0 TS dt 1e-12 time 0.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">errno 14 pid 864558</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t
PAMI::Dev</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">errno 14 pid 864557</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t
PAMI::Dev</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] *** Process received signal ***</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] Signal: Aborted (6)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] Signal code: (-6)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(gsignal+0xd8)[0x200005d796f8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(abort+0x164)[0x200005d53ff4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(+0x3d280)[0x200005d6d280]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 4] [e28n07:864558] *** Process received signal ***</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] Signal: Aborted (6)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] Signal code: (-6)</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(gsignal+0xd8)[0x200005d796f8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(abort+0x164)[0x200005d53ff4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(+0x3d280)[0x200005d6d280]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(__assert_fail+0x64)[0x200005d6d324]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(PAMI_Rget+0x18)[0x20007f2d94a8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(process_rndv_msg+0x46c)[0x2000a80159ac]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [11] /lib64/glibc-hwcaps/power9/<a href="https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!cq445CXteimKBMZKF1HQqgEFTwREIrbMMm5Cn-sCV3wDm2A3tixBsge_FLfW-3YKRxtbYWK9D29cMq338kMstSjHODO3$" target="_blank">libc-2.28.so</a>(__assert_fail+0x64)[0x200005d6d324]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(PAMI_Rget+0x18)[0x20007f2d94a8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(process_rndv_msg+0x46c)[0x2000a80159ac]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(PAMI_Context_advancev+0x6b0)[0x20007f2da540]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">progress+0x6c)[0x20003d60640c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">equest_default_wait_all+0x144)[0x2000034c4b04]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">aitall+0x10c)[0x20000352790c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">(PAMI_Context_advancev+0x6b0)[0x20007f2da540]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">progress+0x6c)[0x20003d60640c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">equest_default_wait_all+0x144)[0x2000034c4b04]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">aitall+0x10c)[0x20000352790c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">_+0xc4)[0x200005f710d4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">00005f7130c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">_+0xc4)[0x200005f710d4]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">00005f7130c]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864557] *** End of error message ***</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">/gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088]</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[e28n07:864558] *** End of error message ***</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">It seems to be pointing to
<a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$" target="_blank">
https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/</a> so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information.</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Best,</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Sophie<br>
</span></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div></blockquote></div>
</blockquote></div>