<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Junchao,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Unfortunately I don't have access to other cuda machines with multiple GPUs.</span></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm pretty stuck, and I think running on a different machine would help isolate the issue.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm sharing the python script and the launch script that Victor wrote.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There is a comment in the launch script with the mpi command I was using to run the python script.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I configured hypre without unified memory. In case it's useful, I also attached the configure.log.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If the issue is with petsc/hypre, it may be in the environment variables described here (e.g. HYPRE_MEMORY_DEVICE):</div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><a href="https://github.com/hypre-space/hypre/wiki/GPUs" id="LPlnk">https://github.com/hypre-space/hypre/wiki/GPUs</a></span></div>
<div class="elementToProof"><br>
</div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thank you for helping me troubleshoot this issue!</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Best,</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Anna<br>
</span></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Thursday, February 1, 2024 9:07 PM<br>
<b>To:</b> Yesypenko, Anna <anna@oden.utexas.edu><br>
<b>Cc:</b> Victor Eijkhout <eijkhout@tacc.utexas.edu>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] errors with hypre with MPI and multiple GPUs on a node</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hi, Anna,</div>
<div> Do you have other CUDA machines to try? If you can share your test, then I will run on Polaris@Argonne to see if it is a petsc/hypre issue. If not, then it must be a GPU-MPI binding problem on TACC. </div>
<div><br>
</div>
Thanks<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Feb 1, 2024 at 5:31 PM Yesypenko, Anna <<a href="mailto:anna@oden.utexas.edu" target="_blank">anna@oden.utexas.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Victor, Junchao,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank you for providing the script, it is very useful! </div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
There are still issues with hypre not binding correctly, and I'm getting the error message occasionally (but much less often).</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I added some additional environment variables to the script that seem to make the behavior more consistent.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">export CUDA_DEVICE_ORDER=PCI_BUS_ID</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">export CUDA_VISIBLE_DEVICES=$MV2_COMM_WORLD_LOCAL_RANK ## as Victor suggested</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">export HYPRE_MEMORY_DEVICE=$MV2_COMM_WORLD_LOCAL_RANK</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">The last environment variable is from hypre's documentation on GPUs.</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">In 30 runs for a small problem size, 4 fail with a hypre-related error. Do you have any other thoughts or suggestions?</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Best,</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Anna</span></div>
<div><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div id="x_m_-278907308663054568m_5798670685468534043appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div dir="ltr" id="x_m_-278907308663054568m_5798670685468534043divRplyFwdMsg"><span style="font-family:Calibri,sans-serif; font-size:11pt; color:rgb(0,0,0)"><b>From:</b> Victor Eijkhout <<a href="mailto:eijkhout@tacc.utexas.edu" target="_blank">eijkhout@tacc.utexas.edu</a>><br>
<b>Sent:</b> Thursday, February 1, 2024 11:26 AM<br>
<b>To:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>>; Yesypenko, Anna <<a href="mailto:anna@oden.utexas.edu" target="_blank">anna@oden.utexas.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] errors with hypre with MPI and multiple GPUs on a node</span>
<div> </div>
</div>
<p style="margin:0in; font-family:Calibri,sans-serif; font-size:11pt"><span style="color:rgb(0,0,0)">Only for mvapich2-gdr:</span></p>
<p style="margin:0in; font-family:Calibri,sans-serif; font-size:11pt"><span style="color:rgb(0,0,0)"> </span></p>
<div id="x_m_-278907308663054568m_5798670685468534043x_mail-editor-reference-message-container">
<div id="x_m_-278907308663054568m_5798670685468534043x_mail-editor-reference-message-container">
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)">#!/bin/bash</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"># Usage: mpirun -n <num_proc> MV2_USE_AFFINITY=0 MV2_ENABLE_AFFINITY=0 ./launch ./bin</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"> </span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)">export CUDA_VISIBLE_DEVICES=$MV2_COMM_WORLD_LOCAL_RANK</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)">case $MV2_COMM_WORLD_LOCAL_RANK in</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"> [0]) cpus=0-3 ;;</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"> [1]) cpus=64-67 ;;</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"> [2]) cpus=72-75 ;;</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)">esac</span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)"> </span></p>
<p style="margin:0in 0in 3pt; font-family:Calibri,sans-serif; font-size:11pt"><span style="font-family:Monaco; font-size:9pt; color:rgb(29,28,29)">numactl --physcpubind=$cpus $@</span></p>
<p style="margin:0in; font-family:Calibri,sans-serif; font-size:11pt"> </p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>