<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Victor, Junchao,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for providing the script, it is very useful! </div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There are still issues with hypre not binding correctly, and I'm getting the error message occasionally (but much less often).</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I added some additional environment variables to the script that seem to make the behavior more consistent.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">export CUDA_DEVICE_ORDER=PCI_BUS_ID</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">export CUDA_VISIBLE_DEVICES=$MV2_COMM_WORLD_LOCAL_RANK ## as Victor suggested</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">export HYPRE_MEMORY_DEVICE=$MV2_COMM_WORLD_LOCAL_RANK</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The last environment variable is from hypre's documentation on GPUs.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">In 30 runs for a small problem size, 4 fail with a hypre-related error. Do you have
any other thoughts or suggestions?</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Best,</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Anna</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Victor Eijkhout <eijkhout@tacc.utexas.edu><br>
<b>Sent:</b> Thursday, February 1, 2024 11:26 AM<br>
<b>To:</b> Junchao Zhang <junchao.zhang@gmail.com>; Yesypenko, Anna <anna@oden.utexas.edu><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] errors with hypre with MPI and multiple GPUs on a node</span>
<div> </div>
</div>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="color: rgb(0, 0, 0);">Only for mvapich2-gdr:</span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="color: rgb(0, 0, 0);"> </span></p>
<div id="x_mail-editor-reference-message-container">
<div id="x_mail-editor-reference-message-container">
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);">#!/bin/bash</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"># Usage: mpirun -n <num_proc> MV2_USE_AFFINITY=0 MV2_ENABLE_AFFINITY=0 ./launch ./bin</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"> </span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);">export CUDA_VISIBLE_DEVICES=$MV2_COMM_WORLD_LOCAL_RANK</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);">case $MV2_COMM_WORLD_LOCAL_RANK in</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"> [0]) cpus=0-3 ;;</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"> [1]) cpus=64-67 ;;</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"> [2]) cpus=72-75 ;;</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);">esac</span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);"> </span></p>
<p style="margin: 0in 0in 3pt; font-family: Calibri, sans-serif; font-size: 11pt;">
<span style="font-family: Monaco; font-size: 9pt; color: rgb(29, 28, 29);">numactl --physcpubind=$cpus $@</span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"> </p>
</div>
</div>
</body>
</html>