<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner4kpm7b1 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton4kpm7b1:hover, .pfptPrimaryButton4kpm7b1:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton4kpm7b1:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
I talked to the MVAPICH people, and they told me to try adding /path/to/mvapich2-gdr/lib64/libmpi. so to LD_PRELOAD (apparently, they've had this issue before). This seemed to do the trick; I can build everything with MVAPICH2-GDR and run
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner4kpm7b1" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner4kpm7b1" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner4kpm7b1" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner4kpm7b1" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">I talked to the MVAPICH people, and they told me to try adding /path/to/mvapich2-gdr/lib64/libmpi.so to LD_PRELOAD (apparently, they've had this issue before). This seemed to do the trick; I can build everything with MVAPICH2-GDR and run with it now. Not sure if this is something you want to add to the docs.<div><br></div><div>Thanks,</div><div>Sreeram</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 17, 2024 at 9:17 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I looked at it before and checked again, and still see <a href="https://urldefense.us/v3/__https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/mpi.html*inter-gpu-communication-with-cuda-aware-mpi__;Iw!!G_uCfscf7eWS!bE42LoNqiOoD5Yu05BdtZqAHbFHxkuFy6S8ljr09QRqymgFnne-nbxx-xywoOtRzBGA3fRvcsOyyVgNDLWPRbI84MA$" target="_blank">https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/mpi.html#inter-gpu-communication-with-cuda-aware-mpi</a></div><div> > <span style="background-color:rgb(252,252,252);color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px">Using both MPI and NCCL to perform transfers between the same sets of CUDA devices concurrently is therefore not guaranteed to be safe.</span></div><div><br></div><div>I was scared by it. It means we have to replace all MPI device communications (what if they are from a third-party library?) with NCCL.</div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 17, 2024 at 8:27 AM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Yes, I saw this paper <a href="https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/abs/pii/S016781912100079X__;!!G_uCfscf7eWS!bE42LoNqiOoD5Yu05BdtZqAHbFHxkuFy6S8ljr09QRqymgFnne-nbxx-xywoOtRzBGA3fRvcsOyyVgNDLWMNhkqWSA$" target="_blank">https://www.sciencedirect.com/science/article/abs/pii/S016781912100079X</a><div dir="auto">that mentioned it, and I heard in Barry's talk at SIAM PP this year about the need for stream-aware MPI, so I was wondering if NCCL would be used in PETSc to do GPU-GPU communication. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 17, 2024, 7:58 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><br></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 17, 2024 at 7:51 AM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Do you know if there are plans for NCCL support in PETSc? </div></blockquote><div>What is your need? Do you mean using NCCL for the MPI communication?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 16, 2024, 10:41 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" rel="noreferrer" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Glad to hear you found a way. Did you use Frontera at TACC? If yes, I could have a try.<div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 16, 2024 at 8:35 PM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
I finally figured out a way to make it work. I had to build PETSc and my application using the (non GPU-aware) Intel MPI. Then, before running, I switch to the MVAPICH2-GDR. I'm not sure why that works, but it's the only way I've
</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_-4042369038964921293m_-6564809205233318008m_-4811160492260985038m_7925349526019486722m_-38707052680559594m_6588751887563132398pfptBanneruaotfdv" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-4042369038964921293m_-6564809205233318008m_-4811160492260985038m_7925349526019486722m_-38707052680559594m_6588751887563132398pfptBanneruaotfdv" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-4042369038964921293m_-6564809205233318008m_-4811160492260985038m_7925349526019486722m_-38707052680559594m_6588751887563132398pfptBanneruaotfdv" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender
</div>
<div id="m_-4042369038964921293m_-6564809205233318008m_-4811160492260985038m_7925349526019486722m_-38707052680559594m_6588751887563132398pfptBanneruaotfdv" style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization.
</div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"> </div>
</div>
<u></u>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">I finally figured out a way to make it work. I had to build PETSc and my application using the (non GPU-aware) Intel MPI. Then, before running, I switch to the MVAPICH2-GDR. <div>I'm not sure why that works, but it's the only way I've found to compile and run successfully without throwing any errors about not having a GPU-aware MPI. </div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 8, 2023 at 5:30 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" rel="noreferrer noreferrer" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You may need to set some env variables. This can be system specific so you might want to look at docs or ask TACC how to run with GPU-aware MPI.<div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 8, 2023 at 5:17 PM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Actually, when I compile my program with this build of PETSc and run, I still get the error:<div><br></div><div><font face="monospace">PETSC ERROR: PETSc is configured with GPU support, but your MPI is not GPU-aware. For better performance, please use a GPU-aware MPI.</font><br></div><div><font face="monospace"><br></font></div><div>I have the mvapich2-gdr module loaded and MV2_USE_CUDA=1.</div><div><br></div><div>Is there anything else I need to do?</div><div><br></div><div>Thanks,</div><div>Sreeram</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 8, 2023 at 3:29 PM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you, changing to CUDA 11.4 fixed the issue. The mvapich2-gdr module didn't require CUDA 11.4 as a dependency, so I was using 12.0</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 8, 2023 at 1:15 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov" rel="noreferrer noreferrer" target="_blank">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Executing: mpicc -show<br>
stdout: icc -I/opt/apps/cuda/11.4/include -I/opt/apps/cuda/11.4/include -lcuda -L/opt/apps/cuda/11.4/lib64/stubs -L/opt/apps/cuda/11.4/lib64 -lcudart -lrt -Wl,-rpath,/opt/apps/cuda/11.4/lib64 -Wl,-rpath,XORIGIN/placeholder -Wl,--build-id -L/opt/apps/cuda/11.4/lib64/ -lm -I/opt/apps/intel19/mvapich2-gdr/2.3.7/include -L/opt/apps/intel19/mvapich2-gdr/2.3.7/lib64 -Wl,-rpath -Wl,/opt/apps/intel19/mvapich2-gdr/2.3.7/lib64 -Wl,--enable-new-dtags -lmpi<br>
<br>
Checking for program /opt/apps/cuda/12.0/bin/nvcc...found<br>
<br>
Looks like you are trying to mix in 2 different cuda versions in this build.<br>
<br>
Perhaps you need to use cuda-11.4 - with this install of mvapich..<br>
<br>
Satish<br>
<br>
On Fri, 8 Dec 2023, Matthew Knepley wrote:<br>
<br>
> On Fri, Dec 8, 2023 at 1:54 PM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:<br>
> <br>
> > I am trying to build PETSc with CUDA using the CUDA-Aware MVAPICH2-GDR.<br>
> ><br>
> > Here is my configure command:<br>
> ><br>
> > ./configure PETSC_ARCH=linux-c-debug-mvapich2-gdr --download-hypre<br>
> > --with-cuda=true --cuda-dir=$TACC_CUDA_DIR --with-hdf5=true<br>
> > --with-hdf5-dir=$TACC_PHDF5_DIR --download-elemental --download-metis<br>
> > --download-parmetis --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90<br>
> ><br>
> > which errors with:<br>
> ><br>
> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for<br>
> > details):<br>
> ><br>
> > ---------------------------------------------------------------------------------------------<br>
> > CUDA compile failed with arch flags " -ccbin mpic++ -std=c++14<br>
> > -Xcompiler -fPIC<br>
> > -Xcompiler -fvisibility=hidden -g -lineinfo -gencode<br>
> > arch=compute_80,code=sm_80"<br>
> > generated from "--with-cuda-arch=80"<br>
> ><br>
> ><br>
> ><br>
> > The same configure command works when I use the Intel MPI and I can build<br>
> > with CUDA. The full config.log file is attached. Please let me know if you<br>
> > need any other information. I appreciate your help with this.<br>
> ><br>
> <br>
> The proximate error is<br>
> <br>
> Executing: nvcc -c -o /tmp/petsc-kn3f29gl/config.packages.cuda/conftest.o<br>
> -I/tmp/petsc-kn3f29gl/config.setCompilers<br>
> -I/tmp/petsc-kn3f29gl/config.types<br>
> -I/tmp/petsc-kn3f29gl/config.packages.cuda -ccbin mpic++ -std=c++14<br>
> -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo -gencode<br>
> arch=compute_80,code=sm_80 /tmp/petsc-kn3f29gl/config.packages.cuda/<br>
> <a href="https://urldefense.us/v3/__http://conftest.cu__;!!G_uCfscf7eWS!duKUz7pE9N0adJ-FOW7PLZ_1cSZvYlnqh7J0TIcZN0v8RLplcWxh1YE8Vis29K0cuw_zAvjdK-H9H2JYYuUUKRXxlA$" rel="noreferrer noreferrer noreferrer" target="_blank">conftest.cu</a><br>
> stdout:<br>
> /opt/apps/cuda/11.4/include/crt/sm_80_rt.hpp(141): error: more than one<br>
> instance of overloaded function "__nv_associate_access_property_impl" has<br>
> "C" linkage<br>
> 1 error detected in the compilation of<br>
> "/tmp/petsc-kn3f29gl/config.packages.cuda/<a href="https://urldefense.us/v3/__http://conftest.cu__;!!G_uCfscf7eWS!duKUz7pE9N0adJ-FOW7PLZ_1cSZvYlnqh7J0TIcZN0v8RLplcWxh1YE8Vis29K0cuw_zAvjdK-H9H2JYYuUUKRXxlA$" rel="noreferrer noreferrer noreferrer" target="_blank">conftest.cu</a>".<br>
> Possible ERROR while running compiler: exit code 1<br>
> stderr:<br>
> /opt/apps/cuda/11.4/include/crt/sm_80_rt.hpp(141): error: more than one<br>
> instance of overloaded function "__nv_associate_access_property_impl" has<br>
> "C" linkage<br>
> <br>
> 1 error detected in the compilation of<br>
> "/tmp/petsc-kn3f29gl/config.packages.cuda<br>
> <br>
> This looks like screwed up headers to me, but I will let someone that<br>
> understands CUDA compilation reply.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Thanks,<br>
> > Sreeram<br>
> ><br>
> <br>
> <br>
> </blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>