<div dir="ltr"><div dir="ltr"><i>>>
You can first fix on what MPI to use - and build everyting else with it. (and avoid mixing in multiple MPIs) </i><br></div><div>It sounds wise to make PETSC point to CUDA MPI.</div><div>I changed PETSC configure to:</div><div><br></div><div>$ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt --with-debugging=0 --with-mpi <b>--with-mpi-dir=${NVHPC_ROOT}/comm_libs/12.9/openmpi4/latest</b> --download-hypre --with-blaslapack --with-cuda --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && make all && make install <br></div><div dir="ltr"><br></div><div>And now it seems hypre is not happy:"Cannot use hypre with 64-bit BLAS/LAPACK indices".</div><div>Any idea on that one?</div><div><br></div><div dir="ltr"></div><div dir="ltr"><i>
>>
Or use 'cuda toolkit' instead of 'nvhpc-sdk'</i><br><div></div><div>We are using nvfortran, which is shipped with nvhpc-sdk AFAIK</div><div><br></div><div><i>>> E4S has containers with PETSc+CUDA installed (via Spack) - I guess you can give it a try </i><br></div><div>I'll take a look. Thanks!<br></div>
</div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sat, Aug 2, 2025 at 12:20 PM Satish Balay <<a href="mailto:balay.anl@fastmail.org">balay.anl@fastmail.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You can first fix on what MPI to use - and build everyting else with it. (and avoid mixing in multiple MPIs)<br>
<br>
I've had issues with Ubuntu built MPI - but if it works for you - thats great!<br>
<br>
You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that this MPI gets picked up] - or specify explicitly to PETSc with:<br>
<br>
Or use 'cuda toolkit' instead of 'nvhpc-sdk'<br>
<br>
./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx --with-fc=/usr/bin/mpif90 --download-hypre<br>
<br>
etc..<br>
<br>
BTW: PETSc configure attempts to run some binaries to detect compiler/library features, and such run tests (when linked in with cuda libraries) can fail if a cuda build is done on a box (container?) without nvidia driver installed.<br>
<br>
Satish<br>
<br>
<br>
On Sat, 2 Aug 2025, Renato E Poli wrote:<br>
<br>
> Understood.<br>
> <br>
> I believe I am running in circles.<br>
> <br>
> I managed to create the container, but it ended up with multiple MPI<br>
> implementations.<br>
> It seems that CUDA SDK has its own MPI, and polutes the PATH.<br>
> Is there any docker container PETSC+CUDA you would recommend?<br>
> This is a snippet of my Dockerfile, but I am not 100% happy with it.<br>
> <br>
> <<<<<<<br>
> FROM <a href="https://urldefense.us/v3/__http://docker.io/ubuntu:24.04__;!!G_uCfscf7eWS!aTD7PwgcN9IdFoU3hurKJnw85X7nNLAsmlyosyd-6P6lS0m9rIFh_yU_TTPD8dNgUi4oMcInCXCJci_QqRyCmg$" rel="noreferrer" target="_blank">docker.io/ubuntu:24.04</a><br>
> # Install necessary build tools and dependencies<br>
> RUN apt-get update && apt-get install -y libmpich-dev gfortran<br>
> libomp-dev (...)<br>
> (...)<br>
> <br>
> ## NVIDIA HPC SDK<br>
> RUN wget<br>
> <a href="https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$</a> <br>
> &&<br>
> tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz &&<br>
> nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent --accept-license<br>
> --instal-dir /opt/nvidia/hpc_sdk<br>
> <br>
> RUN cd petsc-3.23.5 &&\<br>
> ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt<br>
> --with-debugging=0 \<br>
> --with-mpi --download-hypre --with-blaslapack --with-cuda<br>
> --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \<br>
> make all install && \<br>
> cd ..<br>
> <br>
> (...)<br>
> ENV PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH}<br>
> ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH}<br>
> ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH}<br>
> >>>>>><br>
> <br>
> Renato<br>
> <br>
> On Sat, Aug 2, 2025 at 11:11 AM Satish Balay <<a href="mailto:balay.anl@fastmail.org" target="_blank">balay.anl@fastmail.org</a>> wrote:<br>
> <br>
> > Please send configure.log<br>
> ><br>
> > presumably you are using a pre-installed mpi from hpc-sdk - that uses NV<br>
> > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH]<br>
> ><br>
> > If you do not have MPI with gcc/gfortran - you can do something like:<br>
> ><br>
> > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran<br>
> > --download-mpich [other options]<br>
> ><br>
> > When no options are specified - configure guesses - and picks up what it<br>
> > can find - and sometimes that might not be what you intend to use..<br>
> ><br>
> > Satish<br>
> ><br>
> > On Sat, 2 Aug 2025, Renato E Poli wrote:<br>
> ><br>
> > > It seems that the installer is the one choosing to use nvfortran instead<br>
> > of<br>
> > > gfortran.<br>
> > ><br>
> > > I am building a docker container (clean environment).<br>
> > > I simply install CUDA SDK from<br>
> > ><br>
> > <a href="https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$</a><br>
> > > , set paths and call petsc configure.<br>
> > ><br>
> > > RUN cd petsc-3.23.5 &&\<br>
> > > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi<br>
> > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \<br>
> > > make all && make install<br>
> > ><br>
> > > Do you see any reason for using the wrong compiler?<br>
> > ><br>
> > > Renato<br>
> > ><br>
> > > On Sat, Aug 2, 2025 at 12:28 AM Satish Balay <<a href="mailto:balay.anl@fastmail.org" target="_blank">balay.anl@fastmail.org</a>><br>
> > wrote:<br>
> > ><br>
> > > > NVFORTRAN (based on flang-17) does not support some F2008 features that<br>
> > > > are required here (flang-19+ should work)<br>
> > > ><br>
> > > > Suggest using gfortran instead.<br>
> > > ><br>
> > > > Satish<br>
> > > ><br>
> > > > On Fri, 1 Aug 2025, Renato E Poli wrote:<br>
> > > ><br>
> > > > > Hi,<br>
> > > > ><br>
> > > > > I see the error below when compiling with --with-cuda.<br>
> > > > ><br>
> > > > > Any suggestion?<br>
> > > > ><br>
> > > > ><br>
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o<br>
> > > > > CC<br>
> > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o<br>
> > > > > CC<br>
> > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o<br>
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o<br>
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o<br>
> > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 18)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 30)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 42)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 54)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 65)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 76)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 88)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 100)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 112)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 124)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 136)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 148)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:<br>
> > > > 160)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h:<br>
> > > > > 17)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:<br>
> > > > > 6)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:<br>
> > > > > 18)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:<br>
> > > > > 30)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:<br>
> > > > 17)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:<br>
> > > > 29)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6)<br>
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line<br>
> > > > ><br>
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h:<br>
> > > > > 17)<br>
> > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef<br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> ><br>
> <br>
</blockquote></div></div>