[petsc-users] Error compiling with cuda
Satish Balay
balay.anl at fastmail.org
Sat Aug 2 12:20:49 CDT 2025
You can first fix on what MPI to use - and build everyting else with it. (and avoid mixing in multiple MPIs)
I've had issues with Ubuntu built MPI - but if it works for you - thats great!
You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that this MPI gets picked up] - or specify explicitly to PETSc with:
Or use 'cuda toolkit' instead of 'nvhpc-sdk'
./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx --with-fc=/usr/bin/mpif90 --download-hypre
etc..
BTW: PETSc configure attempts to run some binaries to detect compiler/library features, and such run tests (when linked in with cuda libraries) can fail if a cuda build is done on a box (container?) without nvidia driver installed.
Satish
On Sat, 2 Aug 2025, Renato E Poli wrote:
> Understood.
>
> I believe I am running in circles.
>
> I managed to create the container, but it ended up with multiple MPI
> implementations.
> It seems that CUDA SDK has its own MPI, and polutes the PATH.
> Is there any docker container PETSC+CUDA you would recommend?
> This is a snippet of my Dockerfile, but I am not 100% happy with it.
>
> <<<<<<
> FROM docker.io/ubuntu:24.04
> # Install necessary build tools and dependencies
> RUN apt-get update && apt-get install -y libmpich-dev gfortran
> libomp-dev (...)
> (...)
>
> ## NVIDIA HPC SDK
> RUN wget
> https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$
> &&
> tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz &&
> nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent --accept-license
> --instal-dir /opt/nvidia/hpc_sdk
>
> RUN cd petsc-3.23.5 &&\
> ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt
> --with-debugging=0 \
> --with-mpi --download-hypre --with-blaslapack --with-cuda
> --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \
> make all install && \
> cd ..
>
> (...)
> ENV PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH}
> ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH}
> ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH}
> >>>>>>
>
> Renato
>
> On Sat, Aug 2, 2025 at 11:11 AM Satish Balay <balay.anl at fastmail.org> wrote:
>
> > Please send configure.log
> >
> > presumably you are using a pre-installed mpi from hpc-sdk - that uses NV
> > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH]
> >
> > If you do not have MPI with gcc/gfortran - you can do something like:
> >
> > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> > --download-mpich [other options]
> >
> > When no options are specified - configure guesses - and picks up what it
> > can find - and sometimes that might not be what you intend to use..
> >
> > Satish
> >
> > On Sat, 2 Aug 2025, Renato E Poli wrote:
> >
> > > It seems that the installer is the one choosing to use nvfortran instead
> > of
> > > gfortran.
> > >
> > > I am building a docker container (clean environment).
> > > I simply install CUDA SDK from
> > >
> > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$
> > > , set paths and call petsc configure.
> > >
> > > RUN cd petsc-3.23.5 &&\
> > > ./configure --prefix=/usr/local/petsc --with-debugging=0 --with-mpi
> > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda && \
> > > make all && make install
> > >
> > > Do you see any reason for using the wrong compiler?
> > >
> > > Renato
> > >
> > > On Sat, Aug 2, 2025 at 12:28 AM Satish Balay <balay.anl at fastmail.org>
> > wrote:
> > >
> > > > NVFORTRAN (based on flang-17) does not support some F2008 features that
> > > > are required here (flang-19+ should work)
> > > >
> > > > Suggest using gfortran instead.
> > > >
> > > > Satish
> > > >
> > > > On Fri, 1 Aug 2025, Renato E Poli wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I see the error below when compiling with --with-cuda.
> > > > >
> > > > > Any suggestion?
> > > > >
> > > > >
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o
> > > > > CC
> > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o
> > > > > CC
> > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o
> > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o
> > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 18)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 30)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 42)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 54)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 65)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 76)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 88)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 100)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 112)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 124)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 136)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 148)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > 160)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h:
> > > > > 17)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > 6)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > 18)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > 30)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h: 5)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > 17)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > 29)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6)
> > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h:
> > > > > 17)
> > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef
> > > > >
> > > >
> > > >
> > >
> >
>
More information about the petsc-users
mailing list