[petsc-users] Error compiling with cuda
Satish Balay
balay.anl at fastmail.org
Sat Aug 2 13:00:23 CDT 2025
On Sat, 2 Aug 2025, Renato E Poli wrote:
> *>> You can first fix on what MPI to use - and build everyting else with
> it. (and avoid mixing in multiple MPIs) *
> It sounds wise to make PETSC point to CUDA MPI.
> I changed PETSC configure to:
>
> $ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt
> --with-debugging=0 --with-mpi
> *--with-mpi-dir=${NVHPC_ROOT}/comm_libs/12.9/openmpi4/latest*
> --download-hypre --with-blaslapack --with-cuda
> --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && make all &&
> make install
>
> And now it seems hypre is not happy:"Cannot use hypre with 64-bit
> BLAS/LAPACK indices".
> Any idea on that one?
This is due to linking with 'cuda libraries' on a box without nvidia gpu/driver gives run-time errors. This breaks configure.
You can try using: --with-batch=1
>
> * >> Or use 'cuda toolkit' instead of 'nvhpc-sdk'*
> We are using nvfortran, which is shipped with nvhpc-sdk AFAIK
This e-mail thread started with:
>>>
NVFORTRAN-S-0034-Syntax error at or near end of line
(/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h: 6)
<<<
And a recommendation to use gfortran instead of nvfortran
Satish
>
> *>> E4S has containers with PETSc+CUDA installed (via Spack) - I guess you
> can give it a try *
> I'll take a look. Thanks!
>
> On Sat, Aug 2, 2025 at 12:20 PM Satish Balay <balay.anl at fastmail.org> wrote:
>
> > You can first fix on what MPI to use - and build everyting else with it.
> > (and avoid mixing in multiple MPIs)
> >
> > I've had issues with Ubuntu built MPI - but if it works for you - thats
> > great!
> >
> > You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that
> > this MPI gets picked up] - or specify explicitly to PETSc with:
> >
> > Or use 'cuda toolkit' instead of 'nvhpc-sdk'
> >
> > ./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx
> > --with-fc=/usr/bin/mpif90 --download-hypre
> >
> > etc..
> >
> > BTW: PETSc configure attempts to run some binaries to detect
> > compiler/library features, and such run tests (when linked in with cuda
> > libraries) can fail if a cuda build is done on a box (container?) without
> > nvidia driver installed.
> >
> > Satish
> >
> >
> > On Sat, 2 Aug 2025, Renato E Poli wrote:
> >
> > > Understood.
> > >
> > > I believe I am running in circles.
> > >
> > > I managed to create the container, but it ended up with multiple MPI
> > > implementations.
> > > It seems that CUDA SDK has its own MPI, and polutes the PATH.
> > > Is there any docker container PETSC+CUDA you would recommend?
> > > This is a snippet of my Dockerfile, but I am not 100% happy with it.
> > >
> > > <<<<<<
> > > FROM docker.io/ubuntu:24.04
> > > # Install necessary build tools and dependencies
> > > RUN apt-get update && apt-get install -y libmpich-dev gfortran
> > > libomp-dev (...)
> > > (...)
> > >
> > > ## NVIDIA HPC SDK
> > > RUN wget
> > >
> > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$
> > > &&
> > > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz &&
> > > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent
> > --accept-license
> > > --instal-dir /opt/nvidia/hpc_sdk
> > >
> > > RUN cd petsc-3.23.5 &&\
> > > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt
> > > --with-debugging=0 \
> > > --with-mpi --download-hypre --with-blaslapack --with-cuda
> > > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \
> > > make all install && \
> > > cd ..
> > >
> > > (...)
> > > ENV
> > PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH}
> > > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH}
> > > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH}
> > > >>>>>>
> > >
> > > Renato
> > >
> > > On Sat, Aug 2, 2025 at 11:11 AM Satish Balay <balay.anl at fastmail.org>
> > wrote:
> > >
> > > > Please send configure.log
> > > >
> > > > presumably you are using a pre-installed mpi from hpc-sdk - that uses
> > NV
> > > > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH]
> > > >
> > > > If you do not have MPI with gcc/gfortran - you can do something like:
> > > >
> > > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++
> > --with-fc=gfortran
> > > > --download-mpich [other options]
> > > >
> > > > When no options are specified - configure guesses - and picks up what
> > it
> > > > can find - and sometimes that might not be what you intend to use..
> > > >
> > > > Satish
> > > >
> > > > On Sat, 2 Aug 2025, Renato E Poli wrote:
> > > >
> > > > > It seems that the installer is the one choosing to use nvfortran
> > instead
> > > > of
> > > > > gfortran.
> > > > >
> > > > > I am building a docker container (clean environment).
> > > > > I simply install CUDA SDK from
> > > > >
> > > >
> > https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$
> > > > > , set paths and call petsc configure.
> > > > >
> > > > > RUN cd petsc-3.23.5 &&\
> > > > > ./configure --prefix=/usr/local/petsc --with-debugging=0
> > --with-mpi
> > > > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda
> > && \
> > > > > make all && make install
> > > > >
> > > > > Do you see any reason for using the wrong compiler?
> > > > >
> > > > > Renato
> > > > >
> > > > > On Sat, Aug 2, 2025 at 12:28 AM Satish Balay <balay.anl at fastmail.org
> > >
> > > > wrote:
> > > > >
> > > > > > NVFORTRAN (based on flang-17) does not support some F2008 features
> > that
> > > > > > are required here (flang-19+ should work)
> > > > > >
> > > > > > Suggest using gfortran instead.
> > > > > >
> > > > > > Satish
> > > > > >
> > > > > > On Fri, 1 Aug 2025, Renato E Poli wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I see the error below when compiling with --with-cuda.
> > > > > > >
> > > > > > > Any suggestion?
> > > > > > >
> > > > > > >
> > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o
> > > > > > > CC
> > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o
> > > > > > > CC
> > > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o
> > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o
> > > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o
> > > > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > 6)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 18)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 30)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 42)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 54)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 65)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 76)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 88)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 100)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 112)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 124)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 136)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 148)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > > 160)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h:
> > > > > > > 17)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > > 6)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > > 18)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > > 30)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > 5)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > > > 17)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > > > 29)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6)
> > > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > > >
> > > >
> > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h:
> > > > > > > 17)
> > > > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
More information about the petsc-users
mailing list