[petsc-users] OpenMP doesn't work anymore with PETSc building rules

[Qiyue Lu]

Thanks, however, CUDAFLAGS doesn't work.
Even I removed all -fopenmp string from the make file, it still complains
nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a
background -fopenmp option added without -Xcompiler to any compiler.

On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <balay at mcs.anl.gov> wrote:

> I guess the flag you are looking for is CUDAFLAGS
>
> >>>
> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100
> CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp
> /usr/local/cuda/bin/nvcc -o ex100.o -c
> -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include  -ccbin mpicxx
> -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo
> -gencode arch=compute_86,code=sm_86  -Xcompiler -fopenmp
> -I/scratch/balay/petsc/include
> -I/scratch/balay/petsc/arch-linux-c-debug/include
> -I/usr/local/cuda/include    `pwd`/ex100.cu
> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> -fvisibility=hidden -g3 -O0  -fopenmp -Wl,-export-dynamic ex100.o
> -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib
> -L/scratch/balay/petsc/arch-linux-c-debug/lib
> -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64
> -L/usr/local/cuda/lib64/stubs
> -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib
> -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11
> -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart
> -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11
> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
> -lquadmath -o ex100
> rm ex100.o
> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$
> <<<
>
> Satish
>
> On Wed, 25 Oct 2023, Qiyue Lu wrote:
>
> > Even with
> > CXXFLAGS=-Xcompiler -fopenmp -std=c++17
> > LDFLAGS= -Xcompiler -fopenmp
> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
> > include ${PETSC_DIR}/lib/petsc/conf/variables
> > include ${PETSC_DIR}/lib/petsc/conf/rules
> >
> > won't work.
> >
> > On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <qiyuelu1 at gmail.com> wrote:
> >
> > > Thanks for your reply, using this configurations:
> > >
> > > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
> > > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1
> --with-openmp=1
> > > --with-threadsafety=1*
> > > However, I got an error like:
> > > *nvcc fatal   : Unknown option '-fopenmp'*
> > > Previously, when I don't have --with-openmp for the configuration, the
> > > PETSc make system can build my *.cu code using nvcc and g++, of course,
> > > OpenMP doesn't work. Now with this --with-openmp option, it cannot even
> > > build. The interesting thing is, I got this error even after removing
> the
> > > *-fopenmp* from *CXXFLAGS* contents:
> > > CXXFLAGS=-std=c++17
> > > LDFLAGS=
> > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
> > > include ${PETSC_DIR}/lib/petsc/conf/variables
> > > include ${PETSC_DIR}/lib/petsc/conf/rules
> > >
> > >
> > >
> > > Thanks,
> > > Qiyue Lu
> > >
> > > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <balay at mcs.anl.gov>
> wrote:
> > >
> > >>
> > >> On Wed, 25 Oct 2023, Qiyue Lu wrote:
> > >>
> > >> > Hello,
> > >> > I have an in-house code enabled OpenMP and it works. Now I am
> trying to
> > >> > incorporate PETSc as the linear solver and build together using the
> > >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I
> found the
> > >> > OpenMP part doesn't work anymore.
> > >>
> > >> If you are looking at building only your sources with openmp - using
> > >> petsc formatted makefile [using petsc build rules],
> > >> you can specify it via CFLAGS - either in makefile - or on command
> line.
> > >>
> > >> >>>>>>>
> > >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with
> the
> > >> corresponding make fules]
> > >>
> > >> [balay at pj01 tutorials]$ make ex2
> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> > >> -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include
> > >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm
> -lX11
> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath
> -lstdc++
> > >> -lquadmath -o ex2
> > >> [balay at pj01 tutorials]$ make clean
> > >> [balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> > >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include
> > >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm
> -lX11
> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath
> -lstdc++
> > >> -lquadmath -o ex2
> > >> [balay at pj01 tutorials]$
> > >> <<<<<
> > >>
> > >> Satish
> > >>
> > >>
> > >> > Should I re-configure the petsc installation with --with-openmp=1
> > >> option? I
> > >> > wonder are the building rules affected by this missing option?
> > >> >
> > >> > Thanks,
> > >> > Qiyue Lu
> > >> >
> > >>
> > >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231025/b37c0d98/attachment-0001.html>