[petsc-users] OpenMP doesn't work anymore with PETSc building rules

[Satish Balay]

I guess the flag you are looking for is CUDAFLAGS

>>>
balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100 CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp
/usr/local/cuda/bin/nvcc -o ex100.o -c -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include  -ccbin mpicxx -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo -gencode arch=compute_86,code=sm_86  -Xcompiler -fopenmp    -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include -I/usr/local/cuda/include    `pwd`/ex100.cu
mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0  -fopenmp -Wl,-export-dynamic ex100.o  -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex100
rm ex100.o
balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ 
<<<

Satish

On Wed, 25 Oct 2023, Qiyue Lu wrote:

> Even with
> CXXFLAGS=-Xcompiler -fopenmp -std=c++17
> LDFLAGS= -Xcompiler -fopenmp
> CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
> 
> won't work.
> 
> On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <qiyuelu1 at gmail.com> wrote:
> 
> > Thanks for your reply, using this configurations:
> >
> > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
> > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 --with-openmp=1
> > --with-threadsafety=1*
> > However, I got an error like:
> > *nvcc fatal   : Unknown option '-fopenmp'*
> > Previously, when I don't have --with-openmp for the configuration, the
> > PETSc make system can build my *.cu code using nvcc and g++, of course,
> > OpenMP doesn't work. Now with this --with-openmp option, it cannot even
> > build. The interesting thing is, I got this error even after removing the
> > *-fopenmp* from *CXXFLAGS* contents:
> > CXXFLAGS=-std=c++17
> > LDFLAGS=
> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
> > include ${PETSC_DIR}/lib/petsc/conf/variables
> > include ${PETSC_DIR}/lib/petsc/conf/rules
> >
> >
> >
> > Thanks,
> > Qiyue Lu
> >
> > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <balay at mcs.anl.gov> wrote:
> >
> >>
> >> On Wed, 25 Oct 2023, Qiyue Lu wrote:
> >>
> >> > Hello,
> >> > I have an in-house code enabled OpenMP and it works. Now I am trying to
> >> > incorporate PETSc as the linear solver and build together using the
> >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I found the
> >> > OpenMP part doesn't work anymore.
> >>
> >> If you are looking at building only your sources with openmp - using
> >> petsc formatted makefile [using petsc build rules],
> >> you can specify it via CFLAGS - either in makefile - or on command line.
> >>
> >> >>>>>>>
> >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the
> >> corresponding make fules]
> >>
> >> [balay at pj01 tutorials]$ make ex2
> >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> >> -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include
> >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
> >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> >> -L/home/balay/petsc/arch-linux-c-debug/lib
> >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
> >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
> >> -lquadmath -o ex2
> >> [balay at pj01 tutorials]$ make clean
> >> [balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
> >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include
> >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
> >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> >> -L/home/balay/petsc/arch-linux-c-debug/lib
> >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
> >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
> >> -lquadmath -o ex2
> >> [balay at pj01 tutorials]$
> >> <<<<<
> >>
> >> Satish
> >>
> >>
> >> > Should I re-configure the petsc installation with --with-openmp=1
> >> option? I
> >> > wonder are the building rules affected by this missing option?
> >> >
> >> > Thanks,
> >> > Qiyue Lu
> >> >
> >>
> >>
>