[petsc-users] OpenMP doesn't work anymore with PETSc building rules

Barry Smith bsmith at petsc.dev
Wed Oct 25 11:57:45 CDT 2023 

   I apologize, my suggestion of adding the --with-openmp fails in your environment. You should follow Satish's recommendation of adding the flags in your makefile exactly as needed.

If you are looking at building only your sources with openmp - using petsc formatted makefile [using petsc build rules],
you can specify it via CFLAGS - either in makefile - or on command line.

>>>>>>> 
For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the corresponding make fules]

[balay at pj01 tutorials]$ make ex2
mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib -L/home/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex2
[balay at pj01 tutorials]$ make clean
[balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib -L/home/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex2
[balay at pj01 tutorials]$


> On Oct 25, 2023, at 12:47 PM, Qiyue Lu <qiyuelu1 at gmail.com> wrote:
> 
> Thanks, however, CUDAFLAGS doesn't work. 
> Even I removed all -fopenmp string from the make file, it still complains nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a background -fopenmp option added without -Xcompiler to any compiler. 
> 
> On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <balay at mcs.anl.gov <mailto:balay at mcs.anl.gov>> wrote:
>> I guess the flag you are looking for is CUDAFLAGS
>> 
>> >>>
>> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100 CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp
>> /usr/local/cuda/bin/nvcc -o ex100.o -c -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include  -ccbin mpicxx -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo -gencode arch=compute_86,code=sm_86  -Xcompiler -fopenmp    -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include -I/usr/local/cuda/include    `pwd`/ex100.cu <http://ex100.cu/>
>> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0  -fopenmp -Wl,-export-dynamic ex100.o  -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex100
>> rm ex100.o
>> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ 
>> <<<
>> 
>> Satish
>> 
>> On Wed, 25 Oct 2023, Qiyue Lu wrote:
>> 
>> > Even with
>> > CXXFLAGS=-Xcompiler -fopenmp -std=c++17
>> > LDFLAGS= -Xcompiler -fopenmp
>> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
>> > include ${PETSC_DIR}/lib/petsc/conf/variables
>> > include ${PETSC_DIR}/lib/petsc/conf/rules
>> > 
>> > won't work.
>> > 
>> > On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <qiyuelu1 at gmail.com <mailto:qiyuelu1 at gmail.com>> wrote:
>> > 
>> > > Thanks for your reply, using this configurations:
>> > >
>> > > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
>> > > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 --with-openmp=1
>> > > --with-threadsafety=1*
>> > > However, I got an error like:
>> > > *nvcc fatal   : Unknown option '-fopenmp'*
>> > > Previously, when I don't have --with-openmp for the configuration, the
>> > > PETSc make system can build my *.cu code using nvcc and g++, of course,
>> > > OpenMP doesn't work. Now with this --with-openmp option, it cannot even
>> > > build. The interesting thing is, I got this error even after removing the
>> > > *-fopenmp* from *CXXFLAGS* contents:
>> > > CXXFLAGS=-std=c++17
>> > > LDFLAGS=
>> > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
>> > > include ${PETSC_DIR}/lib/petsc/conf/variables
>> > > include ${PETSC_DIR}/lib/petsc/conf/rules
>> > >
>> > >
>> > >
>> > > Thanks,
>> > > Qiyue Lu
>> > >
>> > > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <balay at mcs.anl.gov <mailto:balay at mcs.anl.gov>> wrote:
>> > >
>> > >>
>> > >> On Wed, 25 Oct 2023, Qiyue Lu wrote:
>> > >>
>> > >> > Hello,
>> > >> > I have an in-house code enabled OpenMP and it works. Now I am trying to
>> > >> > incorporate PETSc as the linear solver and build together using the
>> > >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I found the
>> > >> > OpenMP part doesn't work anymore.
>> > >>
>> > >> If you are looking at building only your sources with openmp - using
>> > >> petsc formatted makefile [using petsc build rules],
>> > >> you can specify it via CFLAGS - either in makefile - or on command line.
>> > >>
>> > >> >>>>>>>
>> > >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the
>> > >> corresponding make fules]
>> > >>
>> > >> [balay at pj01 tutorials]$ make ex2
>> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
>> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
>> > >> -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include
>> > >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
>> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
>> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
>> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
>> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
>> > >> -lquadmath -o ex2
>> > >> [balay at pj01 tutorials]$ make clean
>> > >> [balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
>> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
>> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
>> > >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include
>> > >> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
>> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
>> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
>> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
>> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
>> > >> -lquadmath -o ex2
>> > >> [balay at pj01 tutorials]$
>> > >> <<<<<
>> > >>
>> > >> Satish
>> > >>
>> > >>
>> > >> > Should I re-configure the petsc installation with --with-openmp=1
>> > >> option? I
>> > >> > wonder are the building rules affected by this missing option?
>> > >> >
>> > >> > Thanks,
>> > >> > Qiyue Lu
>> > >> >
>> > >>
>> > >>
>> >

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231025/a5be86f7/attachment.html>

More information about the petsc-users mailing list