<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> I apologize, my suggestion of adding the --with-openmp fails in your environment. You should follow Satish's recommendation of adding the flags in your makefile exactly as needed.<div><br></div><div><span style="font-size: 13px;">If you are looking at building only your sources with openmp - using petsc formatted makefile [using petsc build rules],</span><br style="font-size: 13px;"><span style="font-size: 13px;">you can specify it via CFLAGS - either in makefile - or on command line.</span><br style="font-size: 13px;"><br style="font-size: 13px;"><blockquote type="cite" style="color: rgb(0, 0, 0); font-size: 13px;"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><span style="font-size: 13px;">For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the corresponding make fules]</span><br style="font-size: 13px;"><br style="font-size: 13px;"><span style="font-size: 13px;">[balay@pj01 tutorials]$ make ex2</span><br style="font-size: 13px;"><span style="font-size: 13px;">mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib -L/home/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex2</span><br style="font-size: 13px;"><span style="font-size: 13px;">[balay@pj01 tutorials]$ make clean</span><br style="font-size: 13px;"><span style="font-size: 13px;">[balay@pj01 tutorials]$ make ex2 CFLAGS=-fopenmp</span><br style="font-size: 13px;"><span style="font-size: 13px;">mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib -L/home/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex2</span><br style="font-size: 13px;"><span style="font-size: 13px;">[balay@pj01 tutorials]$</span></div><div><font size="2"><br id="lineBreakAtBeginningOfMessage"></font><div><br><blockquote type="cite"><div>On Oct 25, 2023, at 12:47 PM, Qiyue Lu <qiyuelu1@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr">Thanks, however, CUDAFLAGS doesn't work. <div>Even I removed all -fopenmp string from the make file, it still complains nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a background -fopenmp option added without -Xcompiler to any compiler. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I guess the flag you are looking for is CUDAFLAGS<br>
<br>
>>><br>
balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100 CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp<br>
/usr/local/cuda/bin/nvcc -o ex100.o -c -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include -ccbin mpicxx -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo -gencode arch=compute_86,code=sm_86 -Xcompiler -fopenmp -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include -I/usr/local/cuda/include `pwd`/<a href="http://ex100.cu/" rel="noreferrer" target="_blank">ex100.cu</a><br>
mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0 -fopenmp -Wl,-export-dynamic ex100.o -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex100<br>
rm ex100.o<br>
balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ <br>
<<<<br>
<br>
Satish<br>
<br>
On Wed, 25 Oct 2023, Qiyue Lu wrote:<br>
<br>
> Even with<br>
> CXXFLAGS=-Xcompiler -fopenmp -std=c++17<br>
> LDFLAGS= -Xcompiler -fopenmp<br>
> CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common<br>
> include ${PETSC_DIR}/lib/petsc/conf/variables<br>
> include ${PETSC_DIR}/lib/petsc/conf/rules<br>
> <br>
> won't work.<br>
> <br>
> On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <<a href="mailto:qiyuelu1@gmail.com" target="_blank">qiyuelu1@gmail.com</a>> wrote:<br>
> <br>
> > Thanks for your reply, using this configurations:<br>
> ><br>
> > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90<br>
> > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 --with-openmp=1<br>
> > --with-threadsafety=1*<br>
> > However, I got an error like:<br>
> > *nvcc fatal : Unknown option '-fopenmp'*<br>
> > Previously, when I don't have --with-openmp for the configuration, the<br>
> > PETSc make system can build my *.cu code using nvcc and g++, of course,<br>
> > OpenMP doesn't work. Now with this --with-openmp option, it cannot even<br>
> > build. The interesting thing is, I got this error even after removing the<br>
> > *-fopenmp* from *CXXFLAGS* contents:<br>
> > CXXFLAGS=-std=c++17<br>
> > LDFLAGS=<br>
> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common<br>
> > include ${PETSC_DIR}/lib/petsc/conf/variables<br>
> > include ${PETSC_DIR}/lib/petsc/conf/rules<br>
> ><br>
> ><br>
> ><br>
> > Thanks,<br>
> > Qiyue Lu<br>
> ><br>
> > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> ><br>
> >><br>
> >> On Wed, 25 Oct 2023, Qiyue Lu wrote:<br>
> >><br>
> >> > Hello,<br>
> >> > I have an in-house code enabled OpenMP and it works. Now I am trying to<br>
> >> > incorporate PETSc as the linear solver and build together using the<br>
> >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I found the<br>
> >> > OpenMP part doesn't work anymore.<br>
> >><br>
> >> If you are looking at building only your sources with openmp - using<br>
> >> petsc formatted makefile [using petsc build rules],<br>
> >> you can specify it via CFLAGS - either in makefile - or on command line.<br>
> >><br>
> >> >>>>>>><br>
> >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the<br>
> >> corresponding make fules]<br>
> >><br>
> >> [balay@pj01 tutorials]$ make ex2<br>
> >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas<br>
> >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector<br>
> >> -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include<br>
> >> -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic<br>
> >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib<br>
> >> -L/home/balay/petsc/arch-linux-c-debug/lib<br>
> >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib<br>
> >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13<br>
> >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11<br>
> >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++<br>
> >> -lquadmath -o ex2<br>
> >> [balay@pj01 tutorials]$ make clean<br>
> >> [balay@pj01 tutorials]$ make ex2 CFLAGS=-fopenmp<br>
> >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas<br>
> >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector<br>
> >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include<br>
> >> -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic<br>
> >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib<br>
> >> -L/home/balay/petsc/arch-linux-c-debug/lib<br>
> >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib<br>
> >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13<br>
> >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11<br>
> >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++<br>
> >> -lquadmath -o ex2<br>
> >> [balay@pj01 tutorials]$<br>
> >> <<<<<<br>
> >><br>
> >> Satish<br>
> >><br>
> >><br>
> >> > Should I re-configure the petsc installation with --with-openmp=1<br>
> >> option? I<br>
> >> > wonder are the building rules affected by this missing option?<br>
> >> ><br>
> >> > Thanks,<br>
> >> > Qiyue Lu<br>
> >> ><br>
> >><br>
> >><br>
> <br>
</blockquote></div>
</div></blockquote></div><br></div></body></html>