[petsc-users] OpenMP doesn't work anymore with PETSc building rules

[Qiyue Lu]

NO, NO, NO, Any try or suggestions are meaningful. I appreciate help from
all of you. Have a nice day.

Qiyue Lu

On Wed, Oct 25, 2023 at 11:57 AM Barry Smith <bsmith at petsc.dev> wrote:

>
>    I apologize, my suggestion of adding the --with-openmp fails in your
> environment. You should follow Satish's recommendation of adding the flags
> in your makefile exactly as needed.
>
> If you are looking at building only your sources with openmp - using petsc
> formatted makefile [using petsc build rules],
> you can specify it via CFLAGS - either in makefile - or on command line.
>
>
> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the
> corresponding make fules]
>
> [balay at pj01 tutorials]$ make ex2
> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include
> -I/home/balay/petsc/arch-linux-c-debug/include     -Wl,-export-dynamic
> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> -L/home/balay/petsc/arch-linux-c-debug/lib
> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
> -lquadmath -o ex2
> [balay at pj01 tutorials]$ make clean
> [balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include
> -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic ex2.c
>  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
> -L/home/balay/petsc/arch-linux-c-debug/lib
> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11
> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
> -lquadmath -o ex2
> [balay at pj01 tutorials]$
>
>
> On Oct 25, 2023, at 12:47 PM, Qiyue Lu <qiyuelu1 at gmail.com> wrote:
>
> Thanks, however, CUDAFLAGS doesn't work.
> Even I removed all -fopenmp string from the make file, it still complains
> nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a
> background -fopenmp option added without -Xcompiler to any compiler.
>
> On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <balay at mcs.anl.gov> wrote:
>
>> I guess the flag you are looking for is CUDAFLAGS
>>
>> >>>
>> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100
>> CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp
>> /usr/local/cuda/bin/nvcc -o ex100.o -c
>> -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include  -ccbin mpicxx
>> -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo
>> -gencode arch=compute_86,code=sm_86  -Xcompiler -fopenmp
>> -I/scratch/balay/petsc/include
>> -I/scratch/balay/petsc/arch-linux-c-debug/include
>> -I/usr/local/cuda/include    `pwd`/ex100.cu
>> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
>> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
>> -fvisibility=hidden -g3 -O0  -fopenmp -Wl,-export-dynamic ex100.o
>> -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib
>> -L/scratch/balay/petsc/arch-linux-c-debug/lib
>> -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64
>> -L/usr/local/cuda/lib64/stubs
>> -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib
>> -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib
>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11
>> -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart
>> -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11
>> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++
>> -lquadmath -o ex100
>> rm ex100.o
>> balay at petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$
>> <<<
>>
>> Satish
>>
>> On Wed, 25 Oct 2023, Qiyue Lu wrote:
>>
>> > Even with
>> > CXXFLAGS=-Xcompiler -fopenmp -std=c++17
>> > LDFLAGS= -Xcompiler -fopenmp
>> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
>> > include ${PETSC_DIR}/lib/petsc/conf/variables
>> > include ${PETSC_DIR}/lib/petsc/conf/rules
>> >
>> > won't work.
>> >
>> > On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <qiyuelu1 at gmail.com> wrote:
>> >
>> > > Thanks for your reply, using this configurations:
>> > >
>> > > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
>> > > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1
>> --with-openmp=1
>> > > --with-threadsafety=1*
>> > > However, I got an error like:
>> > > *nvcc fatal   : Unknown option '-fopenmp'*
>> > > Previously, when I don't have --with-openmp for the configuration, the
>> > > PETSc make system can build my *.cu code using nvcc and g++, of
>> course,
>> > > OpenMP doesn't work. Now with this --with-openmp option, it cannot
>> even
>> > > build. The interesting thing is, I got this error even after removing
>> the
>> > > *-fopenmp* from *CXXFLAGS* contents:
>> > > CXXFLAGS=-std=c++17
>> > > LDFLAGS=
>> > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common
>> > > include ${PETSC_DIR}/lib/petsc/conf/variables
>> > > include ${PETSC_DIR}/lib/petsc/conf/rules
>> > >
>> > >
>> > >
>> > > Thanks,
>> > > Qiyue Lu
>> > >
>> > > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <balay at mcs.anl.gov>
>> wrote:
>> > >
>> > >>
>> > >> On Wed, 25 Oct 2023, Qiyue Lu wrote:
>> > >>
>> > >> > Hello,
>> > >> > I have an in-house code enabled OpenMP and it works. Now I am
>> trying to
>> > >> > incorporate PETSc as the linear solver and build together using the
>> > >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I
>> found the
>> > >> > OpenMP part doesn't work anymore.
>> > >>
>> > >> If you are looking at building only your sources with openmp - using
>> > >> petsc formatted makefile [using petsc build rules],
>> > >> you can specify it via CFLAGS - either in makefile - or on command
>> line.
>> > >>
>> > >> >>>>>>>
>> > >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with
>> the
>> > >> corresponding make fules]
>> > >>
>> > >> [balay at pj01 tutorials]$ make ex2
>> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
>> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
>> > >> -fvisibility=hidden -g3 -O0  -I/home/balay/petsc/include
>> > >> -I/home/balay/petsc/arch-linux-c-debug/include
>>  -Wl,-export-dynamic
>> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
>> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
>> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm
>> -lX11
>> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath
>> -lstdc++
>> > >> -lquadmath -o ex2
>> > >> [balay at pj01 tutorials]$ make clean
>> > >> [balay at pj01 tutorials]$ make ex2 CFLAGS=-fopenmp
>> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
>> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
>> > >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include
>> > >> -I/home/balay/petsc/arch-linux-c-debug/include
>>  -Wl,-export-dynamic
>> > >> ex2.c  -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -L/home/balay/petsc/arch-linux-c-debug/lib
>> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib
>> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13
>> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm
>> -lX11
>> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath
>> -lstdc++
>> > >> -lquadmath -o ex2
>> > >> [balay at pj01 tutorials]$
>> > >> <<<<<
>> > >>
>> > >> Satish
>> > >>
>> > >>
>> > >> > Should I re-configure the petsc installation with --with-openmp=1
>> > >> option? I
>> > >> > wonder are the building rules affected by this missing option?
>> > >> >
>> > >> > Thanks,
>> > >> > Qiyue Lu
>> > >> >
>> > >>
>> > >>
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231025/8297841c/attachment.html>