[petsc-users] Explicit linking to OpenMP results in performance drop and wrong results

[Roland Richter]

Hei,

when compiling the attached files using the following compilation line

//usr/lib64/mpi/gcc/openmpi3/bin/mpicxx -DBOOST_ALL_NO_LIB
-DBOOST_FILESYSTEM_DYN_LINK -DBOOST_MPI_DYN_LINK
-DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_SERIALIZATION_DYN_LINK
-DUSE_CUDA
-I/home/roland/Dokumente/C++-Projekte/armadillo_with_PETSc/include
-I/opt/intel/compilers_and_libraries_2020.2.254/linux/mkl/include
-I/opt/armadillo/include -isystem /opt/petsc_release/include -isystem
/opt/fftw3/include -isystem /opt/boost/include -march=native
-fopenmp-simd -DMKL_LP64 -m64 -Wall -Wextra -pedantic -fPIC -flto -O2
-funroll-loops -funroll-all-loops -fstrict-aliasing -mavx -march=native
-fopenmp -std=gnu++17 -c <source_files> -o <target_files>/

and linking them with 

//usr/lib64/mpi/gcc/openmpi3/bin/mpicxx  -march=native -fopenmp-simd
-DMKL_LP64 -m64 <target_files> -o bin/armadillo_with_PETSc 
-Wl,-rpath,/opt/boost/lib:/opt/fftw3/lib64:/opt/petsc_release/lib
/usr/lib64/libgsl.so /usr/lib64/libgslcblas.so -lgfortran
/opt/intel/mkl/lib/intel64/libmkl_rt.so
/opt/boost/lib/libboost_filesystem.so.1.72.0
/opt/boost/lib/libboost_mpi.so.1.72.0
/opt/boost/lib/libboost_program_options.so.1.72.0
/opt/boost/lib/libboost_serialization.so.1.72.0
/opt/fftw3/lib64/libfftw3.so /opt/fftw3/lib64/libfftw3_mpi.so
/opt/petsc_release/lib/libpetsc.so
/usr/lib64/gcc/x86_64-suse-linux/9/libgomp.so/

my output is 

/Arma and PETSc/MatScale are equal:                                     
        0//
//Arma-time for a matrix size of [1024,
8192]:                                      24955//
//PETSc-time, pointer for a matrix size of [1024,
8192]:                      28283//
//PETSc-time, MatScale for a matrix size of [1024,
8192]:                  23138/

but when removing the explicit call to openmp (i.e. removing /-fopenmp/
and //usr/lib64/gcc/x86_64-suse-linux/9/libgomp.so/) my result is

/Arma and PETSc/MatScale are equal:                                     
       1//
//Arma-time for a matrix size of [1024,
8192]:                                     24878//
//PETSc-time, pointer for a matrix size of [1024,
8192]:                     18942//
//PETSc-time, MatScale for a matrix size of [1024,
8192]:                 23350/

even though both times the executable is linked to

/        libmkl_intel_lp64.so =>
/opt/intel/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so
(0x00007f9eebd70000)//
//        libmkl_core.so =>
/opt/intel/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64_lin/libmkl_core.so
(0x00007f9ee77aa000)//
//        libmkl_intel_thread.so =>
/opt/intel/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so
(0x00007f9ee42c3000)//
//        libiomp5.so =>
/opt/intel/compilers_and_libraries_2020.2.254/linux/compiler/lib/intel64_lin/libiomp5.so
(0x00007f9ee3ebd000)//
//        libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f9ea98bd000)/

via the petsc-library. Why does the execution time vary by so much, and
why does my result change when calling MatScale (i.e. returning wrong
results) when explicitly linking to OpenMP?

Thanks!

Regards,

Roland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main_short.cpp
Type: text/x-c++src
Size: 287 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_scaling.cpp
Type: text/x-c++src
Size: 4144 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: helper_functions.cpp
Type: text/x-c++src
Size: 2319 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_scaling.hpp
Type: text/x-c++hdr
Size: 576 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: helper_functions.hpp
Type: text/x-c++hdr
Size: 646 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210217/27fdd42a/attachment-0004.bin>