<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 2, 2018 at 10:32 AM, Smith, Barry F. <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
PETSc is for writing parallel codes using MPI, it is not for writing parallel codes with OpenMP. <a href="http://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>miscellaneous/petscthreads.<wbr>html</a></blockquote><div><br></div><div>To follow up:</div><div><br></div><div> 1) We do not have any expertise debugging OpenMP problems, and they are quite compiler specific so it is a big pain. This contributed to us removing explicit OpenMP from PETSc</div><div><br></div><div> 2) Nothing prevents you from using OpenMP yourself, but we do not know how to debug things like this (whereas we can help with MPI)</div><div><br></div><div> 3) We would caution you about using OpenMP. MPI can be used anywhere you can use OpenMP (for example, MPI can declare shared memory regions),</div><div> and it is generally faster, cleaner, and more maintainable.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Barry<br>
<br>
<br>
> On Mar 2, 2018, at 9:00 AM, Adrián Amor <<a href="mailto:aamor@pa.uc3m.es">aamor@pa.uc3m.es</a>> wrote:<br>
><br>
> Hi all,<br>
><br>
> I have been working in the last months with PETSC in a FEM program written on FORTRAN, so far sequential. Now, I want to parallelize it with OpenMP and I have found some problems. Finally, I have built a mockup program trying to localize the error.<br>
><br>
> 1. I have compiled PETSC with these options:<br>
> ./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blas-lapack-dir=/opt/<wbr>intel/mkl/lib/intel64/ --with-debugging=1 --with-scalar-type=complex --with-threadcomm --with-pthreadclasses --with-openmp --with-openmp-include=/opt/<wbr>intel/compilers_and_libraries_<wbr>2016.1.150/linux/compiler/lib/<wbr>intel64_lin --with-openmp-lib=/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>1.150/linux/compiler/lib/<wbr>intel64_lin/libiomp5.a PETSC_ARCH=linux-intel-dbg PETSC-AVOID-MPIF-H=1<br>
><br>
> (I have tried also removing --with-threadcomm --with-pthreadclasses and with libiomp5.so).<br>
><br>
> 2. The program to be executed is composed of two files, one is hellocount.F90:<br>
> MODULE hello_count<br>
> use omp_lib<br>
> IMPLICIT none<br>
><br>
> CONTAINS<br>
> subroutine hello_print ()<br>
> integer :: nthreads,mythread<br>
><br>
> !pragma hello-who-omp-f<br>
> !$omp parallel<br>
> nthreads = omp_get_num_threads()<br>
> mythread = omp_get_thread_num()<br>
> write(*,'("Hello from",i3," out of",i3)') mythread,nthreads<br>
> !$omp end parallel<br>
> !pragma end<br>
> end subroutine hello_print<br>
> END MODULE hello_count<br>
><br>
> and the other one is hellocount_main.F90:<br>
> Program Hello<br>
><br>
> USE hello_count<br>
><br>
> call hello_print<br>
><br>
> STOP<br>
><br>
> end Program Hello<br>
><br>
> 3. To compile these two functions I use:<br>
> rm -rf _obj<br>
> mkdir _obj<br>
><br>
> ifort -E -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-<wbr>intel-dbg/include -c hellocount.F90 >_obj/hellocount.f90<br>
> ifort -E -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-<wbr>intel-dbg/include -c hellocount_main.F90 >_obj/hellocount_main.f90<br>
><br>
> mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -I/home/aamor/MUMPS_5.1.2/<wbr>include -I/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>mkl/include -I/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>mkl/include/intel64/lp64/ -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-<wbr>intel-dbg/include -o _obj/hellocount.o -c _obj/hellocount.f90<br>
> mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -I/home/aamor/MUMPS_5.1.2/<wbr>include -I/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>mkl/include -I/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>mkl/include/intel64/lp64/ -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-<wbr>intel-dbg/include -o _obj/hellocount_main.o -c _obj/hellocount_main.f90<br>
><br>
> mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -o exec/HELLO _obj/hellocount.o _obj/hellocount_main.o /home/aamor/lib_tmp/libarpack_<wbr>LinuxIntel15.a /home/aamor/MUMPS_5.1.2/lib/<wbr>libzmumps.a /home/aamor/MUMPS_5.1.2/lib/<wbr>libmumps_common.a /home/aamor/MUMPS_5.1.2/lib/<wbr>libpord.a /home/aamor/parmetis-4.0.3/<wbr>lib/libparmetis.a /home/aamor/parmetis-4.0.3/<wbr>lib/libmetis.a -L/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpetsc -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_lapack95_lp64 -liomp5 -lpthread -lm -L/home/aamor/lib_tmp -lgidpost -lz /home/aamor/lua-5.3.3/src/<wbr>liblua.a /home/aamor/ESEAS-master/<wbr>libeseas.a -Wl,-rpath,/home/aamor/petsc/<wbr>linux-intel-dbg/lib -L/home/aamor/petsc/linux-<wbr>intel-dbg/lib -Wl,-rpath,/opt/intel/mkl/lib/<wbr>intel64 -L/opt/intel/mkl/lib/intel64 -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib</a> -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016/<wbr>linux/mkl/lib/intel64 -L/opt/intel/compilers_and_<wbr>libraries_2016/linux/mkl/lib/<wbr>intel64 -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>1.150/linux/compiler/lib/<wbr>intel64_lin -L/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_<wbr>64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-<wbr>linux/4.4.7 -Wl,-rpath,/opt/intel/mpi-rt/<wbr>5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/<wbr>5.1/intel64/lib -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcore_pic -lmpicxx -ldl -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib</a> -lmpifort -lmpi -lmpigi -lrt -lpthread -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib</a> -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016/<wbr>linux/mkl/lib/intel64 -L/opt/intel/compilers_and_<wbr>libraries_2016/linux/mkl/lib/<wbr>intel64 -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>1.150/linux/compiler/lib/<wbr>intel64_lin -L/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_<wbr>64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-<wbr>linux/4.4.7 -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016/<wbr>linux/mkl/lib/intel64 -L/opt/intel/compilers_and_<wbr>libraries_2016/linux/mkl/lib/<wbr>intel64 -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib</a> -Wl,-rpath,/opt/intel/mpi-rt/<wbr>5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/<wbr>5.1/intel64/lib -limf -lsvml -lirng -lm -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.<wbr>1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib" rel="noreferrer" target="_blank">5.1.2.150/<wbr>intel64/lib</a> -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016/<wbr>linux/mkl/lib/intel64 -L/opt/intel/compilers_and_<wbr>libraries_2016/linux/mkl/lib/<wbr>intel64 -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>1.150/linux/compiler/lib/<wbr>intel64_lin -L/opt/intel/compilers_and_<wbr>libraries_2016.1.150/linux/<wbr>compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_<wbr>64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-<wbr>linux/4.4.7 -Wl,-rpath,/opt/intel/<wbr>compilers_and_libraries_2016/<wbr>linux/mkl/lib/intel64 -L/opt/intel/compilers_and_<wbr>libraries_2016/linux/mkl/lib/<wbr>intel64 -ldl<br>
><br>
> exec/HELLO<br>
><br>
> 4. Then I have seen that:<br>
> 4.1. If I set OMP_NUM_THREADS=2 and I remove -lpetsc and -lifcore_pic from the last step, I got:<br>
> Hello from 0 out of 2<br>
> Hello from 1 out of 2<br>
> 4.2 But if add -lpetsc and -lifcore_pic (because I want to use PETSC) I get this error:<br>
> Hello from 0 out of 2<br>
> forrtl: severe (40): recursive I/O operation, unit -1, file unknown<br>
> Image PC Routine Line Source<br>
> HELLO 000000000041665C Unknown Unknown Unknown<br>
> HELLO 00000000004083C8 Unknown Unknown Unknown<br>
> libiomp5.so 00007F9C603566A3 Unknown Unknown Unknown<br>
> libiomp5.so 00007F9C60325007 Unknown Unknown Unknown<br>
> libiomp5.so 00007F9C603246F5 Unknown Unknown Unknown<br>
> libiomp5.so 00007F9C603569C3 Unknown Unknown Unknown<br>
> libpthread.so.0 0000003CE76079D1 Unknown Unknown Unknown<br>
> libc.so.6 0000003CE6AE88FD Unknown Unknown Unknown<br>
> If you set OMP_NUM_THREADS to 8, I get:<br>
> forrtl: severe (40): recursive I/O operation, unit -1, file unknown<br>
> forrtl: severe (40): recursive I/O operation, unit -1, file unknown<br>
> forrtl: severe (40): recursive I/O operation, unit -1, file unknown<br>
><br>
> I am sorry if this is a trivial problem because I guess that lots of people use PETSC with OpenMP in FORTRAN, but I have really done my best to figure out where the error is. Can you help me?<br>
><br>
> Thanks a lot!<br>
><br>
> Adrian.<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>