<div dir="ltr">Hi all,<div><br></div><div>I have been working in the last months with PETSC in a FEM program written on FORTRAN, so far sequential. Now, I want to parallelize it with OpenMP and I have found some problems. Finally, I have built a mockup program trying to localize the error. </div><div><br></div><div>1. I have compiled PETSC with these options:</div><div>./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64/ --with-debugging=1 --with-scalar-type=complex --with-threadcomm --with-pthreadclasses --with-openmp --with-openmp-include=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin --with-openmp-lib=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin/libiomp5.a PETSC_ARCH=linux-intel-dbg PETSC-AVOID-MPIF-H=1<br></div><div><br></div><div>(I have tried also removing  <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>--with-threadcomm --with-pthreadclasses and with libiomp5.so).</span></div><div><br></div><div>2. The program to be executed is composed of two files, one is hellocount.F90:</div><div><div>MODULE hello_count</div><div>  use omp_lib</div><div>  IMPLICIT none</div><div>  </div><div>  CONTAINS  </div><div>  subroutine hello_print ()</div><div>     integer :: nthreads,mythread</div><div>     </div><div>   !pragma hello-who-omp-f</div><div>   !$omp parallel</div><div>     nthreads = omp_get_num_threads()</div><div>     mythread = omp_get_thread_num()</div><div>     write(*,'("Hello from",i3," out of",i3)') mythread,nthreads</div><div>   !$omp end parallel</div><div>   !pragma end</div><div>   end subroutine hello_print</div><div>END MODULE hello_count</div></div><div><br></div><div>and the other one is hellocount_main.F90:</div><div><div>Program Hello</div><div><br></div><div>   USE hello_count</div><div>   </div><div>   call hello_print</div><div>   </div><div>   STOP</div><div><br></div><div>end Program Hello</div></div><div><br></div><div>3. To compile these two functions I use:</div><div><div>rm -rf _obj</div><div>mkdir _obj</div><div><br></div><div>ifort -E -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -c hellocount.F90 >_obj/hellocount.f90</div><div>ifort -E -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -c hellocount_main.F90 >_obj/hellocount_main.f90</div><div><br></div><div>mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -I/home/aamor/MUMPS_5.1.2/include   -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include/intel64/lp64/ -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -o _obj/hellocount.o -c _obj/hellocount.f90</div><div>mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -I/home/aamor/MUMPS_5.1.2/include   -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include/intel64/lp64/ -I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -o _obj/hellocount_main.o -c _obj/hellocount_main.f90</div><div><br></div><div>mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module _obj -I./_obj -o exec/HELLO _obj/hellocount.o _obj/hellocount_main.o /home/aamor/lib_tmp/libarpack_LinuxIntel15.a /home/aamor/MUMPS_5.1.2/lib/libzmumps.a /home/aamor/MUMPS_5.1.2/lib/libmumps_common.a /home/aamor/MUMPS_5.1.2/lib/libpord.a /home/aamor/parmetis-4.0.3/lib/libparmetis.a /home/aamor/parmetis-4.0.3/lib/libmetis.a -L/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpetsc -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_lapack95_lp64 -liomp5 -lpthread -lm -L/home/aamor/lib_tmp -lgidpost -lz /home/aamor/lua-5.3.3/src/liblua.a /home/aamor/ESEAS-master/libeseas.a -Wl,-rpath,/home/aamor/petsc/linux-intel-dbg/lib -L/home/aamor/petsc/linux-intel-dbg/lib -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64 -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcore_pic -lmpicxx -ldl -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -lmpifort -lmpi -lmpigi -lrt -lpthread -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -limf -lsvml -lirng -lm -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib/debug_mt">5.1.2.150/intel64/lib/debug_mt</a> -Wl,-rpath,/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -L/opt/intel/impi/<a href="http://5.1.2.150/intel64/lib">5.1.2.150/intel64/lib</a> -Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -ldl</div><div><br></div><div>exec/HELLO</div></div><div><br></div><div>4. Then I have seen that:</div><div>4.1. If I set OMP_NUM_THREADS=2 and I remove -lpetsc and -lifcore_pic from the last step, I got:</div><div><div>Hello from  0 out of  2</div><div>Hello from  1 out of  2</div></div><div>4.2 But if add -lpetsc and -lifcore_pic (because I want to use PETSC) I get this error:</div><div><div>Hello from  0 out of  2</div><div>forrtl: severe (40): recursive I/O operation, unit -1, file unknown</div><div>Image              PC                Routine            Line        Source</div><div>HELLO              000000000041665C  Unknown               Unknown  Unknown</div><div>HELLO              00000000004083C8  Unknown               Unknown  Unknown</div><div>libiomp5.so        00007F9C603566A3  Unknown               Unknown  Unknown</div><div>libiomp5.so        00007F9C60325007  Unknown               Unknown  Unknown</div><div>libiomp5.so        00007F9C603246F5  Unknown               Unknown  Unknown</div><div>libiomp5.so        00007F9C603569C3  Unknown               Unknown  Unknown</div><div>libpthread.so.0    0000003CE76079D1  Unknown               Unknown  Unknown</div><div>libc.so.6          0000003CE6AE88FD  Unknown               Unknown  Unknown</div></div><div>If you set OMP_NUM_THREADS to 8, I get:</div><div><div>forrtl: severe (40): recursive I/O operation, unit -1, file unknown</div><div>forrtl: severe (40): recursive I/O operation, unit -1, file unknown</div><div>forrtl: severe (40): recursive I/O operation, unit -1, file unknown</div></div><div><br></div><div>I am sorry if this is a trivial problem because I guess that lots of people use PETSC with OpenMP in FORTRAN, but I have really done my best to figure out where the error is. Can you help me?</div><div><br></div><div>Thanks a lot!</div><div><br></div><div>Adrian.</div></div>