[petsc-users] Petsc messing with OpenMP

[Smith, Barry F.]

  I don't know what you are trying to do with OpenMP and PETSc, nor do I understand why anyone would use OpenMP, but you cannot call virtually any PETSc function or operation while you are using threads. 

  Best to use PETSc as intended, with one MPI process per core or hardware thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html


  Barry


> On Jan 30, 2018, at 10:57 AM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Dear petsc team,
> 
> For a while, I have been wondering why I have never managed to print what threads are doing in an openMP region in my FORTRAN programs. Some people told me it was normal because threads will get confused all trying to write at the same time. However I realised today that the problem seems to be related to petsc. I have a super simple "hello world" example that reproduces the problem : 
> 
> program hello
>   !$ use omp_lib
>   implicit none
>   integer nthreads, tid
> 
>   !$omp parallel private(nthreads, tid)
>   tid = omp_get_thread_num()
>   print *, 'hello world from thread = ', tid
> 
>   if (tid .eq. 0) then
>      nthreads = omp_get_num_threads()
>      print *, 'number of threads = ', nthreads
>   end if
>   !$omp end parallel
> 
> end program hello
> 
> If I compile it with
> 
> mpif90 -qopenmp -o omp main.f90
> 
> Then I get no problem. But if I link petsc library (as is the case in my code) :
> 
> mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-3.7.3/arch-linux2-c-debug/lib -lpetsc
> 
> 
> Then I get the following error after execution of export OMP_NUM_THREADS=2;./omp
> 
>  hello world from thread =            0
>  number of threads =            2
> forrtl: severe (40): recursive I/O operation, unit -1, file unknown
> Image              PC                Routine            Line        Source             
> omp                0000000000403BC8  Unknown               Unknown  Unknown
> omp                0000000000403572  Unknown               Unknown  Unknown
> libiomp5.so        00002AAAAD3146A3  Unknown               Unknown  Unknown
> libiomp5.so        00002AAAAD2E3007  Unknown               Unknown  Unknown
> libiomp5.so        00002AAAAD2E26F5  Unknown               Unknown  Unknown
> libiomp5.so        00002AAAAD3149C3  Unknown               Unknown  Unknown
> libpthread.so.0    00002AAAAD5BADC5  Unknown               Unknown  Unknown
> libc.so.6          00002AAAAD8C5CED  Unknown               Unknown  Unknown
> 
> 
> This means that in my programs using petsc, I can't use prints to see exactly what the threads are doing, which is a pain when debug is required (that is to say, all the time). Is this issue expected ?
> 
> Thx in advance
> 
> Timothee NICOLAS
>