<div dir="ltr">Thank you very much Barry, it works indeed to build petsc with the qopenmp flags!<div><br></div><div>Best</div><div><br></div><div>Timothee</div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-01-30 18:46 GMT+01:00 Satish Balay <span dir="ltr"><<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Try building PETSc with -qopenmp flag and see if this makes a difference.<br>
<br>
i.e use CFLAGS=-qopenmp FFLAGS=-qopenmp CXXFLAGS=-qopenmp<br>
<br>
Potential explanation:<br>
<br>
Compilers do different things internally [for ex - link to different<br>
libraries] when different compile flags are used.<br>
<br>
When -qopenmp is used - its possible some openmp library gets linked<br>
in. And when -qopenmp is not used - some equivalnet stub library might<br>
get linked in. Only one lib is supporsed to be used at anytime. With<br>
the way petsc gets built - both might get used - causing some<br>
conflict.<br>
<br>
You can verify with:<br>
<br>
<compare logs from><br>
mpicc -v -qopenmp dummy.c<br>
mpicc -v dummy.c<br>
<br>
And 'ldd libpetsc.so'<br>
<span class="HOEnZb"><font color="#888888"><br>
Satish<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Tue, 30 Jan 2018, Timothée Nicolas wrote:<br>
<br>
> Thank you for your answer,<br>
><br>
> I am not responsible for the choices in this code to begin with, but in any<br>
> case, as you can see in the example, I am not using any petsc call. Also in<br>
> our code, even though openmp is used, this is never in interaction with<br>
> petsc, the latter being used only at a specific (but crucial) place in the<br>
> code, and without openMP. In that sense, the model we are using is NOT some<br>
> 'Petsc with threads model', as described in the page you sent me.<br>
> Nevertheless, it appears that just linking petsc is enough to perturb the<br>
> behavior of openMP. I believe it may require further investigation. On top<br>
> of the bug I report in my first message, I have noticed very weird openMP<br>
> behavior lately, the kind of bugs that 'don't make sense' at first sight<br>
> (would be too long/boring to give details at this stage). I am starting to<br>
> realize that it may have something to do with petsc.<br>
><br>
> Again, the webpage you have sent me seems to adopt the point of view that<br>
> hybrid MPI/openMP is useless from the performance point of view, and/or<br>
> that Petsc routines should not be threaded. It does not, however, explain<br>
> why the behaviour of openMP may be changed by merely linking a program with<br>
> petsc, without a single petsc call in the program.<br>
><br>
> Best<br>
><br>
> 2018-01-30 18:02 GMT+01:00 Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>>:<br>
><br>
> ><br>
> > I don't know what you are trying to do with OpenMP and PETSc, nor do I<br>
> > understand why anyone would use OpenMP, but you cannot call virtually any<br>
> > PETSc function or operation while you are using threads.<br>
> ><br>
> > Best to use PETSc as intended, with one MPI process per core or hardware<br>
> > thread and not use OpenMP at all. <a href="http://www.mcs.anl.gov/petsc/" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/</a><br>
> > miscellaneous/petscthreads.<wbr>html<br>
> ><br>
> ><br>
> > Barry<br>
> ><br>
> ><br>
> > > On Jan 30, 2018, at 10:57 AM, Timothée Nicolas <<br>
> > <a href="mailto:timothee.nicolas@gmail.com">timothee.nicolas@gmail.com</a>> wrote:<br>
> > ><br>
> > > Dear petsc team,<br>
> > ><br>
> > > For a while, I have been wondering why I have never managed to print<br>
> > what threads are doing in an openMP region in my FORTRAN programs. Some<br>
> > people told me it was normal because threads will get confused all trying<br>
> > to write at the same time. However I realised today that the problem seems<br>
> > to be related to petsc. I have a super simple "hello world" example that<br>
> > reproduces the problem :<br>
> > ><br>
> > > program hello<br>
> > > !$ use omp_lib<br>
> > > implicit none<br>
> > > integer nthreads, tid<br>
> > ><br>
> > > !$omp parallel private(nthreads, tid)<br>
> > > tid = omp_get_thread_num()<br>
> > > print *, 'hello world from thread = ', tid<br>
> > ><br>
> > > if (tid .eq. 0) then<br>
> > > nthreads = omp_get_num_threads()<br>
> > > print *, 'number of threads = ', nthreads<br>
> > > end if<br>
> > > !$omp end parallel<br>
> > ><br>
> > > end program hello<br>
> > ><br>
> > > If I compile it with<br>
> > ><br>
> > > mpif90 -qopenmp -o omp main.f90<br>
> > ><br>
> > > Then I get no problem. But if I link petsc library (as is the case in my<br>
> > code) :<br>
> > ><br>
> > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-<wbr>3.7.3/arch-linux2-c-debug/lib<br>
> > -lpetsc<br>
> > ><br>
> > ><br>
> > > Then I get the following error after execution of export<br>
> > OMP_NUM_THREADS=2;./omp<br>
> > ><br>
> > > hello world from thread = 0<br>
> > > number of threads = 2<br>
> > > forrtl: severe (40): recursive I/O operation, unit -1, file unknown<br>
> > > Image PC Routine Line<br>
> > Source<br>
> > > omp 0000000000403BC8 Unknown Unknown<br>
> > Unknown<br>
> > > omp 0000000000403572 Unknown Unknown<br>
> > Unknown<br>
> > > libiomp5.so 00002AAAAD3146A3 Unknown Unknown<br>
> > Unknown<br>
> > > libiomp5.so 00002AAAAD2E3007 Unknown Unknown<br>
> > Unknown<br>
> > > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown<br>
> > Unknown<br>
> > > libiomp5.so 00002AAAAD3149C3 Unknown Unknown<br>
> > Unknown<br>
> > > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown<br>
> > Unknown<br>
> > > libc.so.6 00002AAAAD8C5CED Unknown Unknown<br>
> > Unknown<br>
> > ><br>
> > ><br>
> > > This means that in my programs using petsc, I can't use prints to see<br>
> > exactly what the threads are doing, which is a pain when debug is required<br>
> > (that is to say, all the time). Is this issue expected ?<br>
> > ><br>
> > > Thx in advance<br>
> > ><br>
> > > Timothee NICOLAS<br>
> > ><br>
> ><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>