[petsc-users] Communication time with threading enabled.

[Gonçalo Albuquerque]

Dear All,

I'm experimenting with PETSc hybrid MPI/OpenMP capabilities and I have run
a rather simple test case (2D magnetostatic) using PETSc compiled with both
OpenMPI and thread support (both PThreads and OpenMP) on a Ubuntu 12.04
system. I cannot figure out the results obtained when comparing runs made
using the same number of MPI processes (2) and specifying either no threads
(-threadcomm_type nothread) or 1 OpenMP thread (-threadcomm_type openmp
-threadcomm_nthreads 1). I attached the logs of both runs. It seems that
the communication time has literally exploded. A grep over the logs gives:

No threading:
Average time for MPI_Barrier(): 1.38283e-06
Average time for zero size MPI_Send(): 7.03335e-06

1 OpenMP thread:
Average time for MPI_Barrier(): 0.00870218
Average time for zero size MPI_Send(): 0.00614798

The same things is occurring when running ksp ex5 (see attached logs).

Any ideas as to what I'm missing?

Many thanks in advance,

Gonçalo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130214/9a9b0bb9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nothread.log
Type: application/octet-stream
Size: 9356 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130214/9a9b0bb9/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openmp_nthreads_1.log
Type: application/octet-stream
Size: 9375 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130214/9a9b0bb9/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_nothread.log
Type: application/octet-stream
Size: 13199 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130214/9a9b0bb9/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_openmp_nthreads_1.log
Type: application/octet-stream
Size: 13244 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130214/9a9b0bb9/attachment-0007.obj>