[petsc-dev] OpenMP Support

[Gerard Gorman]

Hi Dave

I should just say that we still have not finished a code review with the
petsc-dev team so any faults should be assumed to be ours rather than
theirs!

Michele just put a preprint of our paper covering the preliminary work
on arXiv, which you might found useful:
http://arxiv.org/abs/1205.2005

As we have not yet merged with the truck so you'd need to pull a branch
from:
https://bitbucket.org/wence/petsc-dev-omp
and configure using --with-openmp

A key issue when running is to set thread/core affinity. Unfortunately
there is no general way of doing this - it depends on your compiler. But
likwid-pin can make your life a little easier:
http://code.google.com/p/likwid/wiki/LikwidPin

Cheers
Gerard


Dave Nystrom emailed the following on 10/05/12 05:55:
> Hi Gerard,
>
> Thanks for the info.  Is there any documentation on how to use the petsc
> OpenMP support?  I would be interested in trying it out.
>
> Thanks,
>
> Dave
>
> Gerard Gorman writes:
>  > Hi Dave
>  > 
>  > OpenMP support exists for vec and mat (only AIJ so far). There is a big
>  > difference in performance depending on available memory bandwidth and
>  > the compiler OpenMP implementation. In application codes (such as
>  > Fluidity which is our main target code for this work) there are other
>  > significant costs such as matrix assembly. So in general you have to
>  > consider how easy it will be to thread to the other computationally
>  > expensive sections of your code, otherwise the overall speed-up of your
>  > application will be modest.
>  > 
>  > Cheers
>  > Gerard
>  >  
>  > Dave Nystrom emailed the following on 09/05/12 04:29:
>  > > Is the pthreads support further along than the OpenMP support?  I have not
>  > > tried the pthreads support yet.  Does either the pthreads support or the
>  > > OpenMP support implement the matvec or do they just do vector type
>  > > operations?
>  > >
>  > > Jed Brown writes:
>  > >  > On Tue, May 8, 2012 at 9:23 PM, Dave Nystrom <dnystrom1 at comcast.net> wrote:
>  > >  > 
>  > >  > > I see that petsc-dev now has some OpenMP support.  Would a serial, non-mpi
>  > >  > > code that uses petsc-dev be able to gain much performance improvement from
>  > >  > > it
>  > >  > > now for the case of doing sparse linear solve with cg and jacobi
>  > >  > > preconditioning?
>  > >  > >
>  > >  > 
>  > >  > The kernels are being transitioned to use the threadcomm, which enables
>  > >  > OpenMP and other threading models.
>  > >  > 
>  > >  > We anticipate that pthreads will provide the best performance because
>  > >  > operations can be less synchronous than with OpenMP (for which a parallel
>  > >  >  region implies barrier semantics). But if other parts of an application
>  > >  > are using OpenMP, it would be preferable for PETSc to also use OpenMP so
>  > >  > that it can share the same thread pool. The same applies to TBB.
>  >