[petsc-dev] Status of pthreads and OpenMP support

[Jed Brown]

In general, getting good performance with threads requires a "friendly"
environment. Many other systems use the "interleave all" approach where
they view DRAM as being uniform access and pay the price for lack of
locality. This is more forgiving (for both the programmer and the execution
environment), but we do not believe it has long-term relevance. Instead,
our model retains the semantic information about locality and, we believe,
will support better implementations in the long term. Writing these
implementations has not been fast so far, due to limited time and due to
fine-tuning of the threading model.

One challenge is that systems try to abstract physical memory to the point
where portable code cannot determine where memory is mapped. This causes
unpredictable performance. On Linux, we have libnuma, which gives us this
access, but so far, we have not been using libnuma explicitly within PETSc.
(This is partially because it's an extra dependency that is only sometimes
available, so everything has to also "work" without it; partly because Shri
does most local development on a Mac.) In any case, having the program
automatically debug quirks in the execution environment as well as internal
misuse in newly-written kernels is quite hard.

We need to make this "defensive code" better, but if you are experimenting
with the new threading interfaces, it's important to be familiar with NUMA
tools like numastat and procfs, profiling tools, and debugging tools. Email
trial-and-error performance analysis on unknown NUMA systems is an
extremely time-consuming process. If at all possible, I encourage you to
use these tools to figure out what is happening with threads spawned by
libraries, determining where the physical pages have gone, etc. If you
identify strange behavior due to an "unfriendly" environment, I encourage
you to think about how that environment can be detected automatically.

On Thu, Oct 25, 2012 at 8:24 PM, Shri <abhyshr at mcs.anl.gov> wrote:

> John, Dave,
>    As you've found out through your experimentation with PETSc's threading
> interface, there are still a lot of improvements that we do need to do
> before the threading support is stable. Your input certainly helps us in
> improving the functionality. I'll look at the logs you've sent, now that I
> have some time, and get back to you.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20121025/a28201eb/attachment.html>