[petsc-dev] Fwd: Poisson step in GTS

[Barry Smith]

On Jun 19, 2011, at 8:13 AM, Jed Brown wrote:

> On Sun, Jun 19, 2011 at 03:33, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > No, Cray does not provide any threaded BLAS 1.  Generally speaking it is not worth threading a single nested loop unless the trip count is very high and generally that does not happen often enough to warrant the special BLAS.  In fact, I am not even sure we omp BLAS 2, I don't think so.
> 
> This shouldn't be surprising. It's worth noting that inside of multigrid (or a surface-volume split problem) the same vector operation will be called with quite different sizes. Managing the granularity so that threads are used when they will actually be faster, but not when the size is too small to offset the startup cost is not trivial.

   Huhh? VecDot() {if n is >> big use 2 threads else use 1} I don't see why that is hard?

> 
> A related matter that I keep harping on is that the memory hierarchy is very non-uniform. In the old days, it was reasonably uniform within a socket, but some of the latest hardware has multiple dies within a socket, each with more-or-less independent memory buses.

  So what is the numa.h you've been using. If we allocate vector arrays and matrix arrays then does that give you the locality?

   BTW: If it doesn't do it yet, ./configure needs to check for numa.h and do PETSC_HAVE_NUMA_H


   Barry

> 
> Of course you can always move MPI down to finer granularity (e.g. one MPI process per die instead of one per socket). I think this is a good solution for many applications, and may perform better than threads for reasonably large subdomains (mostly because memory affinity can be reliably managed), but it is less appealing in the strong scaling limit.
> 
> I have yet to see a threading library/language that offers a good platform for bandwidth-constrained strong scaling. The existing solutions tend to base everything on the absurd assumption that parallel computation is about parallelizing the computation. In reality, certainly for the sorts of problems that most PETSc users care about, everything hard about parallelism is how to communicate that which needs to be communicated while retaining good data locality on that which doesn't need to be communicated.
> 
> It's an easy local optimization to get high performance out of a local kernel. In contrast, optimizing for data locality often involves major data structure and algorithm changes.
> 
> The current systems all seem to be really bad at this, with data locality being something that is sometimes provided implicitly, but is ultimately very fragile. The number of recent threading papers that report less than 30% of hardware bandwidth peak for STREAM is not inspiring. (Rather few authors actually state the hardware peak, but if you look up the specs, it's really rare to see something respectable (e.g. 80%).)