[petsc-dev] Status of pthreads and OpenMP support

[Karl Rupp]

Hi,

 > Thanks for your reply.  Doing the memory bandwidth calculation seems 
like a useful exercise.  I'll
> give that a try.  I was also trying to think of this from a higher level perspective.  Does this seem
> reasonable?
>
> T_vec_op = T_vec_compute + T_vec_memory
>
> where these are times but using multiple threads only speeds up the T_vec_compute part while
> T_vec_memory is relatively constant whether I am doing memory operations with a single thread
> or multiple threads.

Yes and no :-)
Due to possible multiple physical memory links and NUMA, T_vec_memory 
shows a dependence on the number and affinity of threads. Also,

  T_vec_op = max(T_vec_compute, T_vec_memory)

can be a better approximation, as memory transfers and actual 
arithmetics may overlap ('prefetching').

Still, the main speed-up when using threads (or multiple processes) is 
in T_vec_compute. However, hardware processing speed has evolved such 
that T_vec_memory is now often dominant (exceptions are mostly BLAS 
level 3 algorithms), making proper data layout and affinity even more 
important.

Best regards,
Karli



> ________________________________________
> From: Karl Rupp [rupp at mcs.anl.gov]
> Sent: Friday, October 26, 2012 10:20 AM
> To: For users of the development version of PETSc
> Cc: Nystrom, William D
> Subject: Re: [petsc-dev] Status of pthreads and OpenMP support
>
> Hi Dave,
>
> let me just comment on the expected speed-up: As the arithmetic
> intensity of vector operations is small, you are in a memory-bandwidth
> limited regime. If you use smaller vectors in order to stay in cache,
> you may still not obtain the expected speedup because then thread
> management overhead becomes more of an issue. I suggest you compute the
> effective memory bandwidth of your vector operations, because I suspect
> you are pretty close to bandwidth saturation already.
>
> Best regards,
> Karli
>
>
> On 10/26/2012 10:58 AM, Nystrom, William D wrote:
>> Jed or Shri,
>>
>> Are there other preconditioners I could use/try now with the petsc thread support besides jacobi?
>> I looked around in the documentation for something like least squares polynomial preconditioning
>> that is referenced in a paper by Li and Saad titled "GPU-Accelerated Preconditioned Iterative
>> Linear Solvers" but did not find anything like that.  Would block jacobi with lu/cholesky for the
>> block solves work with the current thread support?
>>
>> Regarding the performance of my recent runs, I was surprised that I was not getting closer to a
>> 16x speedup for the purely vector operations when using 16 threads compared to 1 thread.  I'm
>> running on a single node of a cluster where the nodes are dual socked sandybridge cpus and
>> the OS is TOSS 2 Linux from Livermore.  So I'm assuming that is not really an "unknown" sort
>> of system.  One thing I am wondering is whether there is an issue with my thread affinities.  I am
>> setting them but am wondering if there could be issues with which chunk of a vector a given
>> threads gets.  For instance, assuming a single mpi process on a single node and using 16 threads,
>> I would assume that the vector occupies a contiguous chunk of memory and that it will get divided
>> into 16 chunks.  If thread 13 is the first to launch, does it get the first chunk of the vector or the
>> 13th chunk of the vector?  If the latter, then I would think my assignment of thread affinities is
>> optimal.  If my thread assignment is optimal, then is the less than 16x speedup in the vector
>> operations because of memory bandwidth limitations or cache effects?
>>
>> What profiling tools do you recommend to use with petsc?  I have investigated and tried Openspeedshop,
>> HPC Toolkit and Tau but have not tried any with petsc.  I was told that there were some issues with
>> using Tau with petsc.  Not sure what they are.  So far, I have liked Tau best.
>>
>> Dave
>>
>> ________________________________________
>> From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on behalf of John Fettig [john.fettig at gmail.com]
>> Sent: Friday, October 26, 2012 7:47 AM
>> To: For users of the development version of PETSc
>> Subject: Re: [petsc-dev] Status of pthreads and OpenMP support
>>
>> On Thu, Oct 25, 2012 at 9:16 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>> On Thu, Oct 25, 2012 at 8:05 PM, John Fettig <john.fettig at gmail.com> wrote:
>>>>
>>>> What I see in your results is about 7x speedup by using 16 threads.  I
>>>> think you should get better results by running 8 threads with 2
>>>> processes because the memory can be allocated on separate memory
>>>> controllers, and the memory will be physically closer to the cores.
>>>> I'm surprised that you get worse results.
>>>
>>>
>>> Our intent is for the threads to use an explicit first-touch policy so that
>>> they get local memory even when you have threads across multiple NUMA zones.
>>
>> Great.  I still think the performance using jacobi (as Dave does)
>> should be no worse using 2x(MPI) and 8x(thread) than it is with
>> 1x(MPI) and 16x(thread).
>>
>>>>
>>>> It doesn't surprise me that an explicit code gets much better speedup.
>>>
>>>
>>> The explicit code is much less dependent on memory bandwidth relative to
>>> floating point.
>>>
>>>>
>>>>
>>>>> I also get about the same performance results on the ex2 problem when
>>>>> running it with just
>>>>> mpi alone i.e. with 16 mpi processes.
>>>>>
>>>>> So from my perspective, the new pthreads/openmp support is looking
>>>>> pretty good assuming
>>>>> the issue with the MKL/external packages interaction can be fixed.
>>>>>
>>>>> I was just using jacobi preconditioning for ex2.  I'm wondering if there
>>>>> are any other preconditioners
>>>>> that might be multi-threaded.  Or maybe a polynomial preconditioner
>>>>> could work well for the
>>>>> new pthreads/openmp support.
>>>>
>>>> GAMG with SOR smoothing seems like a prime candidate for threading.  I
>>>> wonder if anybody has worked on this yet?
>>>
>>>
>>> SOR is not great because it's sequential.
>>
>> For structured grids we have multi-color schemes and temporally
>> blocked schemes as in this paper,
>>
>> http://www.it.uu.se/research/publications/reports/2006-018/2006-018-nc.pdf
>>
>> For unstructured grids, could we do some analagous decomposition using
>> e.g. parmetis?
>>
>> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.135.4764
>>
>> Regards,
>> John
>>
>>> A block Jacobi/SOR parallelizes
>>> fine, but does not guarantee stability without additional
>>> (operator-dependent) damping. Chebyshev/Jacobi smoothing will perform well
>>> with threads (but not all the kernels are ready).
>>>
>>> Coarsening and the Galerkin triple product is more difficult to thread.
>