<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 10, 2018 at 11:27 AM, Richard Tran Mills <span dir="ltr"><<a href="mailto:rtmills@anl.gov" target="_blank">rtmills@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Mon, Jul 9, 2018 at 10:04 AM, Jed Brown <span dir="ltr"><<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><span>Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>> writes:<br>
<br>
> This is the textbook Wrong Way to write OpenMP and the reason that the<br>
> thread-scalability of DOE applications using MPI+OpenMP sucks. It leads to<br>
> codes that do fork-join far too often and suffer from death by Amdahl,<br>
> unless you do a second pass where you fuse all the OpenMP regions and<br>
> replace the serial regions between them with critical sections or similar.<br>
><br>
</span><span>> This isn't how you'd write MPI, is it? No, you'd figure out how to<br>
> decompose your data properly to exploit locality and then implement an<br>
> algorithm that minimizes communication and synchronization. Do that with<br>
> OpenMP.<br>
<br>
</span></span><span class="">The applications that would call PETSc do not do this decomposition and<br>
<span style="background-color:rgb(255,255,0)">the OpenMP programming model does not provide a "communicator" or<br>
similar abstraction to associate the work done by the various threads.<br>
It's all implicit. </span></span></blockquote><div><br></div><div>This is perhaps the biggest single reason that I hate OpenMP.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>--Richard<br></div></font></span><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The idea with PETSc's threadcomm was to provide an<br>
object for this, but nobody wanted to call PETSc that way. It's clear<br>
that applications using OpenMP are almost exclusively interested in its<br>
incrementalism, not in doing it right. It's also pretty clear that the<br>
OpenMP forum agrees, otherwise they would be providing abstractions for<br>
performing collective operations across module boundaries within a<br>
parallel region.<br>
<br>
So the practical solution is to use OpenMP the way everyone else does,<br>
even if the performance is not good, because at least it works with the<br>
programming model the application has chosen.<br>
</blockquote></span></div><br></div></div>
</blockquote></div><br>The counter argument is that users who want this level of control are empowered to implement exactly what they need using the explicit threading model. A thread communicator is just the set of threads that work together and one can implement workshare and collective operations using that information. Given how inefficiently GOMP implementations barriers, folks should probably be rolling their own barriers anyways. If folks are deeply unhappy with this suggestion because of the work it requires, then perhaps DOE needs to fund somebody to write an open-source collectives library for OpenMP.</div><div class="gmail_extra"><br></div><div class="gmail_extra">For what it's worth, there's a active effort to make the teams construct valid in general (i.e. not just in the context of 'target'), which is intended to make NUMA programming easier. Currently, teams are defined by the implementation, but it may be possible to allow them to be user-defined. The challenge is that teams were initially created to support the OpenCL execution model that does not permit synchronization between work groups, so there may be objections to giving users control of their definition.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Jeff</div><div class="gmail_extra"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div>