<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Mar 9, 2017 at 9:05 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Mar 9, 2017 at 8:08 PM, Richard Mills <span dir="ltr"><<a href="mailto:richardtmills@gmail.com" target="_blank">richardtmills@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Thu, Mar 9, 2017 at 7:45 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><span class="m_-5608737590444940916m_-4164452899966886196m_-7928386363841025087m_-7383488433093607975gmail-"><div><br>
</div>
</span><div>I started to play with memkind last summer. At that time, there were plenty of sayings online like this: </div>
<div><span style="font-family:"open sans",helvetica,arial,sans-serif;font-size:14px;font-variant-ligatures:normal;background-color:rgb(255,255,255)">"the </span><i style="margin:0px;padding:0px;border:0px;font-variant-ligatures:normal;font-variant-numeric:inherit;font-stretch:inherit;font-size:14px;line-height:inherit;font-family:"open sans",helvetica,arial,sans-serif;vertical-align:baseline;background-color:rgb(255,255,255)">hbwmalloc</i><span style="font-family:"open sans",helvetica,arial,sans-serif;font-size:14px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"> interface
is stable but </span><i style="margin:0px;padding:0px;border:0px;font-variant-ligatures:normal;font-variant-numeric:inherit;font-stretch:inherit;font-size:14px;line-height:inherit;font-family:"open sans",helvetica,arial,sans-serif;vertical-align:baseline;background-color:rgb(255,255,255)">memkind</i><span style="font-family:"open sans",helvetica,arial,sans-serif;font-size:14px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"> interface
is only partially stable."</span></div>
<div><br></div></div></div></blockquote><div><br></div></span><div>If you want the most stable interface, just use libnuma. It took me less than a day to reimplement hbwmalloc.h on top of libnuma and dlmalloc (<a href="https://github.com/jeffhammond/myhbwmalloc" target="_blank">https://github.com/jeffhammon<wbr>d/myhbwmalloc</a>). Note that myhbwmalloc was an education exercise, not software that I actually think anyone should use. It is intentionally brittle (fast or fail - nothing in between).</div><div><br></div><div>One consequence of using libnuma to manage MCDRAM is that one can call numa_move_pages, which Jed has asserted is the single most important function call in the history of memory management ;-)</div></div></div></div></blockquote><div><br></div></span><div>I think you can also move pages allocated by memkind around by calling numa_move_pages, actually, but this breaks the heap partitioning that memkind does.<br><br></div><div>I actually question whether we even need a heap manager for things like big arrays inside of Vec objects. It should be fine to just call mmap() directly for those. These will tend to be big things that don't get allocated/deallocated too frequently, so it probably won't matter that an expensive system call is required.<span class="m_-5608737590444940916HOEnZb"><font color="#888888"><br><br></font></span></div></div></div></div></blockquote><div><br></div></span><div>I think this is a terrible idea. What happens when a user runs a tiny debug job that takes 1000x longer than it should because every object ctor/dtor requires a system call?<span class="HOEnZb"><font color="#888888"><br></font></span></div></div></div></div></blockquote><div><br></div><div>I'm not necessarily saying it's a great idea, either. But if we are seriously wanting to do things like migrating our own pages around (expensive system call required), then doing our own mmap() and mbind() calls may make sense. You'd only want to do this for large and long-lived objects (like a Jacobian matrix) that you expect may need to reside in the high-bandwidth memory. I did a lot of low-level systems stuff for my dissertation, working on out-of-core calculations, and I wrote a middleware library that handled memory placement and did all of the memory allocation directly with mmap() calls. For long-lived linear algebra objects, the overhead for that was negligible.<br><br></div><div>Personally, I think that memkind is something we should support it in PETSc. That doesn't mean it's the ideal solution, but it has utility now. We can certainly support something else when something better arrives.<br><br></div><div>--Richard<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span class="HOEnZb"><font color="#888888"><br>Jeff</font></span></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span class="m_-5608737590444940916HOEnZb"><font color="#888888"></font></span></div><span class="m_-5608737590444940916HOEnZb"><font color="#888888"><div>--Richard<br></div></font></span><span><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Jeff</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div>
</div>
<div>Perhaps I should try memkind calls since they may become much better.</div>
<div><br>
</div>
<div>Hong (Mr.)</div></div></div></blockquote></span></div><span class="m_-5608737590444940916m_-4164452899966886196m_-7928386363841025087HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-5608737590444940916m_-4164452899966886196m_-7928386363841025087m_-7383488433093607975gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</font></span></div></div>
</blockquote></span></div><br></div></div>
</blockquote></span></div><span class=""><br><br clear="all"><div><br></div>-- <br><div class="m_-5608737590444940916gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</span></div></div>
</blockquote></div><br></div></div>