<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks Matthieu, I didn’t realize there was a caching interface for Margo. Very nice. I will give that a try!<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 29, 2021, at 5:48 PM, Dorier, Matthieu <<a href="mailto:mdorier@anl.gov" class="">mdorier@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Hi Philip,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Margo doesn’t cache bulk handle, but it has some helper functions if you want to manage a cache by yourself:<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><a href="https://github.com/mochi-hpc/mochi-margo/blob/main/include/margo-bulk-pool.h" style="color: blue; text-decoration: underline;" class="">https://github.com/mochi-hpc/mochi-margo/blob/main/include/margo-bulk-pool.h</a><o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">margo_bulk_create is synchronous and blocks the calling ES, it doesn’t yield. If you want to have a dedicated ES handle registration, you’ll have wrap the call in a ULT that you submit to a pool associated with another ES. But I think the best would be to cache bulk handles and reuse them whenever possible.<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Matthieu<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">mochi-devel <<a href="mailto:mochi-devel-bounces@lists.mcs.anl.gov" class="">mochi-devel-bounces@lists.mcs.anl.gov</a>> on behalf of Philip Davis <<a href="mailto:philip.davis@sci.utah.edu" class="">philip.davis@sci.utah.edu</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, 29 November 2021 at 19:00<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b><a href="mailto:mochi-devel@lists.mcs.anl.gov" class="">mochi-devel@lists.mcs.anl.gov</a> <<a href="mailto:mochi-devel@lists.mcs.anl.gov" class="">mochi-devel@lists.mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>[Mochi-devel] Memory Registration time<o:p class=""></o:p></span></p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hello,<br class=""><br class="">Working on Frontera, I am seeing some fairly long times for memory registration. It takes 3.2 ms pretty consistently to do a margo_bulk_create for a 8MiB buffer. In light of recent discoveries regarding huge pages, I disabled THP on Frontera with prctl and saw registration time increase to 3.8ms. Anecdotally, pinning memory is a fairly expensive operation but I’m not sure what to expect. The same call on Summit takes about 500-600 usec. When multiple bulk handle creates are in contention, I see more variance, including some registrations that only take about 1.5 milliseconds, so I assume there’s some amount of caching somewhere. However, I’m running with FI_MR_CACHE_MAX_COUNT=0, since that’s the only way I can make rxm;verbs work for most (all?) recent version of libfabric.<span class="Apple-converted-space"> </span><br class=""><br class="">This leads me to a couple questions. First, is there any bulk handle caching inside Margo? Second, does margo_bulk_create block the calling execution stream in Margo, or are they being completed asynchronously while the handler yields? If the former, is there a different call I can use to do the latter?<br class=""><br class="">Thanks,<br class="">Philip<br class="">_______________________________________________<br class="">mochi-devel mailing list<br class=""><a href="mailto:mochi-devel@lists.mcs.anl.gov" class="">mochi-devel@lists.mcs.anl.gov</a><br class=""><a href="https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel" style="color: blue; text-decoration: underline;" class="">https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel</a><br class=""><a href="https://www.mcs.anl.gov/research/projects/mochi" style="color: blue; text-decoration: underline;" class="">https://www.mcs.anl.gov/research/projects/mochi</a></div></div></div></div></blockquote></div><br class=""></body></html>