[Mochi-devel] Memory Registration time

[Philip Davis]

Thanks Matthieu, I didn’t realize there was a caching interface for Margo. Very nice. I will give that a try!

> On Nov 29, 2021, at 5:48 PM, Dorier, Matthieu <mdorier at anl.gov> wrote:
> 
> Hi Philip,
>  
> Margo doesn’t cache bulk handle, but it has some helper functions if you want to manage a cache by yourself:
> https://github.com/mochi-hpc/mochi-margo/blob/main/include/margo-bulk-pool.h <https://github.com/mochi-hpc/mochi-margo/blob/main/include/margo-bulk-pool.h>
>  
> margo_bulk_create is synchronous and blocks the calling ES, it doesn’t yield. If you want to have a dedicated ES handle registration, you’ll have wrap the call in a ULT that you submit to a pool associated with another ES. But I think the best would be to cache bulk handles and reuse them whenever possible.
>  
> Matthieu
>  
> From: mochi-devel <mochi-devel-bounces at lists.mcs.anl.gov> on behalf of Philip Davis <philip.davis at sci.utah.edu>
> Date: Monday, 29 November 2021 at 19:00
> To: mochi-devel at lists.mcs.anl.gov <mochi-devel at lists.mcs.anl.gov>
> Subject: [Mochi-devel] Memory Registration time
> 
> Hello,
> 
> Working on Frontera, I am seeing some fairly long times for memory registration. It takes 3.2 ms pretty consistently to do a margo_bulk_create for a 8MiB buffer. In light of recent discoveries regarding huge pages, I disabled THP on Frontera with prctl and saw registration time increase to 3.8ms. Anecdotally, pinning memory is a fairly expensive operation but I’m not sure what to expect. The same call on Summit takes about 500-600 usec. When multiple bulk handle creates are in contention, I see more variance, including some registrations that only take about 1.5 milliseconds, so I assume there’s some amount of caching somewhere. However, I’m running with FI_MR_CACHE_MAX_COUNT=0, since that’s the only way I can make rxm;verbs work for most (all?) recent version of libfabric. 
> 
> This leads me to a couple questions. First, is there any bulk handle caching inside Margo? Second, does margo_bulk_create block the calling execution stream in Margo, or are they being completed asynchronously while the handler yields? If the former, is there a different call I can use to do the latter?
> 
> Thanks,
> Philip
> _______________________________________________
> mochi-devel mailing list
> mochi-devel at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel <https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel>
> https://www.mcs.anl.gov/research/projects/mochi <https://www.mcs.anl.gov/research/projects/mochi>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mochi-devel/attachments/20211203/b55a9a76/attachment.html>