[Mochi-devel] Memory Registration time

[Dorier, Matthieu]

Hi Philip,

Margo doesn’t cache bulk handle, but it has some helper functions if you want to manage a cache by yourself:
https://github.com/mochi-hpc/mochi-margo/blob/main/include/margo-bulk-pool.h

margo_bulk_create is synchronous and blocks the calling ES, it doesn’t yield. If you want to have a dedicated ES handle registration, you’ll have wrap the call in a ULT that you submit to a pool associated with another ES. But I think the best would be to cache bulk handles and reuse them whenever possible.

Matthieu

From: mochi-devel <mochi-devel-bounces at lists.mcs.anl.gov> on behalf of Philip Davis <philip.davis at sci.utah.edu>
Date: Monday, 29 November 2021 at 19:00
To: mochi-devel at lists.mcs.anl.gov <mochi-devel at lists.mcs.anl.gov>
Subject: [Mochi-devel] Memory Registration time
Hello,

Working on Frontera, I am seeing some fairly long times for memory registration. It takes 3.2 ms pretty consistently to do a margo_bulk_create for a 8MiB buffer. In light of recent discoveries regarding huge pages, I disabled THP on Frontera with prctl and saw registration time increase to 3.8ms. Anecdotally, pinning memory is a fairly expensive operation but I’m not sure what to expect. The same call on Summit takes about 500-600 usec. When multiple bulk handle creates are in contention, I see more variance, including some registrations that only take about 1.5 milliseconds, so I assume there’s some amount of caching somewhere. However, I’m running with FI_MR_CACHE_MAX_COUNT=0, since that’s the only way I can make rxm;verbs work for most (all?) recent version of libfabric.

This leads me to a couple questions. First, is there any bulk handle caching inside Margo? Second, does margo_bulk_create block the calling execution stream in Margo, or are they being completed asynchronously while the handler yields? If the former, is there a different call I can use to do the latter?

Thanks,
Philip
_______________________________________________
mochi-devel mailing list
mochi-devel at lists.mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel
https://www.mcs.anl.gov/research/projects/mochi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mochi-devel/attachments/20211129/9ced89dc/attachment.html>