<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr" style="text-align:left; direction:ltr;">
<div>Hi Hyogi and Phil</div>
<div><br>
</div>
<div>Yes that's very likely that could be the issue. Indeed handles can only be re-used once the user releases them after a call to HG_Destroy() in the RPC handler callback, meaning that callback also has to be triggered. </div>
<div><br>
</div>
<div>The limit by default is 256 so I'm not sure if that really matches the 512 from your report. At any rate there are two cmake build variables that can affect that behavior:</div>
<div>MERCURY_ENABLE_POST_LIMIT</div>
<div>and</div>
<div>MERCURY_POST_LIMIT (set to 256 by default)</div>
<div><br>
</div>
<div>You can try to simply turn off that limit by setting MERCURY_ENABLE_POST_LIMIT to OFF (if you are using ccmake, press 't' to toggle advanded mode).</div>
<div><br>
</div>
<div>Here is also a quick patch if you are building through spack</div>
<div><br>
</div>
<div>/packages/mercury/package.py</div>
<div>index 31ee318c9..086ab6266 100644</div>
<div>--- a/var/spack/repos/builtin/packages/mercury/package.py</div>
<div>+++ b/var/spack/repos/builtin/packages/mercury/package.py</div>
<div>@@ -69,6 +69,7 @@ def cmake_args(self):</div>
<div> '-DBUILD_SHARED_LIBS:BOOL=%s' % variant_bool('+shared'),</div>
<div> '-DBUILD_TESTING:BOOL=%s' % str(self.run_tests),</div>
<div> '-DMERCURY_ENABLE_PARALLEL_TESTING:BOOL=%s' % str(parallel_tests),</div>
<div>+ '-DMERCURY_ENABLE_POST_LIMIT:BOOL=OFF',</div>
<div> '-DMERCURY_USE_BOOST_PP:BOOL=ON',</div>
<div> '-DMERCURY_USE_CHECKSUMS:BOOL=ON',</div>
<div> '-DMERCURY_USE_EAGER_BULK:BOOL=ON',</div>
<div></div>
<div><br>
</div>
<div>Jerome</div>
<div><br>
</div>
<div><br>
</div>
<div>On Fri, 2020-09-25 at 13:21 +0000, Carns, Philip H. wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Thanks for the clarifications Hyogi (in this and the previous email), that helps quite a bit.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
If it happens with both ofi+tcp and bmi then that (probably) rules out transport problem. Neither requires memory registration either, which is another possible transport-level resource constraint.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Jerome, do you know of anything that could be an issue in Mercury 1.0.1 w.r.t. resource consumption with a pattern like this, where rpc handler handles are held open for an extended period waiting on a chain of dependent RPCs to complete?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I was wondering about the HG_POST_LIMIT. Do incoming RPC handles count against this limit until the handler is completed at trigger time, or until the handle is destroyed?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
If it is the former, margo should recycle them very quickly. The trigger callbacks only execute long enough to spawn detached Argobots ULTS before returning (and those ULTS are using deferred execution, so there isn't much work there). If it is the latter,
that could be an issue for this case, though, because the handles at the root of the trees won't be destroyed until after all of the dependent RPCs are complete, and it might eventually not leave enough buffers free to make progress on the chain.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
-Phil<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt;" face="Calibri, sans-serif" color="#000000"><b>From:</b> Sim, Hyogi <simh@ornl.gov><br>
<b>Sent:</b> Thursday, September 24, 2020 5:33 PM<br>
<b>To:</b> Carns, Philip H. <carns@mcs.anl.gov><br>
<b>Cc:</b> Sim, Hyogi <simh@ornl.gov>; mochi-devel@lists.mcs.anl.gov <mochi-devel@lists.mcs.anl.gov>; Brim, Michael J. <brimmj@ornl.gov>; Wang, Feiyi <fwang2@ornl.gov><br>
<b>Subject:</b> Re: [EXTERNAL] Re: Margo handler hangs on flooded requests</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText"><br>
<br>
> On Sep 24, 2020, at 5:06 PM, Carns, Philip H. <carns@mcs.anl.gov> wrote:<br>
> <br>
> Oh and one other simple high level question: in the reproducer, every server is initiating a broadcast with itself as the root at the same time? Otherwise there would only be one handler in operation at a time per server, and presumably it doesn't hang in
that case?<br>
<br>
For "512 nodes/ppn1", every server is initiating a broadcast at the same time, so 512 broadcasting operations are going at the same time. In this case, each server will handle 511 rpc requests from other peers. And, this runs successful.
<br>
<br>
For "512 nodes/ppn2", every server is initiating two broadcasts at the same time, so 1024 broadcasting at the same time. Each server is expected to handle 511*2 rpc requests. This hangs in the middle, like after handling around 510 requests.
<br>
<br>
In the reproducer, the server operation is triggered by local client applications. The server launches a separate rpc channel (margo instance with na+sm://) for local clients. This part of the server is called 'listener' in the code:<br>
<br>
<a href="https://code.ornl.gov/hyogi/metasim/-/blob/master/server/src/metasim-listener.c">https://code.ornl.gov/hyogi/metasim/-/blob/master/server/src/metasim-listener.c</a><br>
<br>
The client applications of the sum (which asks the local server to initiate the broadcasting sum) is:<br>
<br>
<a href="https://code.ornl.gov/hyogi/metasim/-/blob/master/examples/src/sum.c">https://code.ornl.gov/hyogi/metasim/-/blob/master/examples/src/sum.c</a><br>
<br>
<br>
(I've added some collaborators in this email).<br>
<br>
Thanks,<br>
Hyogi<br>
<br>
<br>
> <br>
> thanks,<br>
> -Phil<br>
> From: mochi-devel <mochi-devel-bounces@lists.mcs.anl.gov> on behalf of Carns, Philip H. <carns@mcs.anl.gov><br>
> Sent: Thursday, September 24, 2020 5:00 PM<br>
> To: Sim, Hyogi <simh@ornl.gov>; mochi-devel@lists.mcs.anl.gov <mochi-devel@lists.mcs.anl.gov><br>
> Subject: Re: [Mochi-devel] Margo handler hangs on flooded requests<br>
> <br>
> Hi Hyogi,<br>
> <br>
> I'm not going to lie, that's a daunting bug report 🙂 Thank you for all of the detailed information and reproducer!<br>
> <br>
> Your theory of resource exhaustion triggering a deadlock seems likely. Off the top of my head I'm not sure what the problematic resource would be, though. Some random ideas would be that a limit on the number of preposted buffers (for incoming RPC's) has
been exhausted, or that the handlers are inadvertently blocking on something that is not Argobots aware and clogging up the RPC pools.<br>
> <br>
> I have some preliminary questions about the test environment (I apologize if this is covered in the repo; I have not dug into it yet):<br>
> • Exactly what metric are you showing in the graphs (as in, what Argobots function are you using to retrieve the information)? Just making sure there is no ambiguity in the interpretation. It looks like the graphs are showing suspended threads and
threads that are eligible for execution.<br>
> • What versions of Mercury, Argobots, and Libfabric (if present) are you using?<br>
> • Which transport are you using in Mercury? Are you modifying it's behavior with any environment variables?<br>
> • How big is the RPC handler pool, and is there a dedicated progress thread for Margo?<br>
> thanks,<br>
> -Phil<br>
> From: mochi-devel <mochi-devel-bounces@lists.mcs.anl.gov> on behalf of Sim, Hyogi <simh@ornl.gov><br>
> Sent: Thursday, September 24, 2020 1:31 PM<br>
> To: mochi-devel@lists.mcs.anl.gov <mochi-devel@lists.mcs.anl.gov><br>
> Subject: [Mochi-devel] Margo handler hangs on flooded requests<br>
> <br>
> Hello,<br>
> <br>
> I am a part of the UnifyFS development team (<a href="https://github.com/LLNL/UnifyFS">https://github.com/LLNL/UnifyFS</a>). We use the margo framework for our rpc communications.<br>
> <br>
> We are currently redesigning our metadata handling, and the new design includes broadcasting operations across server daemons. UnifyFS spawns one sever daemon per compute node. For broadcasting, we first build a binary tree of server ranks, rooted by the
rank who initiates the broadcasting, and then recursively forward the request to child nodes in the tree. When a server doesn't have a child (i.e., a leaf node), it will directly respond to its parent.<br>
> <br>
> While testing our implementation, we've found that the servers hang while handling rpc requests, especially when many broadcasting operations are triggered simultaneously. Although we cannot exactly identify the number of broadcasting operations/servers from
which it starts to fail, we can always reproduce with a sufficiently large scale, for instance, 48 nodes with ppn=16 (48 servers and 16 broadcasts per server). Our tests were performed on Summit and Summitdev.<br>
> <br>
> To isolate and debug the problem, we wrote a separate test program without the unifyfs codepath. Specifically, each server triggers a sum operation that adds up all ranks of servers using the broadcast operation. The following is the snippet of the sum operation
(Full code is at <a href="https://code.ornl.gov/hyogi/metasim/-/blob/master/server/src/metasim-rpc.c):">
https://code.ornl.gov/hyogi/metasim/-/blob/master/server/src/metasim-rpc.c):</a><br>
> <br>
> ---<br>
> <br>
> /*<br>
> * sum rpc (broadcasting)<br>
> */<br>
> <br>
> static int sum_forward(metasim_rpc_tree_t *tree,<br>
> metasim_sum_in_t *in, metasim_sum_out_t *out)<br>
> {<br>
> int ret = 0;<br>
> int i;<br>
> int32_t seed = 0;<br>
> int32_t partial_sum = 0;<br>
> int32_t sum = 0;<br>
> int child_count = tree->child_count;<br>
> int *child_ranks = tree->child_ranks;<br>
> corpc_req_t *req = NULL;<br>
> <br>
> seed = in->seed;<br>
> <br>
> if (child_count == 0) {<br>
> __debug("i have no child (sum=%d)", sum);<br>
> goto out;<br>
> }<br>
> <br>
> __debug("bcasting sum to %d children:", child_count);<br>
> <br>
> for (i = 0; i < child_count; i++)<br>
> __debug("child[%d] = rank %d", i, child_ranks[i]);<br>
> <br>
> /* forward requests to children in the rpc tree */<br>
> req = calloc(child_count, sizeof(*req));<br>
> if (!req) {<br>
> __error("failed to allocate memory for corpc");<br>
> return ENOMEM;<br>
> }<br>
> <br>
> for (i = 0; i < child_count; i++) {<br>
> corpc_req_t *r = &req[i];<br>
> int child = child_ranks[i];<br>
> <br>
> ret = corpc_get_handle(rpcset.sum, child, r);<br>
> if (ret) {<br>
> __error("corpc_get_handle failed, abort rpc");<br>
> goto out;<br>
> }<br>
> <br>
> ret = corpc_forward_request((void *) in, r);<br>
> if (ret) {<br>
> __error("corpc_forward_request failed, abort rpc");<br>
> goto out;<br>
> }<br>
> }<br>
> <br>
> /* collect results */<br>
> for (i = 0; i < child_count; i++) {<br>
> metasim_sum_out_t _out;<br>
> corpc_req_t *r = &req[i];<br>
> <br>
> ret = corpc_wait_request(r);<br>
> if (ret) {<br>
> __error("corpc_wait_request failed, abort rpc");<br>
> goto out;<br>
> }<br>
> <br>
> margo_get_output(r->handle, &_out);<br>
> partial_sum = _out.sum;<br>
> sum += partial_sum;<br>
> <br>
> __debug("sum from child[%d] (rank=%d): %d (sum=%d)",<br>
> i, child_ranks[i], partial_sum, sum);<br>
> <br>
> margo_free_output(r->handle, &_out);<br>
> margo_destroy(r->handle);<br>
> }<br>
> <br>
> out:<br>
> sum += metasim->rank + seed;<br>
> out->sum = sum;<br>
> <br>
> return ret;<br>
> }<br>
> <br>
> static void metasim_rpc_handle_sum(hg_handle_t handle)<br>
> {<br>
> int ret = 0;<br>
> hg_return_t hret;<br>
> metasim_rpc_tree_t tree;<br>
> metasim_sum_in_t in;<br>
> metasim_sum_out_t out;<br>
> <br>
> __debug("sum rpc handler");<br>
> print_margo_handler_pool_info(metasim->mid);<br>
> <br>
> hret = margo_get_input(handle, &in);<br>
> if (hret != HG_SUCCESS) {<br>
> __error("margo_get_input failed");<br>
> return;<br>
> }<br>
> <br>
> metasim_rpc_tree_init(metasim->rank, metasim->nranks, in.root, 2, &tree);<br>
> <br>
> ret = sum_forward(&tree, &in, &out);<br>
> if (ret)<br>
> __error("sum_forward failed");<br>
> <br>
> metasim_rpc_tree_free(&tree);<br>
> margo_free_input(handle, &in);<br>
> <br>
> margo_respond(handle, &out);<br>
> <br>
> margo_destroy(handle);<br>
> }<br>
> DEFINE_MARGO_RPC_HANDLER(metasim_rpc_handle_sum)<br>
> <br>
> int metasim_rpc_invoke_sum(int32_t seed, int32_t *sum)<br>
> {<br>
> int ret = 0;<br>
> int32_t _sum = 0;<br>
> metasim_rpc_tree_t tree;<br>
> metasim_sum_in_t in;<br>
> metasim_sum_out_t out;<br>
> <br>
> ret = metasim_rpc_tree_init(metasim->rank, metasim->nranks, metasim->rank,<br>
> 2, &tree);<br>
> if (ret) {<br>
> __error("failed to initialize the rpc tree (ret=%d)", ret);<br>
> return ret;<br>
> }<br>
> <br>
> in.root = metasim->rank;<br>
> in.seed = seed;<br>
> <br>
> ret = sum_forward(&tree, &in, &out);<br>
> if (ret) {<br>
> __error("sum_forward failed (ret=%d)", ret);<br>
> } else {<br>
> _sum = out.sum;<br>
> __debug("rpc sum final result = %d", _sum);<br>
> <br>
> *sum = _sum;<br>
> }<br>
> <br>
> metasim_rpc_tree_free(&tree);<br>
> <br>
> return ret;<br>
> }<br>
> <br>
> ---<br>
> <br>
> This sum operation runs successfully with 512 nodes/ppn=1 (512 broadcasting), but fails/hangs with 512 nodes/ppn=2 (1024 broadcasting), on Summit. From the log messages, we see a different pattern in the argobot rpc handler pool. In the following plots, the
blue line shows the size of the handler pool, and the red line is the number of blocked elements, each time when the sum handler is invoked. This result is from one server (out of 512), but other servers show a similar pattern. In the failing case (2nd plot),
we see that all ults become blocked. When the server hangs, the blocked count was 255.<br>
> <br>
> <br>
> <logs_success (size and blocked).png><br>
> <br>
> <br>
> <logs_fail (size and blocked).png><br>
> <br>
> <br>
> I am wondering if our way of implementing the broadcasting operation is problematic. Or, is there any parameter that we need to tune for handling a large number of requests? The margo versions that we have test with are v0.4.3 and v0.5.1.<br>
> <br>
> We suspect that somehow a leaf node cannot respond to its parent due to some resource exhaustion, which results in a deadlock.<br>
> <br>
> The full test code is at <a href="https://code.ornl.gov/hyogi/metasim">https://code.ornl.gov/hyogi/metasim</a>, and the full log is at
<a href="https://code.ornl.gov/hyogi/metasim/-/tree/master/logs">https://code.ornl.gov/hyogi/metasim/-/tree/master/logs</a>.<br>
> <br>
> Any advice will be helpful. Thank you!<br>
> <br>
> Best,<br>
> Hyogi<br>
<br>
</div>
</span></font></div>
<pre>_______________________________________________</pre>
<pre>mochi-devel mailing list</pre>
<a href="mailto:mochi-devel@lists.mcs.anl.gov">
<pre>mochi-devel@lists.mcs.anl.gov</pre>
</a>
<pre><br></pre>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel">
<pre>https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel</pre>
</a>
<pre><br></pre>
<a href="https://www.mcs.anl.gov/research/projects/mochi">
<pre>https://www.mcs.anl.gov/research/projects/mochi</pre>
</a>
<pre><br></pre>
</blockquote>
</body>
</html>