<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Hyogi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Is this bug resulting from the same code snippet as you shared in your previous email, or is it more complicated? And you're only using 1 process-per-node, so 64 total processes, right?<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have some time early this week to get on Summit and try to reproduce this as well as the other error you mentioned related to PMIx. This one does appear to be some weird corruption of SSG state that should be investigated.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Sim, Hyogi <simh@ornl.gov><br>
<b>Sent:</b> Friday, September 18, 2020 6:20 PM<br>
<b>To:</b> Sim, Hyogi <simh@ornl.gov><br>
<b>Cc:</b> Latham, Robert J. <robl@mcs.anl.gov>; Snyder, Shane <ssnyder@mcs.anl.gov>; mochi-devel@lists.mcs.anl.gov <mochi-devel@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Mochi-devel] [EXTERNAL] [SSG] pmix initialization failure</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">BTW, today I encountered a segfault on summit, while running with 64 nodes. I do not think this is a deterministic behavior:<br>
<br>
===<br>
<br>
Core was generated by `/gpfs/alpine/proj-shared/stf008/hs2/metasim/summit/sum/progress.debug/sum-paral'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00002000000c1004 in swim_dping_ack_recv_ult (handle=0x231e5010)<br>
at ../src/swim-fd/swim-fd-ping.c:312<br>
312 ABT_rwlock_wrlock(group->swim_ctx->swim_lock);<br>
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.6.ppc64le libatomic-4.8.5-37.el7_6.ppc64le libgcc-4.8.5-37.el7_6.ppc64le libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.ppc64le libmlx4-41mlnx1-OFED.4.7.3.0.3.47329.ppc64le libmlx5-41mlnx1-OFED.4.7.0.3.3.47329.ppc64le
libnl3-3.2.28-4.el7.ppc64le librxe-41mlnx1-OFED.4.4.2.4.6.47329.ppc64le libstdc++-4.8.5-37.el7_6.ppc64le numactl-libs-2.0.9-7.el7.ppc64le<br>
(gdb) bt<br>
#0 0x00002000000c1004 in swim_dping_ack_recv_ult (handle=0x231e5010)<br>
at ../src/swim-fd/swim-fd-ping.c:312<br>
#1 0x00002000000c1460 in swim_dping_ack_recv_ult_wrapper (handle=0x231e5010)<br>
at ../src/swim-fd/swim-fd-ping.c:378<br>
#2 0x000020000065ac44 in ABTD_thread_func_wrapper_thread ()<br>
from /autofs/nccs-svm1_proj/csc300/mjbrim/spack.mjb/opt/spack/linux-rhel7-power8le/gcc-4.8.5/argobots-1.0rc2-2kq6htkm2ura7u3qj6wpr5xhqyns5ikp/lib/libabt.so.0<br>
#3 0x000020000065b7b8 in make_fcontext ()<br>
from /autofs/nccs-svm1_proj/csc300/mjbrim/spack.mjb/opt/spack/linux-rhel7-power8le/gcc-4.8.5/argobots-1.0rc2-2kq6htkm2ura7u3qj6wpr5xhqyns5ikp/lib/libabt.so.0<br>
(gdb) f 0<br>
#0 0x00002000000c1004 in swim_dping_ack_recv_ult (handle=0x231e5010)<br>
at ../src/swim-fd/swim-fd-ping.c:312<br>
312 ABT_rwlock_wrlock(group->swim_ctx->swim_lock);<br>
(gdb) p group<br>
$1 = (ssg_group_t *) 0x24978ca0<br>
(gdb) p group->swim_ctx<br>
$2 = (swim_context_t *) 0x0<br>
(gdb) set print pretty on<br>
(gdb) p group[0]<br>
$3 = {<br>
mid_state = 0x231eff40,<br>
name = 0x249792e0 "\360{\227$",<br>
view = {<br>
size = 33,<br>
member_map = 0x0,<br>
rank_array = 0x24979510<br>
},<br>
config = {<br>
swim_period_length_ms = 3000,<br>
swim_suspect_timeout_periods = 5,<br>
swim_subgroup_member_count = -1,<br>
ssg_credential = -1<br>
},<br>
dead_members = 0x0,<br>
swim_ctx = 0x0,<br>
update_cb = 0x10002034 <ssg_group_update_cb>,<br>
update_cb_dat = 0x0,<br>
lock = 0xf<br>
}<br>
<br>
<br>
<br>
> On Sep 16, 2020, at 11:25 AM, Sim, Hyogi <simh@ornl.gov> wrote:<br>
> <br>
> Thanks, Rob.<br>
> <br>
> I have just tested with ssg_group_create_mpi(), and it works fine (tested up to 1024 nodes, ppn=1). It seems like the problem is with pmix.<br>
> <br>
> Best,<br>
> Hyogi<br>
> <br>
> <br>
> <br>
>> On Sep 16, 2020, at 10:15 AM, Latham, Robert J. <robl@mcs.anl.gov> wrote:<br>
>> <br>
>> On Wed, 2020-09-16 at 14:09 +0000, Sim, Hyogi wrote:<br>
>>> Hi Shane,<br>
>>> <br>
>>> Your are correct that ssg_group_create_pmix() is failing. I heard<br>
>>> from other coworker that he avoided using pmix because he frequently<br>
>>> observed unreliable behavior from pmix. I am not sure which exactly<br>
>>> causes a problem for now.<br>
>>> <br>
>>> As for the application, it is like a service daemon and spawns<br>
>>> exactly one process per node. I can probably with test with other<br>
>>> group creation function (possible with MPI), then see if it still<br>
>>> fails or not.<br>
>> <br>
>> <br>
>> We've been investigating SSG behavior at larger scale. I added a<br>
>> pointer to your message to this SSG issue:<br>
>> <br>
>> <a href="https://xgitlab.cels.anl.gov/sds/ssg/-/issues/21">https://xgitlab.cels.anl.gov/sds/ssg/-/issues/21</a><br>
>> <br>
>> ==rob<br>
>> <br>
>>> <br>
>>>> On Sep 15, 2020, at 6:27 PM, Snyder, Shane <ssnyder@mcs.anl.gov><br>
>>>> wrote:<br>
>>>> <br>
>>>> Hi Hyogi,<br>
>>>> <br>
>>>> Thanks for the heads up!<br>
>>>> <br>
>>>> What exactly fails when you run the code snippet you shared? Is it<br>
>>>> the ssg_group_create_pmix() call? That would make the most sense<br>
>>>> but just making sure it's not something else like the Margo or SSG<br>
>>>> init calls. I don't see anything obviously wrong in your code.<br>
>>>> <br>
>>>> I'm currently investigating some other errors that result in<br>
>>>> sporadic hangs or crashes on Summit when using SSG to launch many<br>
>>>> processes on a node (e.g,. starting 64 SSG processes on one Summit<br>
>>>> node) -- maybe this is just another variation of that particular<br>
>>>> bug (though I'm using verbs and you're using tcp)? It might be<br>
>>>> helpful if you can confirm exactly how many processes and nodes it<br>
>>>> takes to trigger the problem so I can try to reproduce. Your<br>
>>>> original mail mentions 512-nodes, but not clear whether you're<br>
>>>> starting just a single process on each node or more than that, etc.<br>
>>>> It would be nice if we could simplify the reproducer so we don't<br>
>>>> need so many nodes to debug, if possible.<br>
>>>> <br>
>>>> --Shane<br>
>>>> From: mochi-devel <mochi-devel-bounces@lists.mcs.anl.gov> on behalf<br>
>>>> of Sim, Hyogi <simh@ornl.gov><br>
>>>> Sent: Tuesday, September 15, 2020 10:59 AM<br>
>>>> To: mochi-devel@lists.mcs.anl.gov <mochi-devel@lists.mcs.anl.gov><br>
>>>> Subject: [Mochi-devel] [SSG] pmix initialization failure<br>
>>>> <br>
>>>> Hi,<br>
>>>> <br>
>>>> I am initializing a SSG group using pmix on Summit@. The<br>
>>>> initialization works as expected, but only up to a certain number<br>
>>>> of compute nodes (~ 256 nodes). The group initialization seems<br>
>>>> always unsuccessful with 512+ nodes. Assuming that ssg itself has<br>
>>>> been tested in a larger scale, I am wondering if you see any<br>
>>>> obvious problems in my code below. <br>
>>>> <br>
>>>> For ssg_config and ssg_group_update_cb, I just copied directly from<br>
>>>> the mochi documentation (<br>
>>>> <a href="https://mochi.readthedocs.io/en/latest/ssg/05_create_pmix.html">https://mochi.readthedocs.io/en/latest/ssg/05_create_pmix.html</a>). I<br>
>>>> am using v0.4.1.<br>
>>>> <br>
>>>> ===<br>
>>>> <br>
>>>> static int comm_init(void)<br>
>>>> {<br>
>>>> int ret = 0;<br>
>>>> int i = 0;<br>
>>>> int rank = 0;<br>
>>>> int nranks = 0;<br>
>>>> pmix_proc_t proc;<br>
>>>> margo_instance_id mid;<br>
>>>> ssg_group_id_t gid;<br>
>>>> <br>
>>>> __debug("initializa the communication");<br>
>>>> <br>
>>>> ret = PMIx_Init(&proc, NULL, 0);<br>
>>>> if (ret != PMIX_SUCCESS) {<br>
>>>> __error("PMIx_Init failed: %s", PMIx_Error_string(ret));<br>
>>>> return ret;<br>
>>>> }<br>
>>>> <br>
>>>> __debug("pmix initialized");<br>
>>>> <br>
>>>> mid = margo_init("ofi+tcp://", MARGO_SERVER_MODE, 1, 4);<br>
>>>> if (mid == MARGO_INSTANCE_NULL) {<br>
>>>> __error("failed to initialize margo");<br>
>>>> return EIO;<br>
>>>> }<br>
>>>> <br>
>>>> __debug("margo initialized");<br>
>>>> <br>
>>>> ret = ssg_init();<br>
>>>> if (ret != SSG_SUCCESS) {<br>
>>>> __error("ssg_init() failed");<br>
>>>> return ret;<br>
>>>> }<br>
>>>> <br>
>>>> gid = ssg_group_create_pmix(mid, "servergroup", proc,<br>
>>>> &ssg_config, ssg_group_update_cb,<br>
>>>> NULL);<br>
>>>> if (gid == SSG_GROUP_ID_INVALID) {<br>
>>>> __error("ssg_group_create_pmix() failed");<br>
>>>> return ret;<br>
>>>> }<br>
>>>> <br>
>>>> rank = ssg_get_group_self_rank(gid);<br>
>>>> nranks = ssg_get_group_size(gid);<br>
>>>> <br>
>>>> __debug("ssg group (gid=%llu, rank=%d, nranks=%d)",<br>
>>>> (unsigned long long) gid, (int) rank, nranks);<br>
>>>> <br>
>>>> ssg_group_dump(gid);<br>
>>>> <br>
>>>> return 0;<br>
>>>> }<br>
>>>> <br>
>>>> ===<br>
>>>> <br>
>>>> Thanks,<br>
>>>> Hyogi<br>
>>>> _______________________________________________<br>
>>>> mochi-devel mailing list<br>
>>>> mochi-devel@lists.mcs.anl.gov<br>
>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel">https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel</a><br>
>>>> <a href="https://www.mcs.anl.gov/research/projects/mochi">https://www.mcs.anl.gov/research/projects/mochi</a><br>
>>> <br>
>>> _______________________________________________<br>
>>> mochi-devel mailing list<br>
>>> mochi-devel@lists.mcs.anl.gov<br>
>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel">https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel</a><br>
>>> <a href="https://www.mcs.anl.gov/research/projects/mochi">https://www.mcs.anl.gov/research/projects/mochi</a><br>
> <br>
<br>
</div>
</span></font></div>
</body>
</html>