<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi Abid,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
It still sounds like your libfabric build does not include the verbs and rxm providers. If you want to confirm this, then load the libfabric package and (on a compute node, within a job script) run "fi_info" and save the output.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The output should include information about "verbs" and "rxm" interfaces in it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
If it does not, then you should uninstall libfabric and all Mochi packages that depend on it ("spack uninstall --all --dependents libfabric"), place the packages.yaml that Rob Latham provided in ~/.spack/linux/, install your software again, and repeat the above
experiment.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The default libfabric build will not include the network support that you need for Summit; you really need to use Rob's packages.yaml not only to get the right transports but to have it use the right verbs library that's provided in the Summit environment.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
thanks!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
-Phil<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Abid Malik <abidmuslim@gmail.com><br>
<b>Sent:</b> Tuesday, November 12, 2019 7:53 AM<br>
<b>To:</b> Carns, Philip H. <carns@mcs.anl.gov><br>
<b>Cc:</b> Dorier, Matthieu <mdorier@anl.gov>; Latham, Robert J. via mochi-devel <mochi-devel@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Mochi-devel] running mochi examples</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div class="x_gmail_default" style="font-family:courier new,monospace; font-size:small">
Thank Mathieu and Philip,</div>
<div class="x_gmail_default" style="font-family:courier new,monospace; font-size:small">
<br>
</div>
<div class="x_gmail_default" style="font-family:courier new,monospace; font-size:small">
The messages are the same as appeared on the issue. Please let me know if you want me to build Mochi or run some examples in a way that can help to solve the problem.</div>
<div class="x_gmail_default" style="font-family:courier new,monospace; font-size:small">
<br>
</div>
<div class="x_gmail_default" style="font-family:courier new,monospace; font-size:small">
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Nov 12, 2019 at 7:46 AM Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov">carns@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Abid followed up some more off-list; I think his code building now but he still has some run time problems to work through.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
I have a quick follow up on one of the tangents in this discussion; the issue that was preventing "verbs://" from working correctly as the transport name has been fixed (you will need to uninstall and then re-install the mercury package to pick this up):</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<a href="https://github.com/mercury-hpc/mercury/issues/322" target="_blank">https://github.com/mercury-hpc/mercury/issues/322</a></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
That wasn't the core problem here, but something confusing that we hit along the way.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
-Phil<br>
</div>
<div id="x_gmail-m_-8314707741128562759appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-8314707741128562759divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Dorier, Matthieu <<a href="mailto:mdorier@anl.gov" target="_blank">mdorier@anl.gov</a>><br>
<b>Sent:</b> Sunday, November 10, 2019 3:06 PM<br>
<b>To:</b> Abid Malik <<a href="mailto:abidmuslim@gmail.com" target="_blank">abidmuslim@gmail.com</a>><br>
<b>Cc:</b> Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>>; Latham, Robert J. via mochi-devel <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [Mochi-devel] running mochi examples</font>
<div> </div>
</div>
<div lang="EN-GB">
<div>
<p><span lang="EN-US">Hi Abid,</span></p>
<p><span lang="EN-US"> </span></p>
<p><span lang="EN-US">Sorry for the late response, I’m on vacations with limited access to my computer.</span></p>
<p><span lang="EN-US">Let’s try to figure out what changed since your earlier messages about NA errors. Which libraries were rebuilt?</span></p>
<p><span lang="EN-US"> </span></p>
<p><span lang="EN-US">Thanks,</span></p>
<p><span lang="EN-US"> </span></p>
<p><span lang="EN-US">Matthieu</span></p>
<p><span lang="EN-US"> </span></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0cm 0cm">
<p><b><span style="font-size:12pt; color:black">From: </span></b><span style="font-size:12pt; color:black">Abid Malik <<a href="mailto:abidmuslim@gmail.com" target="_blank">abidmuslim@gmail.com</a>><br>
<b>Date: </b>Wednesday, 6 November 2019 at 17:56<br>
<b>To: </b>"Dorier, Matthieu" <<a href="mailto:mdorier@anl.gov" target="_blank">mdorier@anl.gov</a>><br>
<b>Cc: </b>"Carns, Philip H." <<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>>, "Latham, Robert J. via mochi-devel" <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [Mochi-devel] running mochi examples</span></p>
</div>
<div>
<p> </p>
</div>
<div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">Hi Matthieu,</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">For simple cases, I am using </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">gcc test.server.c -o server $(pkg-config --libs margo)</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">I pretty much tried everything. I just installed libfabric manually. I am not sure what to try next.</span></p>
</div>
</div>
<p> </p>
<div>
<div>
<p>On Wed, Nov 6, 2019 at 12:07 PM Dorier, Matthieu <<a href="mailto:mdorier@anl.gov" target="_blank">mdorier@anl.gov</a>> wrote:</p>
</div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0cm 0cm 0cm 6pt; margin-left:4.8pt; margin-right:0cm">
<div>
<div>
<p><span lang="EN-US">Hi Abid, </span></p>
<p><span lang="EN-US"> </span></p>
<p><span lang="EN-US">At this top of my head this could be a problem with the order in which libraries are linked. Could you send your CMakeLists.txt file?</span></p>
<p><span lang="EN-US">Thanks,</span></p>
<p><span lang="EN-US"> </span></p>
<p><span lang="EN-US">Matthieu</span></p>
<p><span lang="EN-US"> </span></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0cm 0cm">
<p><b><span style="font-size:12pt; color:black">From: </span></b><span style="font-size:12pt; color:black">mochi-devel <<a href="mailto:mochi-devel-bounces@lists.mcs.anl.gov" target="_blank">mochi-devel-bounces@lists.mcs.anl.gov</a>> on behalf of Abid Malik
via mochi-devel <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>><br>
<b>Reply to: </b>Abid Malik <<a href="mailto:abidmuslim@gmail.com" target="_blank">abidmuslim@gmail.com</a>><br>
<b>Date: </b>Tuesday, 5 November 2019 at 00:01<br>
<b>To: </b>"Carns, Philip H." <<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>><br>
<b>Cc: </b>"Latham, Robert J. via mochi-devel" <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [Mochi-devel] running mochi examples</span></p>
</div>
<div>
<p> </p>
</div>
<div>
<div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
</div>
<div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">Thanks.</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">I am getting the following linking error:</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<p>Linking C executable 01_margo_hello_server<br>
/usr/bin/ld: CMakeFiles/01_margo_hello_server.dir/hello_server.c.o: undefined reference to symbol 'ABT_thread_create'<br>
/gpfs/alpine/csc299/world-shared/amalik/mochi/spack/opt/spack/linux-rhel7-power9le/gcc-6.4.0/argobots-develop-uwvdjjdzrkn475ax7bw4w7hztawzidg3/lib/libabt.so.0: error adding symbols: DSO missing from command line<br>
/usr/bin/sha1sum: 01_margo_hello_server: No such file or directory<br>
collect2: error: ld returned 1 exit status<br>
make[2]: *** [01_margo_hello_server] Error 1<br>
make[1]: *** [CMakeFiles/01_margo_hello_server.dir/all] Error 2</p>
<div>
<p><span style="font-size:12pt; font-family:Arial,sans-serif">make: *** [all] Error 2</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">currently, I have the following:</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">Currently Loaded Modules:<br>
1) xl/16.1.1-3 6) darshan-runtime/3.1.7 11) boost-1.70.0-gcc-6.4.0-dw376xg<br>
2) spectrum-mpi/10.3.0.1-20190611 7) DefApps 12) libfabric-1.8.1-gcc-6.4.0-d4kxa4p<br>
3) hsi/5.0.2.p5 8) argobots-develop-gcc-6.4.0-uwvdjjd 13) mercury-master-gcc-6.4.0-nbp34rm<br>
4) xalt/1.1.4 9) bzip2-1.0.8-gcc-6.4.0-mdsmice 14) margo-0.5.2-gcc-6.4.0-gniufvt<br>
5) lsf-tools/2.0 10) zlib-1.2.11-gcc-6.4.0-nbedgs2 15) cmake/3.15.2</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">Argobot is there. Am I missing something?</span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New""> </span></p>
</div>
<div>
<p><span style="font-size:12pt; font-family:"Courier New"">Abid</span></p>
</div>
<p> </p>
</div>
<div>
<p> </p>
</div>
<p> </p>
<div>
<div>
<p>On Mon, Nov 4, 2019 at 5:10 PM Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>> wrote:</p>
</div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0cm 0cm 0cm 6pt; margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p style="background:white"><span style="font-size:12pt; color:black">For the transport it should work Ok to just use "verbs://" now (the trailing colon and slashes are important, though). Mercury will automatically convert that to the more verbose string
internally.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12pt; color:black"> </span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12pt; color:black">thanks,</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12pt; color:black">-Phil</span></p>
</div>
<div align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_gmail-m_-8314707741128562759x_gmail-m_4718529552322078054gmail-m_8051748058728153858divRplyFwdMsg">
<p><b><span style="color:black">From:</span></b><span style="color:black"> mochi-devel <<a href="mailto:mochi-devel-bounces@lists.mcs.anl.gov" target="_blank">mochi-devel-bounces@lists.mcs.anl.gov</a>> on behalf of Latham, Robert J. via mochi-devel <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>><br>
<b>Sent:</b> Monday, November 4, 2019 3:11 PM<br>
<b>To:</b> <a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a> <<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a>>;
<a href="mailto:abidmuslim@gmail.com" target="_blank">abidmuslim@gmail.com</a> <<a href="mailto:abidmuslim@gmail.com" target="_blank">abidmuslim@gmail.com</a>><br>
<b>Subject:</b> Re: [Mochi-devel] running mochi examples</span> </p>
<div>
<p> </p>
</div>
</div>
<div>
<div>
<p>On Mon, 2019-11-04 at 07:37 -0500, Abid Malik via mochi-devel wrote:<br>
> <br>
> Hello,<br>
> <br>
> I am trying to play with the mochi examples at <br>
> <br>
> <a href="https://xgitlab.cels.anl.gov/sds/sds-examples.git" target="_blank">https://xgitlab.cels.anl.gov/sds/sds-examples.git</a><br>
> <br>
> I am trying the example on summit:<br>
> <br>
> /mercury/01_hello/01_hg_hello_server<br>
> <br>
> How should I start the server/client? I am trying it on login2 node<br>
> and trying to initiate it like:<br>
> <br>
> ./01_hg_hello_server bmi+tcp://login2:1234<br>
<br>
I see you are using spack. If so, I think you would have had to go out<br>
of your way to request the 'bmi' transport. You probably built<br>
libfabric. You can find out for sure with 'spack find'<br>
<br>
You tell margo (mercury, really) which protocol to use but the string<br>
for "libfabric using the verbs provider" is a little odd:<br>
<br>
"ofi+verbs;ofi_rxm://"<br>
<br>
> and getting errors.<br>
> <br>
> # NA -- Error -- /tmp/amalik/spack-stage/spack-stage-mercury-master-<br>
> nbp34rmg6rkc2ayiqxxxuupamvmdusr7/spack-src/src/na/na.c:281<br>
> # NA_Initialize_opt(): Specified class name does not support<br>
> requested protocol<br>
<br>
This says mercury did not recognize 'bmi+tcp'. You'd get the same<br>
error if you requested "moon_rock".<br>
<br>
I don't have a lot of guidance to offer you (yet). I just got started<br>
on summit a few days ago. When I try to take the advice I gave you I<br>
get an error, too:<br>
<br>
$ jsrun -r1 -n1 ./margo-example-server verbs<br>
# NA -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-master-<br>
c6nwqortoqjsyviyinlg5t6sfay6wv7l/spack-src/src/na/na.c:279<br>
# NA_Initialize_opt(): Specified class name does not support requested<br>
protocol<br>
<br>
That's what we expect. margo/mercury don't know what 'verbs'<br>
is. Instead, we need to tell margo/mercury to use OFI (libfabric):<br>
<br>
$ jsrun -r1 -n1 ./margo-example-server "ofi+verbs;ofi_rxm://"<br>
# NA -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-master-<br>
c6nwqortoqjsyviyinlg5t6sfay6wv7l/spack-src/src/na/na_ofi.c:1406<br>
# na_ofi_getinfo(): fi_getinfo() failed, rc: -61(No data available)<br>
<br>
Uh, I'm still working on that one...<br>
<br>
==rob<br>
<br>
_______________________________________________<br>
mochi-devel mailing list<br>
<a href="mailto:mochi-devel@lists.mcs.anl.gov" target="_blank">mochi-devel@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mochi-devel</a><br>
<a href="https://www.mcs.anl.gov/research/projects/mochi" target="_blank">https://www.mcs.anl.gov/research/projects/mochi</a></p>
</div>
</div>
</div>
</blockquote>
</div>
<p><br clear="all">
</p>
<div>
<p> </p>
</div>
<p>-- </p>
<div>
<div>
<div>
<p>Abid M. Malik<br>
******************************************************<br>
"I have learned silence from the talkative, toleration from the intolerant, and kindness from the unkind"---Gibran<br>
"Success is not for the chosen few, but for the few who choose" --- John Maxwell<br>
"Being a good person does not depend on your religion or status in life, your race or skin color, political views or culture. IT DEPENDS ON HOW GOOD YOU TREAT OTHERS"--- Abid<br>
"The Universe is talking to us, and the language of the Universe is mathematics."----Abid<br>
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p><br clear="all">
</p>
<div>
<p> </p>
</div>
<p>-- </p>
<div>
<div>
<div>
<p>Abid M. Malik<br>
******************************************************<br>
"I have learned silence from the talkative, toleration from the intolerant, and kindness from the unkind"---Gibran<br>
"Success is not for the chosen few, but for the few who choose" --- John Maxwell<br>
"Being a good person does not depend on your religion or status in life, your race or skin color, political views or culture. IT DEPENDS ON HOW GOOD YOU TREAT OTHERS"--- Abid<br>
"The Universe is talking to us, and the language of the Universe is mathematics."----Abid<br>
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>Abid M. Malik<br>
******************************************************<br>
"I have learned silence from the talkative, toleration from the intolerant, and kindness from the unkind"---Gibran<br>
"Success is not for the chosen few, but for the few who choose" --- John Maxwell<br>
"Being a good person does not depend on your religion or status in life, your race or skin color, political views or culture. IT DEPENDS ON HOW GOOD YOU TREAT OTHERS"--- Abid<br>
"The Universe is talking to us, and the language of the Universe is mathematics."----Abid<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>