<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
That's good to know- thanks for following up!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Phil<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> pramod kumbhar <pramod.s.kumbhar@gmail.com><br>
<b>Sent:</b> Saturday, August 10, 2019 6:08 AM<br>
<b>To:</b> Carns, Philip H. <carns@mcs.anl.gov><br>
<b>Cc:</b> Harms, Kevin <harms@alcf.anl.gov>; darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Darshan-users] About HPE MPI (MPT) support in Darshan (MPT ERROR: PMI2_Init)</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">I am quite late here but just to update this old thread:
<div><br>
</div>
<div>The issue seems to be related to TaskProlog in the slurm.conf when HPE-MPI is used (even if the task prolog script is empty). For now, I workaround the issue by using native mpirun launcher from HPE-MPI (and avoiding all SLURM env variables).</div>
<div><br>
</div>
<div>-Pramod </div>
<pre class="x_gmail-code-java" style="margin-top:0px; margin-bottom:0px; padding:0px; max-height:30em; overflow:auto; white-space:pre-wrap; word-wrap:normal; color:rgb(51,51,51); font-size:12px"></pre>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Wed, Jul 25, 2018 at 8:58 PM Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov">carns@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div style="font-family:"Segoe UI",Frutiger,"Frutiger Linotype","Dejavu Sans","Helvetica Neue",Arial,sans-serif; font-size:14px">
<div class="x_gmail-m_2470649348877864817hiri-body-wrapper">
<div>That's interesting that the segfault comes from the prolog rather than your MPI executable.<br>
<br>
Is there a way in Slurm to pass an LD_PRELOAD environment variable that *only* affects the user executable and not the prolog?<br>
<br>
I'm not sure what's in the prolog, but it appears to be something that is incompatible with a Darshan library that's been built against HPE MPI. If there is an MPI program in there that would make sense; maybe there is a binary incompatibility.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="x_gmail-m_2470649348877864817hiri-extra">
<p>On 2018-07-21 05:25:08-04:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); margin:0px">
<div>
<div dir="ltr">Dear Kevin,
<div></div>
<div>Sorry for delay in response. I looked into docs but couldn't find anything specific. I tried:</div>
<div></div>
<div><font color="#0000ff">$ srun -n 2 --export=LD_PRELOAD=/some_path/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so,LD_LIBRARY_PATH=/opt/hpe/hpc/mpt/mpt-2.16/lib ./hello</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">srun: error: r2i0n34: task 1: Exited with exit code 255</font></div>
<div></div>
<div> And core files doesn't say anything useful, its generated by slurm prolog:</div>
<div></div>
<div>
<div><font color="#0000ff">warning: core file may not match specified executable file.</font></div>
<div><font color="#0000ff">[New LWP 72075]</font></div>
<div><font color="#0000ff">[New LWP 72080]</font></div>
<div><font color="#0000ff">Core was generated by `/bin/bash /etc/slurm/slurm.taskprolog'.</font></div>
<div><font color="#0000ff">Program terminated with signal 11, Segmentation fault.</font></div>
<div><font color="#0000ff">#0 0x00002aaaac1cc1e1 in ?? ()</font></div>
<div><font color="#0000ff">(gdb) bt</font></div>
<div><font color="#0000ff">Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fffffffddf8:</font></div>
</div>
<div></div>
<div><span style="background-color:rgb(243,243,243)">If I try to do :</span></div>
<div></div>
<div>
<div><font color="#0000ff">$ LD_PRELOAD=/gpfs/<a href="http://bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so" target="_blank">bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so</a>
ls</font></div>
<div>MPT ERROR: PMI2_Init</div>
</div>
<div></div>
<div>Let me know if you have any suggestion to debug this further.</div>
<div></div>
<div>Regards,</div>
<div>Pramod</div>
<div>
<div class="x_gmail_extra">
<div class="x_gmail_quote">On Thu, Jul 5, 2018 at 5:54 PM, Harms, Kevin <span dir="ltr">
<<a href="mailto:harms@alcf.anl.gov" target="_blank">harms@alcf.anl.gov</a>></span> wrote:
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
Pramod,<br>
<br>
are there ayn environment variables that can be set to print out what the error code is? Did you build libdarshan using the HP MPI?<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of pramod kumbhar <<a href="mailto:pramod.s.kumbhar@gmail.com" target="_blank">pramod.s.kumbhar@gmail.com</a>><br>
Sent: Wednesday, July 4, 2018 7:54:21 AM<br>
To: <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a><br>
Subject: [Darshan-users] About HPE MPI (MPT) support in Darshan (MPT ERROR: PMI2_Init)<br>
<br>
Dear All,<br>
<br>
I was trying to use Darshan (3.1.6) with HPE MPI (MPT)<<a rel="noreferrer" href="https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037728en_us&docLocale=en_US" target="_blank">https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037728en_us&docLocale=en_US</a>>
on our system and seeing below error:
<div class="x_gmail-m_2470649348877864817gmail-HOEnZb">
<div class="x_gmail-m_2470649348877864817gmail-h5"><br>
LD_PRELOAD=/gpfs/some_path/lib/libdarshan.so srun -n 1 /gpfs/some_another_path/bin/ior -a MPIIO -b 1G -t 4M -c -i 3<br>
MPT ERROR: PMI2_Init<br>
<br>
With other MPI (e.g. intel-mpi), everything is working fine.<br>
<br>
Do you have any suggestions / workaround? Please let me know and I could help to debug/test the issue.<br>
<br>
Regards,<br>
Pramod<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>