<div dir="ltr"><div dir="ltr">I am quite late here but just to update this old thread:<div><br></div><div>The issue seems to be related to TaskProlog in the slurm.conf when HPE-MPI is used (even if the task prolog script is empty). For now, I workaround the issue by using native mpirun launcher from HPE-MPI (and avoiding all SLURM env variables).</div><div><br></div><div>-Pramod </div><pre class="gmail-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);font-size:12px"></pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 25, 2018 at 8:58 PM Carns, Philip H. <<a href="mailto:carns@mcs.anl.gov">carns@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div style="font-family:"Segoe UI",Frutiger,"Frutiger Linotype","Dejavu Sans","Helvetica Neue",Arial,sans-serif;font-size:14px">
<div class="gmail-m_2470649348877864817hiri-body-wrapper">
<div>That's interesting that the segfault comes from the prolog rather than your MPI executable.<br>
<br>
Is there a way in Slurm to pass an LD_PRELOAD environment variable that *only* affects the user executable and not the prolog?<br>
<br>
I'm not sure what's in the prolog, but it appears to be something that is incompatible with a Darshan library that's been built against HPE MPI. If there is an MPI program in there that would make sense; maybe there is a binary incompatibility.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="gmail-m_2470649348877864817hiri-extra">
<p>On 2018-07-21 05:25:08-04:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);margin:0px">
<div>
<div dir="ltr">Dear Kevin,
<div></div>
<div>Sorry for delay in response. I looked into docs but couldn't find anything specific. I tried:</div>
<div></div>
<div><font color="#0000ff">$ srun -n 2 --export=LD_PRELOAD=/some_path/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so,LD_LIBRARY_PATH=/opt/hpe/hpc/mpt/mpt-2.16/lib ./hello</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">srun: error: r2i0n34: task 1: Exited with exit code 255</font></div>
<div></div>
<div> And core files doesn't say anything useful, its generated by slurm prolog:</div>
<div></div>
<div>
<div><font color="#0000ff">warning: core file may not match specified executable file.</font></div>
<div><font color="#0000ff">[New LWP 72075]</font></div>
<div><font color="#0000ff">[New LWP 72080]</font></div>
<div><font color="#0000ff">Core was generated by `/bin/bash /etc/slurm/slurm.taskprolog'.</font></div>
<div><font color="#0000ff">Program terminated with signal 11, Segmentation fault.</font></div>
<div><font color="#0000ff">#0 0x00002aaaac1cc1e1 in ?? ()</font></div>
<div><font color="#0000ff">(gdb) bt</font></div>
<div><font color="#0000ff">Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fffffffddf8:</font></div>
</div>
<div></div>
<div><span style="background-color:rgb(243,243,243)">If I try to do :</span></div>
<div></div>
<div>
<div><font color="#0000ff">$ LD_PRELOAD=/gpfs/<a href="http://bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so" target="_blank">bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so</a>
ls</font></div>
<div>MPT ERROR: PMI2_Init</div>
</div>
<div></div>
<div>Let me know if you have any suggestion to debug this further.</div>
<div></div>
<div>Regards,</div>
<div>Pramod</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jul 5, 2018 at 5:54 PM, Harms, Kevin <span dir="ltr">
<<a href="mailto:harms@alcf.anl.gov" target="_blank">harms@alcf.anl.gov</a>></span> wrote:
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Pramod,<br>
<br>
are there ayn environment variables that can be set to print out what the error code is? Did you build libdarshan using the HP MPI?<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of pramod kumbhar <<a href="mailto:pramod.s.kumbhar@gmail.com" target="_blank">pramod.s.kumbhar@gmail.com</a>><br>
Sent: Wednesday, July 4, 2018 7:54:21 AM<br>
To: <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a><br>
Subject: [Darshan-users] About HPE MPI (MPT) support in Darshan (MPT ERROR: PMI2_Init)<br>
<br>
Dear All,<br>
<br>
I was trying to use Darshan (3.1.6) with HPE MPI (MPT)<<a rel="noreferrer" href="https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037728en_us&docLocale=en_US" target="_blank">https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037728en_us&docLocale=en_US</a>>
on our system and seeing below error:
<div class="gmail-m_2470649348877864817gmail-HOEnZb">
<div class="gmail-m_2470649348877864817gmail-h5"><br>
LD_PRELOAD=/gpfs/some_path/lib/libdarshan.so srun -n 1 /gpfs/some_another_path/bin/ior -a MPIIO -b 1G -t 4M -c -i 3<br>
MPT ERROR: PMI2_Init<br>
<br>
With other MPI (e.g. intel-mpi), everything is working fine.<br>
<br>
Do you have any suggestions / workaround? Please let me know and I could help to debug/test the issue.<br>
<br>
Regards,<br>
Pramod<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote></div>