<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-body-wrapper" contenteditable="true">
<div>That's interesting that the segfault comes from the prolog rather than your MPI executable.<br>
<br>
Is there a way in Slurm to pass an LD_PRELOAD environment variable that *only* affects the user executable and not the prolog?<br>
<br>
I'm not sure what's in the prolog, but it appears to be something that is incompatible with a Darshan library that's been built against HPE MPI. If there is an MPI program in there that would make sense; maybe there is a binary incompatibility.<br>
<br>
thanks,<br>
-Phil<br>
</div>
</div>
<div class="hiri-extra" contenteditable="false">
<p>On 2018-07-21 05:25:08-04:00 Darshan-users wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div dir="ltr">Dear Kevin,
<div></div>
<div>Sorry for delay in response. I looked into docs but couldn't find anything specific. I tried:</div>
<div></div>
<div><font color="#0000ff">$ srun -n 2 --export=LD_PRELOAD=/some_path/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so,LD_LIBRARY_PATH=/opt/hpe/hpc/mpt/mpt-2.16/lib ./hello</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">MPT ERROR: PMI2_Init</font></div>
<div><font color="#0000ff">srun: error: r2i0n34: task 1: Exited with exit code 255</font></div>
<div></div>
<div> And core files doesn't say anything useful, its generated by slurm prolog:</div>
<div></div>
<div>
<div><font color="#0000ff">warning: core file may not match specified executable file.</font></div>
<div><font color="#0000ff">[New LWP 72075]</font></div>
<div><font color="#0000ff">[New LWP 72080]</font></div>
<div><font color="#0000ff">Core was generated by `/bin/bash /etc/slurm/slurm.taskprolog'.</font></div>
<div><font color="#0000ff">Program terminated with signal 11, Segmentation fault.</font></div>
<div><font color="#0000ff">#0 0x00002aaaac1cc1e1 in ?? ()</font></div>
<div><font color="#0000ff">(gdb) bt</font></div>
<div><font color="#0000ff">Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fffffffddf8:</font></div>
</div>
<div></div>
<div><span style="background-color:rgb(243,243,243)">If I try to do :</span></div>
<div></div>
<div>
<div><font color="#0000ff">$ LD_PRELOAD=/gpfs/<a href="http://bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so">bbp.cscs.ch/data/project/proj16/kumbhar/soft/MPI_COMPARE/HPE_MPI/install/linux-rhel7-x86_64/intel-18.0.1/darshan-runtime-3.1.6-ryds66/lib/libdarshan.so</a>
ls</font></div>
<div>MPT ERROR: PMI2_Init</div>
</div>
<div></div>
<div>Let me know if you have any suggestion to debug this further.</div>
<div></div>
<div>Regards,</div>
<div>Pramod</div>
<div>
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jul 5, 2018 at 5:54 PM, Harms, Kevin <span dir="ltr">
<<a target="_blank" href="mailto:harms@alcf.anl.gov">harms@alcf.anl.gov</a>></span> wrote:
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Pramod,<br>
<br>
are there ayn environment variables that can be set to print out what the error code is? Did you build libdarshan using the HP MPI?<br>
<br>
kevin<br>
<br>
______________________________<wbr>__________<br>
From: Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov">darshan-users-bounces@lists.<wbr>mcs.anl.gov</a>> on behalf of pramod kumbhar <<a href="mailto:pramod.s.kumbhar@gmail.com">pramod.s.kumbhar@gmail.com</a>><br>
Sent: Wednesday, July 4, 2018 7:54:21 AM<br>
To: <a href="mailto:darshan-users@lists.mcs.anl.gov">darshan-users@lists.mcs.anl.<wbr>gov</a><br>
Subject: [Darshan-users] About HPE MPI (MPT) support in Darshan (MPT ERROR: PMI2_Init)<br>
<br>
Dear All,<br>
<br>
I was trying to use Darshan (3.1.6) with HPE MPI (MPT)<<a rel="noreferrer" target="_blank" href="https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00037728en_us&docLocale=en_US">https://support.hpe.com/<wbr>hpsc/doc/public/display?docId=<wbr>emr_na-a00037728en_us&<wbr>docLocale=en_US</a>>
on our system and seeing below error:
<div class="gmail-HOEnZb">
<div class="gmail-h5"><br>
LD_PRELOAD=/gpfs/some_path/<wbr>lib/libdarshan.so srun -n 1 /gpfs/some_another_path/bin/<wbr>ior -a MPIIO -b 1G -t 4M -c -i 3<br>
MPT ERROR: PMI2_Init<br>
<br>
With other MPI (e.g. intel-mpi), everything is working fine.<br>
<br>
Do you have any suggestions / workaround? Please let me know and I could help to debug/test the issue.<br>
<br>
Regards,<br>
Pramod<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>