<div dir="ltr"><font face="tahoma, sans-serif">Hi Shane,</font><div><font face="tahoma, sans-serif"><br></font></div><div><font face="tahoma, sans-serif">Thanks for information. That was most likely the case because python was</font></div><div><font face="tahoma, sans-serif">loading lots of python modules (.pyc) at runtime and I saw lots of read-only files</font></div><div><font face="tahoma, sans-serif">with <span style="color:rgb(0,0,0);white-space:nowrap">darshan-parser --file-list.</span></font></div><div><font face="tahoma, sans-serif"><br></font></div><div><font color="#000000" face="tahoma, sans-serif"><span style="white-space:nowrap">I was trying to exclude the directories under PYTHONPATH</span></font></div><div><font face="tahoma, sans-serif"><font color="#000000"><span style="white-space:nowrap">and I came across </span></font><span style="color:rgb(0,0,0);white-space:nowrap">DARSHAN_EXCLUDE_DIRS documented <a href="https://software.ecmwf.int/wiki/display/UDOC/How+to+use+Darshan+to+profile+IO#HowtouseDarshantoprofileIO-excludedir">here</a>.</span></font></div><div><font face="tahoma, sans-serif">But it doesn't work and I don't see any reference in the source code either.</font></div><div><font face="tahoma, sans-serif"><br></font></div><div><font face="tahoma, sans-serif">As quick workaround, in my local installation I hard-coded directory paths in darshan-runtime/lib/darshan-core.c.</font></div><div><font face="tahoma, sans-serif">I wonder what is the option to exclude or include specific directories.</font></div><div><font face="tahoma, sans-serif"><br></font></div><div><font face="tahoma, sans-serif">Regards,</font></div><div><font face="tahoma, sans-serif">Pramod</font></div><div><br></div><div><br><div class="gmail_extra"><div class="gmail_quote">On Thu, Apr 13, 2017 at 7:31 PM, Shane Snyder <span dir="ltr"><<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF">
Do you know how many files this application is creating? <br>
<br>
By default, Darshan should stop tracking new file records on a
specific process after either tracking more than 1,024 files for a
given module or if it has exhausted all of its allocated memory.
This is to keep its memory footprint bounded. The warning message
you get from darshan-job-summary seems to imply this is what's going
on, but if your application isn't opening that many files, maybe
something else is wrong?<br>
<br>
Thanks,<br>
--Shane<div><div class="gmail-m_-8713385937731976445h5"><br>
<br>
<div class="gmail-m_-8713385937731976445m_8146946484151609813moz-cite-prefix">On 04/09/2017 10:49 AM, pramod kumbhar
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="gmail-m_-8713385937731976445h5">
<div dir="ltr"><font face="arial, helvetica, sans-serif">It seems
like hdf5 library is not loaded into memory and hence dlsym()
is failing ?</font>
<div><font face="arial, helvetica, sans-serif">I tried
LD_PRELOAD-ing both darshan as well as hdf5 library and the
above example is working fine.</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">darshan-parser
--file-list show the hdf5 file and then I use <span style="color:rgb(0,0,0);white-space:nowrap">darshan-convert
--file to get profile of specific hdf5 that I am
interested in.</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0);white-space:nowrap"><br>
</span></font></div>
<div><font color="#000000" face="arial, helvetica, sans-serif"><span style="white-space:nowrap">But, while profiling actual
application, I have another issue. In the profile of large
application, </span></font><font face="arial, helvetica,
sans-serif">darshan-parser --file-list doesn't show the hdf5
created by application. What could be wrong?</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">The application
is using scipy, numpy, Image etc. Initially I thought
importing scipy modules somehow causing the issue but I am
not able to reproduce it with small isolated test.</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Could someone
provide any hints? (note that the job_name.darshan log file
is generated but pdf from </font><a href="http://darshan-job-summary.pl" target="_blank">darshan-job-summary.pl</a><font face="arial, helvetica, sans-serif"> sa<wbr>ys "This Darshan log
contains incomplete data which...").</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Thanks in
advance!</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">-Pramod</font></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Apr 9, 2017 at 1:02 PM,
pramod kumbhar <span dir="ltr"><<a href="mailto:pramod.s.kumbhar@gmail.com" target="_blank">pramod.s.kumbhar@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hello All,
<div><br>
</div>
<div>I have used darshan for parallel c/c++ application
without issue but trying to analyse python(3)
application first time. This application uses h5py.</div>
<div><br>
</div>
<div>I compiled darshan with --enable-HDF5-pre-1.10
(or --enable-HDF5-post-1.10).</div>
<div><br>
</div>
<div>When I now run simple h5py test serially I get :</div>
<div><br>
</div>
<div>
<div>$ python3 test.py</div>
<div>Darshan failed to map symbol: H5get_libversion</div>
</div>
<div><br>
</div>
<div>If I don't enable hdf5 during configure then I see
profiles being generated.</div>
<div>What am I missing here? Do I need parallel h5py
application? Any hints will be helpful!</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pramod</div>
<div><br>
</div>
<div>p.s. h5py test program</div>
<div><br>
</div>
<div>import h5py</div>
<div>a = [1,2,3,4,5]<br>
</div>
<div>h5f = h5py.File('data.h5', 'w')</div>
<div>h5f.create_dataset('dataset_1'<wbr>, data=a)</div>
<div>h5f.close()</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="gmail-m_-8713385937731976445m_8146946484151609813mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
Darshan-users mailing list
<a class="gmail-m_-8713385937731976445m_8146946484151609813moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a>
<a class="gmail-m_-8713385937731976445m_8146946484151609813moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a>
</pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a><br>
<br></blockquote></div><br></div></div></div>