<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Pramod,<br>
<br>
The DARSHAN_EXCLUDE_DIRS mechanism you found is not currently part
of the Darshan master branch -- the documentation there is for a
fork of Darshan being managed independently at ECMWF. I have asked
the developer who manages it to contribute it back upstream to
Darshan master so hopefully we can include it in our next release.
Being able to tell Darshan to filter out files in certain
directories at runtime seems pretty useful in general, so will be
nice to have that mechanism.<br>
<br>
In the mean time, your temporary workaround seems like the best bet.
I'm glad it wasn't too much trouble for you to figure out how to
manually hack Darshan to get this working. <br>
<br>
Thanks,<br>
--Shane<br>
<br>
<div class="moz-cite-prefix">On 04/18/2017 02:30 PM, pramod kumbhar
wrote:<br>
</div>
<blockquote
cite="mid:CAJ5+C=swidP0e+Vj2pf1TA2kM_Rn+UPLx4NHD98xJT3_W58z1g@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr"><font face="tahoma, sans-serif">Hi Shane,</font>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">Thanks for information.
That was most likely the case because python was</font></div>
<div><font face="tahoma, sans-serif">loading lots of python
modules (.pyc) at runtime and I saw lots of read-only files</font></div>
<div><font face="tahoma, sans-serif">with <span
style="color:rgb(0,0,0);white-space:nowrap">darshan-parser
--file-list.</span></font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif" color="#000000"><span
style="white-space:nowrap">I was trying to exclude the
directories under PYTHONPATH</span></font></div>
<div><font face="tahoma, sans-serif"><font color="#000000"><span
style="white-space:nowrap">and I came across </span></font><span
style="color:rgb(0,0,0);white-space:nowrap">DARSHAN_EXCLUDE_DIRS
documented <a moz-do-not-send="true"
href="https://software.ecmwf.int/wiki/display/UDOC/How+to+use+Darshan+to+profile+IO#HowtouseDarshantoprofileIO-excludedir">here</a>.</span></font></div>
<div><font face="tahoma, sans-serif">But it doesn't work and I
don't see any reference in the source code either.</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">As quick workaround, in my
local installation I hard-coded directory paths
in darshan-runtime/lib/darshan-core.c.</font></div>
<div><font face="tahoma, sans-serif">I wonder what is the option
to exclude or include specific directories.</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">Regards,</font></div>
<div><font face="tahoma, sans-serif">Pramod</font></div>
<div><br>
</div>
<div><br>
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Apr 13, 2017 at 7:31 PM,
Shane Snyder <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"> Do you know how many files this
application is creating? <br>
<br>
By default, Darshan should stop tracking new file
records on a specific process after either tracking
more than 1,024 files for a given module or if it has
exhausted all of its allocated memory. This is to keep
its memory footprint bounded. The warning message you
get from darshan-job-summary seems to imply this is
what's going on, but if your application isn't opening
that many files, maybe something else is wrong?<br>
<br>
Thanks,<br>
--Shane
<div>
<div class="gmail-m_-8713385937731976445h5"><br>
<br>
<div
class="gmail-m_-8713385937731976445m_8146946484151609813moz-cite-prefix">On
04/09/2017 10:49 AM, pramod kumbhar wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="gmail-m_-8713385937731976445h5">
<div dir="ltr"><font face="arial, helvetica,
sans-serif">It seems like hdf5 library is
not loaded into memory and hence dlsym() is
failing ?</font>
<div><font face="arial, helvetica, sans-serif">I
tried LD_PRELOAD-ing both darshan as well
as hdf5 library and the above example is
working fine.</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">darshan-parser
--file-list show the hdf5 file and then I
use <span
style="color:rgb(0,0,0);white-space:nowrap">darshan-convert
--file to get profile of specific hdf5
that I am interested in.</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span
style="color:rgb(0,0,0);white-space:nowrap"><br>
</span></font></div>
<div><font face="arial, helvetica, sans-serif"
color="#000000"><span
style="white-space:nowrap">But, while
profiling actual application, I have
another issue. In the profile of large
application, </span></font><font
face="arial, helvetica, sans-serif">darshan-parser
--file-list doesn't show the hdf5 created
by application. What could be wrong?</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">The
application is using scipy, numpy, Image
etc. Initially I thought importing scipy
modules somehow causing the issue but I am
not able to reproduce it with small
isolated test.</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Could
someone provide any hints? (note that the
job_name.darshan log file is generated but
pdf from </font><a moz-do-not-send="true"
href="http://darshan-job-summary.pl"
target="_blank">darshan-job-summary.pl</a><font
face="arial, helvetica, sans-serif"> sa<wbr>ys
"This Darshan log contains incomplete data
which...").</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Thanks
in advance!</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">-Pramod</font></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Apr 9, 2017
at 1:02 PM, pramod kumbhar <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:pramod.s.kumbhar@gmail.com" target="_blank">pramod.s.kumbhar@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hello All,
<div><br>
</div>
<div>I have used darshan for parallel
c/c++ application without issue but
trying to analyse python(3)
application first time. This
application uses h5py.</div>
<div><br>
</div>
<div>I compiled darshan
with --enable-HDF5-pre-1.10
(or --enable-HDF5-post-1.10).</div>
<div><br>
</div>
<div>When I now run simple h5py test
serially I get :</div>
<div><br>
</div>
<div>
<div>$ python3 test.py</div>
<div>Darshan failed to map symbol:
H5get_libversion</div>
</div>
<div><br>
</div>
<div>If I don't enable hdf5 during
configure then I see profiles being
generated.</div>
<div>What am I missing here? Do I need
parallel h5py application? Any hints
will be helpful!</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pramod</div>
<div><br>
</div>
<div>p.s. h5py test program</div>
<div><br>
</div>
<div>import h5py</div>
<div>a = [1,2,3,4,5]<br>
</div>
<div>h5f = h5py.File('data.h5', 'w')</div>
<div>h5f.create_dataset('dataset_1'<wbr>,
data=a)</div>
<div>h5f.close()</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset
class="gmail-m_-8713385937731976445m_8146946484151609813mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
Darshan-users mailing list
<a moz-do-not-send="true" class="gmail-m_-8713385937731976445m_8146946484151609813moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a>
<a moz-do-not-send="true" class="gmail-m_-8713385937731976445m_8146946484151609813moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
Darshan-users mailing list
<a moz-do-not-send="true" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a>
<a moz-do-not-send="true" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a>
</blockquote></div>
</div></div></div>
</blockquote>
</body></html>