[Darshan-users] h5py python application error : Darshan failed to map symbol: H5get_libversion

[pramod kumbhar]

Hi Shane,

Thanks for information. That was most likely the case because python was
loading lots of python modules (.pyc) at runtime and I saw lots of
read-only files
with darshan-parser --file-list.

I was trying to exclude the directories under PYTHONPATH
and I came across DARSHAN_EXCLUDE_DIRS documented here
<https://software.ecmwf.int/wiki/display/UDOC/How+to+use+Darshan+to+profile+IO#HowtouseDarshantoprofileIO-excludedir>
.
But it doesn't work and I don't see any reference in the source code either.

As quick workaround, in my local installation I hard-coded directory paths
in darshan-runtime/lib/darshan-core.c.
I wonder what is the option to exclude or include specific directories.

Regards,
Pramod


On Thu, Apr 13, 2017 at 7:31 PM, Shane Snyder <ssnyder at mcs.anl.gov> wrote:

> Do you know how many files this application is creating?
>
> By default, Darshan should stop tracking new file records on a specific
> process after either tracking more than 1,024 files for a given module or
> if it has exhausted all of its allocated memory. This is to keep its memory
> footprint bounded. The warning message you get from darshan-job-summary
> seems to imply this is what's going on, but if your application isn't
> opening that many files, maybe something else is wrong?
>
> Thanks,
> --Shane
>
>
> On 04/09/2017 10:49 AM, pramod kumbhar wrote:
>
> It seems like hdf5 library is not loaded into memory and hence dlsym() is
> failing ?
> I tried LD_PRELOAD-ing both darshan as well as hdf5 library and the above
> example is working fine.
>
> darshan-parser --file-list show the hdf5 file and then I use darshan-convert
> --file to get profile of specific hdf5 that I am interested in.
>
> But, while profiling actual application, I have another issue. In the
> profile of large application, darshan-parser --file-list doesn't show the
> hdf5 created by application. What could be wrong?
>
> The application is using scipy, numpy, Image etc. Initially I thought
> importing scipy modules somehow causing the issue but I am not able to
> reproduce it with small isolated test.
>
> Could someone provide any hints? (note that the job_name.darshan log file
> is generated but pdf from darshan-job-summary.pl says "This Darshan log
> contains incomplete data which...").
>
> Thanks in advance!
>
> -Pramod
>
> On Sun, Apr 9, 2017 at 1:02 PM, pramod kumbhar <pramod.s.kumbhar at gmail.com
> > wrote:
>
>> Hello All,
>>
>> I have used darshan for parallel c/c++ application without issue but
>> trying to analyse python(3) application first time. This application uses
>> h5py.
>>
>> I compiled darshan with --enable-HDF5-pre-1.10
>> (or --enable-HDF5-post-1.10).
>>
>> When I now run simple h5py test serially I get :
>>
>> $ python3 test.py
>> Darshan failed to map symbol: H5get_libversion
>>
>> If I don't enable hdf5 during configure then I see profiles being
>> generated.
>> What am I missing here? Do I need parallel h5py application? Any hints
>> will be helpful!
>>
>> Regards,
>> Pramod
>>
>> p.s. h5py test program
>>
>> import h5py
>> a = [1,2,3,4,5]
>> h5f = h5py.File('data.h5', 'w')
>> h5f.create_dataset('dataset_1', data=a)
>> h5f.close()
>>
>
>
>
> _______________________________________________
> Darshan-users mailing listDarshan-users at lists.mcs.anl.govhttps://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>
>
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20170418/f0d6bf53/attachment.html>