[Darshan-users] Darshan-users Digest, Vol 47, Issue 3
Adnan Haider
ahaider3 at hawk.iit.edu
Wed Jul 8 17:27:49 CDT 2015
Hi Phil,
HDF5 is being called by the netcdf library; our application does not call
it directly. I did remove hdf5-stubs and hdf5 from the makefile. However, I
get the same error. I did this rather quick so I may have done it
incorrectly. I will go over in detail and let you know. Thanks for the help.
Thanks, Adnan Haider
B.S Candidate, Computer Science
Illinois Institute of Technology
On Wed, Jul 8, 2015 at 8:05 AM, <darshan-users-request at lists.mcs.anl.gov>
wrote:
> Send Darshan-users mailing list submissions to
> darshan-users at lists.mcs.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
> or, via email, send a message with subject or body 'help' to
> darshan-users-request at lists.mcs.anl.gov
>
> You can reach the person managing the list at
> darshan-users-owner at lists.mcs.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Darshan-users digest..."
>
>
> Today's Topics:
>
> 1. Re: Darshan-users Digest, Vol 47, Issue 1 (Phil Carns)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 8 Jul 2015 11:05:27 -0400
> From: Phil Carns <carns at mcs.anl.gov>
> To: <darshan-users at lists.mcs.anl.gov>
> Subject: Re: [Darshan-users] Darshan-users Digest, Vol 47, Issue 1
> Message-ID: <559D3C37.1090208 at mcs.anl.gov>
> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>
> Is HDF5 being called directly, or is it an indirect dependency of the
> netcdf library in this case?
>
> Thanks, and sorry for the slow response times this week.
>
> If you want to get off the ground with something, you could probably
> modify the darhan-runtime makefile to omit the hdf5 and hdf5-stubs
> objects from libdarshan.so. They aren't strictly required (and won't
> tell you much except that HDF was used for a given file). That may get
> you to a point where you can observe the normal POSIX and MPIIO counters
> for now.
>
> thanks,
> -Phil
>
> On 07/06/2015 06:23 PM, Adnan Haider wrote:
> > Hey Phil,
> > After exclusively using LD_PRELOAD, i get the following error:
> > Darshan failed to map symbol: H5FOpen
> > Darshan failed to map symbol: H5FOpen
> > Darshan failed to map symbol: H5FOpen
> > Darshan failed to map symbol: H5FOpen
> >
> > Also you are correct; our application during runtime loads mpi4py
> > which contains the MPI_Init and MPI_Finalize calls.
> >
> > Thanks, Adnan Haider
> > B.S Candidate, Computer Science
> > Illinois Institute of Technology
> >
> > On Thu, Jul 2, 2015 at 10:00 AM,
> > <darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>> wrote:
> >
> > Send Darshan-users mailing list submissions to
> > darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
> > <https://lists.mcs.anl.gov/mailman/listinfo/darshan-users>
> > or, via email, send a message with subject or body 'help' to
> > darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>
> >
> > You can reach the person managing the list at
> > darshan-users-owner at lists.mcs.anl.gov
> > <mailto:darshan-users-owner at lists.mcs.anl.gov>
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Darshan-users digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Re: Darshan-users Digest, Vol 45, Issue 6 (Phil Carns)
> >
> >
> >
> ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 2 Jul 2015 09:40:10 -0400
> > From: Phil Carns <carns at mcs.anl.gov <mailto:carns at mcs.anl.gov>>
> > To: <darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>>
> > Subject: Re: [Darshan-users] Darshan-users Digest, Vol 45, Issue 6
> > Message-ID: <55953F3A.9020005 at mcs.anl.gov
> > <mailto:55953F3A.9020005 at mcs.anl.gov>>
> > Content-Type: text/plain; charset="windows-1252"; Format="flowed"
> >
> > Hi Adnan,
> >
> > Python indeed might be a little trickier, but I think its possible.
> > I'll see if I can find more information about this and report back.
> >
> > I *think* that you need to actually do the reverse and disable the
> > static instrumentation and use the dynamic LD_PRELOAD approach
> instead
> > in this case. I imagine that the Python MPI bindings are dynamically
> > loading the MPI library at runtime even though the Python program
> > itself
> > doesn't show up as a dynamic executable.
> >
> > The critical point in getting a Darshan log to be generated is that
> it
> > must intercept MPI_Init() and MPI_Finalize(). If it can achieve that
> > much then it should at least produce a log file.
> >
> > Can you try that method again (disabling the static
> > instrumentation and
> > instead using LD_PRELOAD exclusively) and confirm the error?
> >
> > thanks,
> > -Phil
> >
> > On 06/29/2015 02:47 PM, Adnan Haider wrote:
> > > Hi Phil,
> > >
> > > Sorry for the late reply, we were focusing on some other aspects
> but
> > > now we are back to using Darshan for analyzing our application. Our
> > > executable is statically linked; the previous error was caused by
> > > LD_PRELOAD being set in our environment. After resetting LD_PRELOAD
> > > (so that we are not dynamically instrumenting), the application
> runs
> > > and there are no symbol mapping errors. However after job
> > completion,
> > > no trace file appears in the log directory. We used environment
> > > variables with the latest MPICH release in an attempt to statically
> > > instrument our application. Compiling a simple mpi application with
> > > the environment variables set does produce a Darshan trace file,
> but
> > > our application does not. Also for our executable, using ldd
> <python
> > > script> we get: "not a dynamic executable"
> > >
> > >
> > > Here is some more detail about our application and its
> dependencies.
> > > Our application is written in python. The application uses
> > mpi4py for
> > > all MPI calls. Also we are reading and writing netcdf files but are
> > > using PyNIO which breaks down python calls into standard netcdf
> I/O.
> > > This site <http://www.alcf.anl.gov/user-guides/darshan> says that
> > > dealing with languages other than c,c++,and Fortran could cause
> > > issues. I was thinking the python code could be causing the issue.
> > > Possibly, the PyNIO package on top of netcdf or mpi4py are not
> being
> > > detected by Darshan. Or more likely, I am not correctly
> > compiling the
> > > correct software with the environment variables. The only software
> > > that directly uses the mpi compiler is mpi4py, and I did recompile
> > > this after setting the environment variables. I also recompiled
> > PyNIO
> > > and our application for good measure.
> > >
> > >
> > > Thanks, Adnan Haider
> > > B.S Candidate, Computer Science
> > > Illinois Institute of Technology
> > >
> > > On Fri, May 29, 2015 at 10:00 AM,
> > > <darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>
> > > <mailto:darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>>> wrote:
> > >
> > > Send Darshan-users mailing list submissions to
> > > darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>
> > > <mailto:darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>>
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
> > > <https://lists.mcs.anl.gov/mailman/listinfo/darshan-users>
> > > or, via email, send a message with subject or body 'help' to
> > > darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>
> > > <mailto:darshan-users-request at lists.mcs.anl.gov
> > <mailto:darshan-users-request at lists.mcs.anl.gov>>
> > >
> > > You can reach the person managing the list at
> > > darshan-users-owner at lists.mcs.anl.gov
> > <mailto:darshan-users-owner at lists.mcs.anl.gov>
> > > <mailto:darshan-users-owner at lists.mcs.anl.gov
> > <mailto:darshan-users-owner at lists.mcs.anl.gov>>
> > >
> > > When replying, please edit your Subject line so it is more
> > specific
> > > than "Re: Contents of Darshan-users digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > > 1. Darshan error with HDF5 (Adnan Haider)
> > > 2. Re: Darshan error with HDF5 (Phil Carns)
> > >
> > >
> > >
> >
> ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Thu, 28 May 2015 14:20:07 -0600
> > > From: Adnan Haider <ahaider3 at hawk.iit.edu
> > <mailto:ahaider3 at hawk.iit.edu>
> > > <mailto:ahaider3 at hawk.iit.edu <mailto:ahaider3 at hawk.iit.edu>>>
> > > To: darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>
> > > <mailto:darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>>
> > > Subject: [Darshan-users] Darshan error with HDF5
> > > Message-ID:
> > >
> > >
> > <CAM2fH+yEoqtPOnxVPvZNm853kCXEmhWQMrDpWWxZ8UEWC8jUcA at mail.gmail.com
> > <mailto:
> CAM2fH%2ByEoqtPOnxVPvZNm853kCXEmhWQMrDpWWxZ8UEWC8jUcA at mail.gmail.com>
> > <mailto:
> CAM2fH%2ByEoqtPOnxVPvZNm853kCXEmhWQMrDpWWxZ8UEWC8jUcA at mail.gmail.com
> > <mailto:
> CAM2fH%252ByEoqtPOnxVPvZNm853kCXEmhWQMrDpWWxZ8UEWC8jUcA at mail.gmail.com>>>
> > > Content-Type: text/plain; charset="utf-8"
> > >
> > > Hi,
> > > I am hoping to use Darshan for analyzing our application's
> > > performance. We
> > > are using HDF5 and netcdf 4. I installed Darshan version
> > 2.3.1 and
> > > mvapich2
> > > for our mpi implementation. Darshan works fine for a
> > standard mpi
> > > pingpong
> > > application. When we run our application, however, it gives the
> > > following
> > > error:
> > > Darshan failed to map symbol: H5Fopen
> > > Darshan failed to map symbol: H5Fopen
> > > Darshan failed to map symbol: H5Fopen
> > > Darshan failed to map symbol: H5Fopen
> > >
> > > The four instances of the error probably have to do with the
> > 4 mpi
> > > ranks
> > > specified at runtime. Nothing is written into the log
> directory.
> > > This is
> > > the only output, unless there is some error file I am not aware
> > > of. I used
> > > the static instrumentation approach for enabling Darshan.
> Thanks
> > > for the
> > > help!
> > >
> > > Thanks, Adnan Haider
> > > B.S Candidate, Computer Science
> > > Illinois Institute of Technology
> > > -------------- next part --------------
> > > An HTML attachment was scrubbed...
> > > URL:
> > >
> > <
> http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20150528/5ebdf005/attachment-0001.html
> >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 28 May 2015 16:32:04 -0400
> > > From: Phil Carns <carns at mcs.anl.gov
> > <mailto:carns at mcs.anl.gov> <mailto:carns at mcs.anl.gov
> > <mailto:carns at mcs.anl.gov>>>
> > > To: <darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>
> > > <mailto:darshan-users at lists.mcs.anl.gov
> > <mailto:darshan-users at lists.mcs.anl.gov>>>
> > > Subject: Re: [Darshan-users] Darshan error with HDF5
> > > Message-ID: <55677B44.7060107 at mcs.anl.gov
> > <mailto:55677B44.7060107 at mcs.anl.gov>
> > > <mailto:55677B44.7060107 at mcs.anl.gov
> > <mailto:55677B44.7060107 at mcs.anl.gov>>>
> > > Content-Type: text/plain; charset="windows-1252";
> > Format="flowed"
> > >
> > > Hi Adnan!
> > >
> > > Is everything in the application statically linked (maybe check
> > > with ldd
> > > <executable>)? That error message is supposed to only be
> > generated by
> > > the dynamic (ld-preloaded) darshan library, because it has
> > to locate
> > > underlying symbols at run time. The symbol name mapping
> > should be set
> > > at link time for a static executable.
> > >
> > > thanks,
> > > -Phil
> > >
> > > On 05/28/2015 04:20 PM, Adnan Haider wrote:
> > > > Hi,
> > > > I am hoping to use Darshan for analyzing our application's
> > > > performance. We are using HDF5 and netcdf 4. I installed
> > Darshan
> > > > version 2.3.1 and mvapich2 for our mpi implementation.
> > Darshan works
> > > > fine for a standard mpi pingpong application. When we run our
> > > > application, however, it gives the following error:
> > > > Darshan failed to map symbol: H5Fopen
> > > > Darshan failed to map symbol: H5Fopen
> > > > Darshan failed to map symbol: H5Fopen
> > > > Darshan failed to map symbol: H5Fopen
> > > >
> > > > The four instances of the error probably have to do with
> > the 4 mpi
> > > > ranks specified at runtime. Nothing is written into the log
> > > directory.
> > > > This is the only output, unless there is some error file I
> > am not
> > > > aware of. I used the static instrumentation approach for
> > enabling
> > > > Darshan. Thanks for the help!
> > > >
> > > > Thanks, Adnan Haider
> > > > B.S Candidate, Computer Science
> > > > Illinois Institute of Technology
> > >
> > >
> > > -------------- next part --------------
> > > An HTML attachment was scrubbed...
> > > URL:
> > >
> > <
> http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20150528/bc72a5df/attachment-0001.html
> >
> > >
> > > ------------------------------
> > >
> > > _______________________________________________
> > > Darshan-users mailing list
> > > Darshan-users at lists.mcs.anl.gov
> > <mailto:Darshan-users at lists.mcs.anl.gov>
> > > <mailto:Darshan-users at lists.mcs.anl.gov
> > <mailto:Darshan-users at lists.mcs.anl.gov>>
> > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
> > >
> > >
> > > End of Darshan-users Digest, Vol 45, Issue 6
> > > ********************************************
> > >
> > >
> >
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> > <
> http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20150702/f7894a89/attachment-0001.html
> >
> >
> > ------------------------------
> >
> > _______________________________________________
> > Darshan-users mailing list
> > Darshan-users at lists.mcs.anl.gov
> > <mailto:Darshan-users at lists.mcs.anl.gov>
> > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
> >
> >
> > End of Darshan-users Digest, Vol 47, Issue 1
> > ********************************************
> >
> >
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20150708/6ab23d99/attachment.html
> >
>
> ------------------------------
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>
>
> End of Darshan-users Digest, Vol 47, Issue 3
> ********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20150708/38418e17/attachment-0001.html>
More information about the Darshan-users
mailing list