[Darshan-users] Instrumenting statically-linked applications
Dragos Constantin
dragos.constantin at stanford.edu
Wed Nov 28 22:10:50 CST 2012
Hi Phil,
Here is the ldd output:
login1$ ldd application
libz.so.1 => /usr/lib64/libz.so.1 (0x000000358e400000)
libmpichcxx.so.1.2 =>
/opt/apps/gcc4_4/mvapich2/1.6/lib/libmpichcxx.so.1.2 (0x00002b5742527000)
libmpich.so.1.2 =>
/opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 (0x00002b574274a000)
liblimic2.so.0 => /opt/apps/limic2/0.5.4//lib/liblimic2.so.0
(0x00002b5742bb0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000358e000000)
librdmacm.so.1 => /opt/ofed/lib64/librdmacm.so.1
(0x00002b5742db1000)
libibverbs.so.1 => /opt/ofed/lib64/libibverbs.so.1
(0x00002b5742fb9000)
libibumad.so.3 => /opt/ofed/lib64/libibumad.so.3
(0x00002b57431c7000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000358d800000)
librt.so.1 => /lib64/librt.so.1 (0x000000358e800000)
libstdc++.so.6 => /opt/apps/gcc/4.4.5/lib64/libstdc++.so.6
(0x00002b57433ce000)
libm.so.6 => /lib64/libm.so.6 (0x000000358dc00000)
libgcc_s.so.1 => /opt/apps/gcc/4.4.5/lib64/libgcc_s.so.1
(0x00002b57436df000)
libc.so.6 => /lib64/libc.so.6 (0x000000358d400000)
/lib64/ld-linux-x86-64.so.2 (0x000000358d000000)
Also this is the nm output of the application and as you can see symbols
from both libdarshan-posix.a and libdarshan-mpi-io.a are present.
login1$ nm application |grep darshan
00000000007e17c0 T darshan_condense
00000000007e16a0 t darshan_file_by_fd
000000000040ac90 t darshan_file_by_fh
00000000007e1670 T darshan_file_by_handle
00000000007e19a0 T darshan_file_by_name
00000000007e1d00 T darshan_file_by_name_sethandle
00000000007e16d0 T darshan_file_closehandle
0000000000412d50 t darshan_file_reduce
00000000007e17a0 T darshan_finalize
000000000040a8f0 T darshan_get_exe_and_mounts
00000000023e0380 B darshan_global_job
00000000007e63e0 T darshan_hash
00000000007e1e70 T darshan_initialize
00000000023cd248 d darshan_mem_alignment
000000000040a870 T darshan_mnt_id_from_path
0000000000414890 T darshan_mpi_initialize
0000000000413690 T darshan_shutdown
000000000040abc0 T darshan_walk_file_accesses
00000000007e1e40 T darshan_wtime
login1$ nm libdarshan-posix.a|grep darshan
darshan-posix.o:
0000000000000210 T darshan_condense
00000000000000f0 t darshan_file_by_fd
00000000000000c0 T darshan_file_by_handle
00000000000003f0 T darshan_file_by_name
0000000000000750 T darshan_file_by_name_sethandle
0000000000000120 T darshan_file_closehandle
00000000000001f0 T darshan_finalize
U darshan_get_exe_and_mounts
0000000000000000 B darshan_global_job
U darshan_hash
00000000000008c0 T darshan_initialize
0000000000000028 d darshan_mem_alignment
U darshan_mnt_id_from_path
0000000000000890 T darshan_wtime
0000000000000220 T darshan_hashlittle
0000000000000000 T darshan_hash
login1$ nm libdarshan-mpi-io.a|grep darshan
darshan-mpi-io.o:
00000000000005a0 t darshan_file_by_fh
U darshan_file_by_handle
U darshan_file_by_name_sethandle
U darshan_file_closehandle
0000000000008660 t darshan_file_reduce
U darshan_finalize
0000000000000200 T darshan_get_exe_and_mounts
U darshan_global_job
U darshan_hash
U darshan_initialize
0000000000000180 T darshan_mnt_id_from_path
000000000000a1a0 T darshan_mpi_initialize
0000000000008fa0 T darshan_shutdown
00000000000004d0 T darshan_walk_file_accesses
U darshan_wtime
darshan-mpi-init-finalize.o:
U darshan_mpi_initialize
U darshan_shutdown
darshan-pnetcdf.o:
U darshan_file_by_handle
U darshan_file_by_name_sethandle
U darshan_file_closehandle
darshan-hdf5.o:
U darshan_file_by_handle
U darshan_file_by_name_sethandle
U darshan_file_closehandle
I believe LD_PRELOAD is used by the dynamic library loader (ld.so) to
find the darshan dynamic library 'libdarshan.so'. I might be wrong so
please do not hesitate to correct me. In any case, it does not hurt to
export this environment variable but if I am right this will not change
a thing as 'ld.so' does not touch 'libdarshan.so'.
Thanks,
Dragos
Dragos Constantin, PhD
Research Associate
Department of Radiology
Stanford University
Lucas MRS Center
1201 Welch Rd., PS-055
Stanford CA 94305
Office: (650) 736-9961
Fax: (650) 723-5795
On 11/28/2012 6:12 PM, Phil Carns wrote:
> Hi Dragos and Bill,
>
> It might also be the case that even in the mode you are building now
> (with a mix of static and dynamic libraries) that you'll need to use
> LD_PRELOAD. The important thing from Darshan's perspective is whether
> the libraries that provide the I/O calls are dynamic or static. In
> other words, even if your application libraries are static but they
> are calling POSIX file access functions provided by a shared libc
> and/or MPI-IO functions provided by a shared libmpi, then you'll
> probably need to ld preload darshan to catch those calls.
>
> Can you post the output from "ldd <application>" to check which
> libraries are dynamically linked?
>
> thanks,
> -Phil
>
> On 11/28/2012 08:31 PM, Bill Barth wrote:
>> Dragos,
>>
>> Your directories are available on all the compute nodes on Ranger, so if
>> your darshan dynamic libs are in any of your directories, you should be
>> able to set your LD_LIBRARYPATH or the executable rpath appropriately to
>> point at your version of the darshan dynamic libraries.
>>
>> Is there a reason you prefer the static version?
>>
>> Bill.
>> --
>> Bill Barth, Ph.D., Director, HPC
>> bbarth at tacc.utexas.edu | Phone: (512) 232-7069
>> Office: ROC 1.435 | Fax: (512) 475-9445
>>
>>
>>
>>
>>
>>
>>
>> On 11/28/12 6:20 PM, "Dragos Constantin"
>> <dragos.constantin at stanford.edu>
>> wrote:
>>
>>> Hi Phil,
>>> So, v2.2.4-pre6 works on both Ranger and Lonestar. I can confirm that
>>> darshan generates a log file. However, I am a bit confused because when
>>> I parse the log file it says my application did not open any file
>>> and in
>>> fact I am at least generating several output files. Maybe I have to
>>> configure something or supply some flags at compilation time so I can
>>> capture the full I/O load generated by my application.
>>>
>>> Do you think I should use the darshan-test application and see what the
>>> output looks like?
>>>
>>> You are right one cannot build 100% static executables on Ranger and
>>> Lonestar. However, the dynamic libs such as libverbs are installed on
>>> each compute node so, it is not an issue. What is more important is
>>> that
>>> the darshan lib and all my other libs are statically linked because
>>> they
>>> are not deployed system wide. In any case, I am thinking I should have
>>> seen in the darshan log files that some I/O activity occurred
>>> because of
>>> these dynamic libs.
>>>
>>>
>>> Thanks,
>>> Dragos
>>>
>>>
>>>
>>> Dragos Constantin, PhD
>>>
>>> Research Associate
>>> Department of Radiology
>>> Stanford University
>>> Lucas MRS Center
>>> 1201 Welch Rd., PS-055
>>> Stanford CA 94305
>>>
>>> Office: (650) 736-9961
>>> Fax: (650) 723-5795
>>>
>>> On 11/28/2012 11:25 AM, Phil Carns wrote:
>>>> Hi Dragos,
>>>>
>>>> Could you try this pre-release version of Darshan and us know if it
>>>> works for you?
>>>>
>>>> ftp://ftp.mcs.anl.gov/pub/darshan/releases/darshan-2.2.4-pre6.tar.gz
>>>>
>>>> The darshan-gen-* scripts will only work with mvapich2.
>>>>
>>>> I noticed an unrelated issue when trying to test this release on
>>>> Ranger, however. I was not able to build a static executable using
>>>> mvapich2 (with or without darshan) because it could not find a static
>>>> version of the libverbs library. I was trying to generate a static
>>>> executable by just adding -static to the mpicc command line. Maybe
>>>> there is an additional step needed to get a fully static executable?
>>>>
>>>> thanks,
>>>> -Phil
>>>>
>>>> On 11/27/2012 10:25 AM, Phil Carns wrote:
>>>>> Hi Dragos,
>>>>>
>>>>> Thanks for the bug report. It looks like the
>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc is just ordering the link
>>>>> arguments differently than darshan-gen-cc.pl expected. We should be
>>>>> able to work around this without too much trouble. In terms of the
>>>>> perl code I think we just need to modify the regular expression to
>>>>> collect a "$link_cmd_prefix" in addition to a "$link_cmd_suffix" if
>>>>> anything appears in the link command line from the first '-'
>>>>> character up to the object name. We can then just pass those
>>>>> arguments as is into the generated script. In this example the
>>>>> link_cmd_prefix would be:
>>>>>
>>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>>
>>>>> I would like to see that particular mpicc script before making any
>>>>> changes, though, to make sure that we don't accidentally break
>>>>> something, but as (bad) luck would have it Ranger is in maintenance
>>>>> today. We'll have a look at it tomorrow.
>>>>>
>>>>> thanks,
>>>>> -Phil
>>>>>
>>>>> On 11/26/2012 03:27 PM, Dragos Constantin wrote:
>>>>>> Hi Kevin,
>>>>>> The problem is not with the argument parsing. This is what I get in
>>>>>> both cases:
>>>>>>
>>>>>> login4$ ./darshan-gen-cc.pl /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>>> --output mpicc.darshan
>>>>>> CC_from_link = gcc
>>>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>>> CC_from_compile = gcc
>>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>>
>>>>>> login4$ ./darshan-gen-cc.pl --output mpicc.darshan
>>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>>> CC_from_link = gcc
>>>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>>> CC_from_compile = gcc
>>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>>
>>>>>> As you can see:
>>>>>>
>>>>>> CC_from_compile = gcc
>>>>>>
>>>>>> but CC_from_link is not gcc and if I am not mistaken it should be
>>>>>> gcc. I just started to look at the script and you might know better
>>>>>> what is going on here.
>>>>>>
>>>>>> Thanks,
>>>>>> Dragos
>>>>>>
>>>>>>
>>>>>> Dragos Constantin, PhD
>>>>>>
>>>>>> Research Associate
>>>>>> Department of Radiology
>>>>>> Stanford University
>>>>>> Lucas MRS Center
>>>>>> 1201 Welch Rd., PS-055
>>>>>> Stanford CA 94305
>>>>>>
>>>>>> Office: (650) 736-9961
>>>>>> Fax: (650) 723-5795
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Kevin Harms" <harms at alcf.anl.gov>
>>>>>> To: "Dragos Constantin" <dragos.constantin at stanford.edu>
>>>>>> Cc: darshan-users at lists.mcs.anl.gov
>>>>>> Sent: Monday, November 26, 2012 12:23:00 PM
>>>>>> Subject: Re: [Darshan-users] Instrumenting statically-linked
>>>>>> applications
>>>>>>
>>>>>>
>>>>>> I think this might be a simple issue with argument parsing. Try
>>>>>> this instead:
>>>>>>
>>>>>>> ./darshan-gen-cc.pl --output mpicc.darshan
>>>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>>> kevin
>>>>>>
>>>>>> On Nov 26, 2012, at 2:16 PM, Dragos Constantin wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I've installed and configured darshan-2.2.3 on TACC Ranger in my
>>>>>>> user space. I have used gcc-4.4.5 (and mvapich-1.0.1).
>>>>>>>
>>>>>>> When I try to generate the MPI compiler scripts for
>>>>>>> statically-linked applications I get the following error:
>>>>>>>
>>>>>>> login4$ ./darshan-gen-cc.pl
>>>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc --output mpicc.darshan
>>>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>>>
>>>>>>> I am not quite sure what triggered this. Any ideas how to quickly
>>>>>>> fix the issue? I will look at the perl script to see what is going
>>>>>>> on there.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dragos
>>>>>>>
>>>>>>>
>>>>>>> Dragos Constantin, PhD
>>>>>>>
>>>>>>> Research Associate
>>>>>>> Department of Radiology
>>>>>>> Stanford University
>>>>>>> Lucas MRS Center
>>>>>>> 1201 Welch Rd., PS-055
>>>>>>> Stanford CA 94305
>>>>>>>
>>>>>>> Office: (650) 736-9961
>>>>>>> Fax: (650) 723-5795
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Darshan-users mailing list
>>>>>>> Darshan-users at lists.mcs.anl.gov
>>>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>>>> _______________________________________________
>>>>>> Darshan-users mailing list
>>>>>> Darshan-users at lists.mcs.anl.gov
>>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>>> _______________________________________________
>>>>> Darshan-users mailing list
>>>>> Darshan-users at lists.mcs.anl.gov
>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>>
>>> _______________________________________________
>>> Darshan-users mailing list
>>> Darshan-users at lists.mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>
>
More information about the Darshan-users
mailing list