[Darshan-users] Instrumenting statically-linked applications

Phil Carns carns at mcs.anl.gov
Wed Nov 28 20:12:21 CST 2012 

Hi Dragos and Bill,

It might also be the case that even in the mode you are building now 
(with a mix of static and dynamic libraries) that you'll need to use 
LD_PRELOAD.  The important thing from Darshan's perspective is whether 
the libraries that provide the I/O calls are dynamic or static.  In 
other words, even if your application libraries are static but they are 
calling POSIX file access functions provided by a shared libc and/or 
MPI-IO functions provided by a shared libmpi, then you'll probably need 
to ld preload darshan to catch those calls.

Can you post the output from "ldd <application>" to check which 
libraries are dynamically linked?

thanks,
-Phil

On 11/28/2012 08:31 PM, Bill Barth wrote:
> Dragos,
>
> Your directories are available on all the compute nodes on Ranger, so if
> your darshan dynamic libs are in any of your directories, you should be
> able to set your LD_LIBRARYPATH or the executable rpath appropriately to
> point at your version of the darshan dynamic libraries.
>
> Is there a reason you prefer the static version?
>
> Bill.
> --
> Bill Barth, Ph.D., Director, HPC
> bbarth at tacc.utexas.edu        |   Phone: (512) 232-7069
> Office: ROC 1.435             |   Fax:   (512) 475-9445
>
>
>
>
>
>
>
> On 11/28/12 6:20 PM, "Dragos Constantin" <dragos.constantin at stanford.edu>
> wrote:
>
>> Hi Phil,
>> So, v2.2.4-pre6 works on both Ranger and Lonestar. I can confirm that
>> darshan generates a log file. However, I am a bit confused because when
>> I parse the log file it says my application did not open any file and in
>> fact I am at least generating several output files. Maybe I have to
>> configure something or supply some flags at compilation time so I can
>> capture the full I/O load generated by my application.
>>
>> Do you think I should use the darshan-test application and see what the
>> output looks like?
>>
>> You are right one cannot build 100% static executables on Ranger and
>> Lonestar. However, the dynamic libs such as libverbs are installed on
>> each compute node so, it is not an issue. What is more important is that
>> the darshan lib and all my other libs are statically linked because they
>> are not deployed system wide. In any case, I am thinking I should have
>> seen in the darshan log files that some I/O activity occurred because of
>> these dynamic libs.
>>
>>
>> Thanks,
>> Dragos
>>
>>
>>
>> Dragos Constantin, PhD
>>
>> Research Associate
>> Department of Radiology
>> Stanford University
>> Lucas MRS Center
>> 1201 Welch Rd., PS-055
>> Stanford CA 94305
>>
>> Office: (650) 736-9961
>> Fax: (650) 723-5795
>>
>> On 11/28/2012 11:25 AM, Phil Carns wrote:
>>> Hi Dragos,
>>>
>>> Could you try this pre-release version of Darshan and us know if it
>>> works for you?
>>>
>>> ftp://ftp.mcs.anl.gov/pub/darshan/releases/darshan-2.2.4-pre6.tar.gz
>>>
>>> The darshan-gen-* scripts will only work with mvapich2.
>>>
>>> I noticed an unrelated issue when trying to test this release on
>>> Ranger, however.  I was not able to build a static executable using
>>> mvapich2 (with or without darshan) because it could not find a static
>>> version of the libverbs library. I was trying to generate a static
>>> executable by just adding -static to the mpicc command line.  Maybe
>>> there is an additional step needed to get a fully static executable?
>>>
>>> thanks,
>>> -Phil
>>>
>>> On 11/27/2012 10:25 AM, Phil Carns wrote:
>>>> Hi Dragos,
>>>>
>>>> Thanks for the bug report.  It looks like the
>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc is just ordering the link
>>>> arguments differently than darshan-gen-cc.pl expected.  We should be
>>>> able to work around this without too much trouble. In terms of the
>>>> perl code I think we just need to modify the regular expression to
>>>> collect a "$link_cmd_prefix" in addition to a "$link_cmd_suffix" if
>>>> anything appears in the link command line from the first '-'
>>>> character up to the object name.  We can then just pass those
>>>> arguments as is into the generated script. In this example the
>>>> link_cmd_prefix would be:
>>>>
>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>
>>>> I would like to see that particular mpicc script before making any
>>>> changes, though, to make sure that we don't accidentally break
>>>> something, but as (bad) luck would have it Ranger is in maintenance
>>>> today.  We'll have a look at it tomorrow.
>>>>
>>>> thanks,
>>>> -Phil
>>>>
>>>> On 11/26/2012 03:27 PM, Dragos Constantin wrote:
>>>>> Hi Kevin,
>>>>> The problem is not with the argument parsing. This is what I get in
>>>>> both cases:
>>>>>
>>>>> login4$ ./darshan-gen-cc.pl /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>> --output mpicc.darshan
>>>>> CC_from_link = gcc
>>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>> CC_from_compile = gcc
>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>
>>>>> login4$ ./darshan-gen-cc.pl --output mpicc.darshan
>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>> CC_from_link = gcc
>>>>> -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link
>>>>> -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib
>>>>> CC_from_compile = gcc
>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>
>>>>> As you can see:
>>>>>
>>>>> CC_from_compile = gcc
>>>>>
>>>>> but CC_from_link is not gcc and if I am not mistaken it should be
>>>>> gcc. I just started to look at the script and you might know better
>>>>> what is going on here.
>>>>>
>>>>> Thanks,
>>>>> Dragos
>>>>>
>>>>>
>>>>> Dragos Constantin, PhD
>>>>>
>>>>> Research Associate
>>>>> Department of Radiology
>>>>> Stanford University
>>>>> Lucas MRS Center
>>>>> 1201 Welch Rd., PS-055
>>>>> Stanford CA 94305
>>>>>
>>>>> Office: (650) 736-9961
>>>>> Fax: (650) 723-5795
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Kevin Harms" <harms at alcf.anl.gov>
>>>>> To: "Dragos Constantin" <dragos.constantin at stanford.edu>
>>>>> Cc: darshan-users at lists.mcs.anl.gov
>>>>> Sent: Monday, November 26, 2012 12:23:00 PM
>>>>> Subject: Re: [Darshan-users] Instrumenting statically-linked
>>>>> applications
>>>>>
>>>>>
>>>>>     I think this might be a simple issue with argument parsing. Try
>>>>> this instead:
>>>>>
>>>>>> ./darshan-gen-cc.pl --output mpicc.darshan
>>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc
>>>>> kevin
>>>>>
>>>>> On Nov 26, 2012, at 2:16 PM, Dragos Constantin wrote:
>>>>>
>>>>>> Hi,
>>>>>> I've installed and configured darshan-2.2.3 on TACC Ranger in my
>>>>>> user space. I have used gcc-4.4.5 (and mvapich-1.0.1).
>>>>>>
>>>>>> When I try to generate the MPI compiler scripts for
>>>>>> statically-linked applications I get the following error:
>>>>>>
>>>>>> login4$ ./darshan-gen-cc.pl
>>>>>> /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc --output mpicc.darshan
>>>>>> Error: cannot find matching CC from: gcc -c foo.c
>>>>>> -I/opt/apps/gcc4_4/mvapich/1.0.1/include
>>>>>> and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared
>>>>>> -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich
>>>>>> -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt
>>>>>>
>>>>>> I am not quite sure what triggered this. Any ideas how to quickly
>>>>>> fix the issue? I will look at the perl script to see what is going
>>>>>> on there.
>>>>>>
>>>>>> Thanks,
>>>>>> Dragos
>>>>>>
>>>>>>
>>>>>> Dragos Constantin, PhD
>>>>>>
>>>>>> Research Associate
>>>>>> Department of Radiology
>>>>>> Stanford University
>>>>>> Lucas MRS Center
>>>>>> 1201 Welch Rd., PS-055
>>>>>> Stanford CA 94305
>>>>>>
>>>>>> Office: (650) 736-9961
>>>>>> Fax: (650) 723-5795
>>>>>>
>>>>>> _______________________________________________
>>>>>> Darshan-users mailing list
>>>>>> Darshan-users at lists.mcs.anl.gov
>>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>>> _______________________________________________
>>>>> Darshan-users mailing list
>>>>> Darshan-users at lists.mcs.anl.gov
>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>> _______________________________________________
>>>> Darshan-users mailing list
>>>> Darshan-users at lists.mcs.anl.gov
>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>
>> _______________________________________________
>> Darshan-users mailing list
>> Darshan-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

More information about the Darshan-users mailing list