[Darshan-users] Instrumenting statically-linked applications

Thu Nov 29 13:34:19 CST 2012

Hi Phil,
Thank you for taking time to explain what is going on. Everything is 
clear now. I thought darshan uses the same procedures for static and 
dynamic libraries.

So, I have introduced the LD_PRELOAD environment variable but I have 
issues when I launch the job. I have to work with TACC sysadmins because 
there is something in the launch script that hangs the launch procedure.

I will also try to see if TACC sysadmins can generate static libs.

I will keep you posted.

Thanks,
Dragos

Dragos Constantin, PhD

Research Associate
Department of Radiology
Stanford University
Lucas MRS Center
1201 Welch Rd., PS-055
Stanford CA 94305

Office: (650) 736-9961
Fax: (650) 723-5795

On 11/29/2012 8:48 AM, Phil Carns wrote:
> On 11/29/2012 11:35 AM, Dragos Constantin wrote:
>> Hi Phil,
>> I have to recognize I am a bit confused. Let me tell you what I was 
>> thinking.
>>
>> 1. The shared and static darshan libraries were generated from the 
>> same code, i.e. they contain the same functions.
>> 2. The only difference between them is how these functions are called 
>> at run time.
>> 3. Darshan I/O monitoring functions catch any I/O event generated by 
>> the application regardless the way the application is compiled.
>
> No.  The darshan.so library can only capture I/O events that are 
> handled by shared libraries.  The darshan.a library (in conjunction 
> with a modified mpicc script) can only capture I/O events that are 
> handled by static libraries.  You can only use one or the other, not 
> both,  unfortunately.
>
>> 4. In my mind both methods should generate the same output regardless 
>> the way the program is linked.
>>
>> Now, the Geant4 toolkit have static libraries, hence my application 
>> is going to be a mix of static and shared libraries. From what you 
>> are saying is that my application should use 100% static libraries or 
>> 100% shared libraries. In other words there is no solution for my case.
>
> No, not at all!  Keep doing what you are doing.  The solution for the 
> way that you are building right now is to use darshan with the 
> LD_PRELOAD approach that I described.  Darshan does not mind if you 
> have a mix of static and shared libaries.  It's oblivious to that. The 
> only two libraries it cares about (in general) are libc and libmpich.  
> As long as those two are dynamic, then you must preload the 
> libdarshan.so library to capture I/O events.  How you handle the rest 
> of your libraries makes no difference at all.
>
>>
>> How hard it is to modify darshan to work with mixed cases like mine?
>
> No modification is needed; see above.  Darshan has no interest in 
> libraries that don't define the symbols that it intends to instrument.
>
>>
>> I will also try to see if TACC sysadmins can generate static libraries.
>
> That's up to you.  You can use LD_PRELOAD and Darshan with what you 
> have now.  It basically breaks down like this:
>
> libc and libmpich are statically linked: use the modified mpicc script 
> to inject instrumentation at compile time
> libc and libmpich are dynamically linked: use the LD_PRELOAD method to 
> inject instrumentation at run time
>
> Otherwise the rest is totally up to you as to whether you use a mix of 
> static and dynamic libraries or not.
>
> Does that help?
>
> I was confused initially because I generally assume that an 
> application will either be purely static or purely dynamic, but that 
> was just my assumption going in.  It isn't a limitation of Darshan.
>
> thanks!
> -Phil
>>
>> Thanks,
>> Dragos
>>
>>
>> Dragos Constantin, PhD
>>
>> Research Associate
>> Department of Radiology
>> Stanford University
>> Lucas MRS Center
>> 1201 Welch Rd., PS-055
>> Stanford CA 94305
>>
>> Office: (650) 736-9961
>> Fax: (650) 723-5795
>>
>> On 11/29/2012 6:45 AM, Phil Carns wrote:
>>> On 11/29/2012 09:42 AM, Phil Carns wrote:
>>>> On 11/28/2012 11:10 PM, Dragos Constantin wrote:
>>>>> Hi Phil,
>>>>> Here is the ldd output:
>>>>>
>>>> <snip>
>>>>> libmpich.so.1.2 => 
>>>>> /opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 
>>>>> (0x00002b574274a000)
>>>> <snip>
>>>>
>>>>> libc.so.6 => /lib64/libc.so.6 (0x000000358d400000)
>>>>
>>>> Hi Dragos, sorry about the wild goose chase, but because these two 
>>>> libraries are shared, you will need to use the LD_PRELOAD method to 
>>>> capture your I/O calls.  The mechanism that you have been using 
>>>> (with a modified mpicc script) only works when those libraries are 
>>>> statically linked.
>>>>
>>>>>
>>>>> I believe LD_PRELOAD is used by the dynamic library loader (ld.so) 
>>>>> to find the darshan dynamic library 'libdarshan.so'. I might be 
>>>>> wrong so please do not hesitate to correct me. In any case, it 
>>>>> does not hurt to export this environment variable but if I am 
>>>>> right this will not change a thing as 'ld.so' does not touch 
>>>>> 'libdarshan.so'.
>>>>
>>>> The general idea with this approach is that you link your 
>>>> application normally at compile time and then set LD_PRELOAD to 
>>>> point to libdarshan.so at run time.  When your application 
>>>> launches, ld.so will link in this LD_PRELOAD library before 
>>>> anything else, which gives Darshan a chance to intercept your I/O 
>>>> function symbols before they get to libc or libmpich. LD_PRELOAD 
>>>> doesn't just set a library path, it actually specifies a specific 
>>>> .so file (previously unknown to the application) that the loader 
>>>> will link in.
>>>>
>>>> So the steps that you need to try are as follows:
>>>>
>>>> 1) recompile and relink your application using the stock mpicc 
>>>> (with no darshan modifications).  You should not see any darshan 
>>>> symbols in the resulting executable.
>>>> 2) add a line like this near the job of your job script:
>>>>
>>>> #$ -v 
>>>> LD_PRELOAD=/full/path/to/your/darshan/installation/lib/libdarshan.so
>>>
>>> Just a quick note to anyone else that may run across this in the 
>>> list archives; the format of that line is specific to the scheduler 
>>> used on Ranger.  In general you might just do "export 
>>> LD_PRELOAD=..." to set the environment variable or else use whatever 
>>> is appropriate for your scheduler or job launcher.
>>>
>>> -Phil
>>>
>>>>
>>>> 3) run your application as usual
>>>>
>>>> I think that should work in your case.  As Bill mentioned you do 
>>>> need to call MPI_Init() and MPI_Finalize(), but it sounds like you 
>>>> are already doing that (otherwise you would not get the empty log 
>>>> that you are seeing now), so you should be all set.
>>>>
>>>> Sorry about the confusion.  It is pretty annoying that there isn't 
>>>> one method that works for both static and dynamic linking.  You 
>>>> have to go one way or the other depending on how those libraries 
>>>> are used.
>>>>
>>>> -Phil
>>>>
>>>> _______________________________________________
>>>> Darshan-users mailing list
>>>> Darshan-users at lists.mcs.anl.gov
>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>
>>> _______________________________________________
>>> Darshan-users mailing list
>>> Darshan-users at lists.mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>>>
>>
>
>