[Darshan-users] Instrumenting statically-linked applications

[Phil Carns]

On 11/29/2012 09:42 AM, Phil Carns wrote:
> On 11/28/2012 11:10 PM, Dragos Constantin wrote:
>> Hi Phil,
>> Here is the ldd output:
>>
> <snip>
>> libmpich.so.1.2 => /opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 
>> (0x00002b574274a000)
> <snip>
>
>> libc.so.6 => /lib64/libc.so.6 (0x000000358d400000)
>
> Hi Dragos, sorry about the wild goose chase, but because these two 
> libraries are shared, you will need to use the LD_PRELOAD method to 
> capture your I/O calls.  The mechanism that you have been using (with 
> a modified mpicc script) only works when those libraries are 
> statically linked.
>
>>
>> I believe LD_PRELOAD is used by the dynamic library loader (ld.so) to 
>> find the darshan dynamic library 'libdarshan.so'. I might be wrong so 
>> please do not hesitate to correct me. In any case, it does not hurt 
>> to export this environment variable but if I am right this will not 
>> change a thing as 'ld.so' does not touch 'libdarshan.so'.
>
> The general idea with this approach is that you link your application 
> normally at compile time and then set LD_PRELOAD to point to 
> libdarshan.so at run time.  When your application launches, ld.so will 
> link in this LD_PRELOAD library before anything else, which gives 
> Darshan a chance to intercept your I/O function symbols before they 
> get to libc or libmpich.  LD_PRELOAD doesn't just set a library path, 
> it actually specifies a specific .so file (previously unknown to the 
> application) that the loader will link in.
>
> So the steps that you need to try are as follows:
>
> 1) recompile and relink your application using the stock mpicc (with 
> no darshan modifications).  You should not see any darshan symbols in 
> the resulting executable.
> 2) add a line like this near the job of your job script:
>
> #$ -v 
> LD_PRELOAD=/full/path/to/your/darshan/installation/lib/libdarshan.so

Just a quick note to anyone else that may run across this in the list 
archives; the format of that line is specific to the scheduler used on 
Ranger.  In general you might just do "export LD_PRELOAD=..." to set the 
environment variable or else use whatever is appropriate for your 
scheduler or job launcher.

-Phil

>
> 3) run your application as usual
>
> I think that should work in your case.  As Bill mentioned you do need 
> to call MPI_Init() and MPI_Finalize(), but it sounds like you are 
> already doing that (otherwise you would not get the empty log that you 
> are seeing now), so you should be all set.
>
> Sorry about the confusion.  It is pretty annoying that there isn't one 
> method that works for both static and dynamic linking.  You have to go 
> one way or the other depending on how those libraries are used.
>
> -Phil
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users