[Darshan-users] Instrumenting statically-linked applications

[Phil Carns]

On 11/28/2012 11:10 PM, Dragos Constantin wrote:
> Hi Phil,
> Here is the ldd output:
>
<snip>
> libmpich.so.1.2 => /opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 
> (0x00002b574274a000)
<snip>

> libc.so.6 => /lib64/libc.so.6 (0x000000358d400000)

Hi Dragos, sorry about the wild goose chase, but because these two 
libraries are shared, you will need to use the LD_PRELOAD method to 
capture your I/O calls.  The mechanism that you have been using (with a 
modified mpicc script) only works when those libraries are statically 
linked.

>
> I believe LD_PRELOAD is used by the dynamic library loader (ld.so) to 
> find the darshan dynamic library 'libdarshan.so'. I might be wrong so 
> please do not hesitate to correct me. In any case, it does not hurt to 
> export this environment variable but if I am right this will not 
> change a thing as 'ld.so' does not touch 'libdarshan.so'.

The general idea with this approach is that you link your application 
normally at compile time and then set LD_PRELOAD to point to 
libdarshan.so at run time.  When your application launches, ld.so will 
link in this LD_PRELOAD library before anything else, which gives 
Darshan a chance to intercept your I/O function symbols before they get 
to libc or libmpich.  LD_PRELOAD doesn't just set a library path, it 
actually specifies a specific .so file (previously unknown to the 
application) that the loader will link in.

So the steps that you need to try are as follows:

1) recompile and relink your application using the stock mpicc (with no 
darshan modifications).  You should not see any darshan symbols in the 
resulting executable.
2) add a line like this near the job of your job script:

#$ -v LD_PRELOAD=/full/path/to/your/darshan/installation/lib/libdarshan.so

3) run your application as usual

I think that should work in your case.  As Bill mentioned you do need to 
call MPI_Init() and MPI_Finalize(), but it sounds like you are already 
doing that (otherwise you would not get the empty log that you are 
seeing now), so you should be all set.

Sorry about the confusion.  It is pretty annoying that there isn't one 
method that works for both static and dynamic linking.  You have to go 
one way or the other depending on how those libraries are used.

-Phil