[Darshan-users] Profiling Python C/C++ extensions with Darshan

[Ivan Raikov]

The Blue Waters support staff worked out a solution, so I am posting
it here for the sake of posterity.
The following environment variable settings are necessary to work
around the socket error on MPI startup:

export PMI_NO_FORK=1
export PMI_NO_PREINITIALIZE=1
#both hdf5 and darshan are needed.
export LD_PRELOAD=/sw/xe/darshan/2.3.0/darshan-2.3.0_cle52/lib/libdarshan.so:/opt/cray/hdf5-parallel/1.8.16/GNU/4.9/lib/libhdf5_parallel_gnu_49.so.10
aprun ...

On Mon, Feb 27, 2017 at 11:09 AM, Ivan Raikov <ivan.g.raikov at gmail.com> wrote:
> Hello,
>
> I have a Python extension written in C++ which calls the HDF library,
> and I would like to use Darshan to profile an MPI-based Python
> application which invokes that extension. However, compiling the
> extension and running the application on Blue Waters (Cray-based
> system) with Darshan 2.3 results in no profile data being generated.
> Using LD_PRELOAD to load Darshan dynamically results in some kind of
> socket error when (I think) the MPI process manager is launching the
> Python executable. I have also tried the LD_PRELOAD mechanism with
> Darshan 3.1.3 built by the HDF group, but I  get the same error. I was
> wondering if anyone here has done profiling with dynamically linked
> Python extensions, or if anyone has any advice how to approach this.
> I invoke the application with LD_PRELOAD thus:
>
> export LD_PRELOAD=/path/to/darshan/3.1.3/lib/libdarshan.so
> aprun -n 4096 python main.py ...
>
> or
>
> aprun -n 4096 -e LD_PRELOAD=/path/to/darshan/3.1.3/.../lib/libdarshan.so
>
> The error I get with LD_PRELOAD is the following:
>
> [PE_3344]:inet_listen_socket_setup:inet_setup_listen_socket: bind
> failed port 1371 listen_sock = 88 Address already in use
> [PE_3344]:_pmi_inet_listen_socket_setup:socket setup failed
> [PE_3344]:_pmi_init:_pmi_inet_listen_socket_setup (full) returned -1
> [PE_208]:inet_listen_socket_setup:inet_setup_listen_socket: bind
> failed port 1371 listen_sock = 74 Address already in use
> [PE_208]:_pmi_inet_listen_socket_setup:socket setup failed
> [PE_208]:_pmi_init:_pmi_inet_listen_socket_setup (full) returned -1
>
> Thanks and best regards,