[Darshan-users] Profiling Python C/C++ extensions with Darshan

[Harms, Kevin]

Ivan,

  I have not tried it myself but it should be possible. For your python application, are you using the Cray provided MPI library? Are you using the cluster compatibility mode (CCM)?

kevin

-----Original Message-----
From: <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Ivan Raikov <ivan.g.raikov at gmail.com>
Date: Monday, February 27, 2017 at 1:09 PM
To: "darshan-users at lists.mcs.anl.gov" <darshan-users at lists.mcs.anl.gov>
Subject: [Darshan-users] Profiling Python C/C++ extensions with Darshan

    Hello,
    
    I have a Python extension written in C++ which calls the HDF library,
    and I would like to use Darshan to profile an MPI-based Python
    application which invokes that extension. However, compiling the
    extension and running the application on Blue Waters (Cray-based
    system) with Darshan 2.3 results in no profile data being generated.
    Using LD_PRELOAD to load Darshan dynamically results in some kind of
    socket error when (I think) the MPI process manager is launching the
    Python executable. I have also tried the LD_PRELOAD mechanism with
    Darshan 3.1.3 built by the HDF group, but I  get the same error. I was
    wondering if anyone here has done profiling with dynamically linked
    Python extensions, or if anyone has any advice how to approach this.
    I invoke the application with LD_PRELOAD thus:
    
    export LD_PRELOAD=/path/to/darshan/3.1.3/lib/libdarshan.so
    aprun -n 4096 python main.py ...
    
    or
    
    aprun -n 4096 -e LD_PRELOAD=/path/to/darshan/3.1.3/.../lib/libdarshan.so
    
    The error I get with LD_PRELOAD is the following:
    
    [PE_3344]:inet_listen_socket_setup:inet_setup_listen_socket: bind
    failed port 1371 listen_sock = 88 Address already in use
    [PE_3344]:_pmi_inet_listen_socket_setup:socket setup failed
    [PE_3344]:_pmi_init:_pmi_inet_listen_socket_setup (full) returned -1
    [PE_208]:inet_listen_socket_setup:inet_setup_listen_socket: bind
    failed port 1371 listen_sock = 74 Address already in use
    [PE_208]:_pmi_inet_listen_socket_setup:socket setup failed
    [PE_208]:_pmi_init:_pmi_inet_listen_socket_setup (full) returned -1
    
    Thanks and best regards,
    _______________________________________________
    Darshan-users mailing list
    Darshan-users at lists.mcs.anl.gov
    https://lists.mcs.anl.gov/mailman/listinfo/darshan-users