[Darshan-users] LD_PRELOAD fails with OpenMPI

Fri Dec 28 05:00:18 CST 2012

Hi,
When I use LD_PRELOAD to get an application instrumented with Darshan at 
run time, a SEGMENTATION FAULT raises. 
Also, the behaviour is the same when simply running mpicc to get the 
version - and the generated coredump displays the same stack - which seems 
to point the MPI wrappers as being the potential suspect.

# which mpicc
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc
# mpicc --showme:version
Erreur de segmentation (core dumped)
# gdb mpicc core.14109
GNU gdb (GDB) bullx Linux (7.2-50.bl6.Bull.1.20120306)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
Reading symbols from 
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc...done.
[New Thread 14109]
Missing separate debuginfo for 
/home_nfs/botallam/install/darshan.4/lib/libdarshan.so
....
Reading symbols from 
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1...done.
Loaded symbols for 
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1
Core was generated by 
`/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000036b7879446 in calloc () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
glibc-2.12-1.47.bl6_2.9.x86_64 libgcc-4.4.5-6.bl6.x86_64 
numactl-2.0.3-9.bl6.x86_64 zlib-1.2.3-25.bl6.x86_64
(gdb) where
#0  0x00000036b7879446 in calloc () from /lib64/libc.so.6
#1  0x00000036b7c01310 in _dlerror_run () from /lib64/libdl.so.2
#2  0x00000036b7c0107a in dlsym () from /lib64/libdl.so.2
#3  0x00007f50f98c3487 in __xstat (vers=1, path=0x7f50f966ffac 
"/dev/ummunotify", buf=0x7fff6b6f08c0) at lib/darshan-posix.c:711
#4  0x00007f50f9661a64 in opal_memory_linux_malloc_init_hook () at 
hooks.c:756
#5  0x00000036b7875b63 in ptmalloc_init () from /lib64/libc.so.6
#6  0x00000036b7879987 in malloc_hook_ini () from /lib64/libc.so.6
#7  0x00000036b78a6da1 in __alloc_dir () from /lib64/libc.so.6
#8  0x00000036b94053cd in ?? () from /usr/lib64/libnuma.so.1
#9  0x00000036b740e515 in _dl_init_internal () from 
/lib64/ld-linux-x86-64.so.2
#10 0x00000036b7400b3a in _dl_start_user () from 
/lib64/ld-linux-x86-64.so.2
#11 0x0000000000000002 in ?? ()
#12 0x00007fff6b6f1a43 in ?? ()
#13 0x00007fff6b6f1a9e in ?? ()
#14 0x0000000000000000 in ?? ()
(gdb)

Can someone help me to understand the issue?
Thanks,
Myriam.

HERE IS the environment:

LD_PRELOAD=/home_nfs/botallam/install/darshan.4/lib/libdarshan.so

The Darshan library was configured and generated with environment variable 
OMPI_CC=gcc
using mpicc: Open MPI 1.6.2 (Language: C)

# which mpicc
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc
# CC=mpicc CFLAGS=-g ./configure 
--prefix=/home_nfs/botallam/install/darshan.4 --with-mem-align=16 
--with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID

# ldd /home_nfs/botallam/install/darshan.4/lib/libdarshan.so
        linux-vdso.so.1 =>  (0x00007fffe9e93000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a753bd000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a751a0000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f8a74f98000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f8a74d83000)
        libmpi.so.1 => 
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1 
(0x00007f8a74967000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8a746e3000)
        libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f8a744db000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f8a742c1000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f8a740be000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8a73d2e000)
        /lib64/ld-linux-x86-64.so.2 (0x00000036b7400000)
        libimf.so => 
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libimf.so 
(0x00007f8a73871000)
        libsvml.so => 
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libsvml.so 
(0x00007f8a72fa3000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a72d8d000)
        libintlc.so.5 => 
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libintlc.so.5 
(0x00007f8a72b3e000)
#
#
# nm /home_nfs/botallam/install/darshan.4/lib/libdarshan.so|grep mpi
0000000000227020 B __real_ncmpi_close
0000000000227010 B __real_ncmpi_create
0000000000227018 B __real_ncmpi_open
00000000000048cc T darshan_mpi_initialize
00000000000179d2 T ncmpi_close
000000000001763c T ncmpi_create
0000000000017807 T ncmpi_open
                 U ompi_mpi_byte
                 U ompi_mpi_char
                 U ompi_mpi_comm_world
                 U ompi_mpi_double
                 U ompi_mpi_info_null
                 U ompi_mpi_int
                 U ompi_mpi_long
                 U ompi_mpi_op_land
                 U ompi_mpi_op_lor
                 U ompi_mpi_op_max
                 U ompi_mpi_op_null
                 U ompi_mpi_op_sum
0000000000016444 T resolve_mpi_symbols
#

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20121228/8f171962/attachment.html>