<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Myriam,<br>
<br>
Thank you for the detailed bug report. We'll try to reproduce
this and get back to you. I assume that Open MPI is configured to
use IB in this environment? <br>
<br>
I think the issue here is that Open MPI very early on is setting
up its own wrappers for malloc, and it happens to make a stat() or
fstat() call as part of that process. This is problematic because
Darshan wants to intercept the stat() calls, but it needs malloc
working (as part of the symbol resolution process) before it can
intercept any functions via LD_PRELOAD. I'm not yet sure how to
handle this but we'll have a look at it.<br>
<br>
-Phil <br>
<br>
<br>
On 12/28/2012 06:00 AM, <a class="moz-txt-link-abbreviated" href="mailto:myriam.botalla@bull.net">myriam.botalla@bull.net</a> wrote:<br>
</div>
<blockquote
cite="mid:OF02E1FAC6.6D59C096-ONC1257AE1.0039265C-C1257AE2.003D00C1@bull.net"
type="cite"><font face="sans-serif" size="2">Hi,</font>
<br>
<font face="sans-serif" size="2">When I use LD_PRELOAD to get an
application
instrumented with Darshan at run time, a SEGMENTATION FAULT
raises. </font>
<br>
<font face="sans-serif" size="2">Also, the behaviour is the same
when
simply running mpicc to get the version - and the generated
coredump displays
the same stack - which seems to point the MPI wrappers as being
the potential
suspect.</font>
<br>
<br>
<font face="sans-serif" size="2"># which mpicc</font>
<br>
<font face="sans-serif" size="2">/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc</font>
<br>
<font face="sans-serif" size="2"># mpicc --showme:version</font>
<br>
<font face="sans-serif" size="2">Erreur de segmentation (core
dumped)</font>
<br>
<font face="sans-serif" size="2"># gdb mpicc core.14109</font>
<br>
<font face="sans-serif" size="2">GNU gdb (GDB) bullx Linux
(7.2-50.bl6.Bull.1.20120306)</font>
<br>
<font face="sans-serif" size="2">Copyright (C) 2010 Free Software
Foundation,
Inc.</font>
<br>
<font face="sans-serif" size="2">License GPLv3+: GNU GPL version 3
or
later <</font><a moz-do-not-send="true"
href="http://gnu.org/licenses/gpl.html"><font face="sans-serif"
size="2">http://gnu.org/licenses/gpl.html</font></a><font
face="sans-serif" size="2">></font>
<br>
<font face="sans-serif" size="2">This is free software: you are
free
to change and redistribute it.</font>
<br>
<font face="sans-serif" size="2">Reading symbols from
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc...done.</font><br>
<font face="sans-serif" size="2">[New Thread 14109]</font>
<br>
<font face="sans-serif" size="2">Missing separate debuginfo for
/home_nfs/botallam/install/darshan.4/lib/libdarshan.so</font>
<br>
<font face="sans-serif" size="2">....</font>
<br>
<font face="sans-serif" size="2">Reading symbols from
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1...done.</font><br>
<font face="sans-serif" size="2">Loaded symbols for
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1</font><br>
<font face="sans-serif" size="2">Core was generated by
`/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.'.</font><br>
<font face="sans-serif" size="2">Program terminated with signal
11, Segmentation
fault.</font>
<br>
<font face="sans-serif" size="2">#0 0x00000036b7879446 in calloc
() from /lib64/libc.so.6</font>
<br>
<font face="sans-serif" size="2">Missing separate debuginfos, use:
debuginfo-install
glibc-2.12-1.47.bl6_2.9.x86_64 libgcc-4.4.5-6.bl6.x86_64
numactl-2.0.3-9.bl6.x86_64
zlib-1.2.3-25.bl6.x86_64</font>
<br>
<font face="sans-serif" size="2">(gdb) where</font>
<br>
<font face="sans-serif" size="2">#0 0x00000036b7879446 in calloc
() from /lib64/libc.so.6</font>
<br>
<font face="sans-serif" size="2">#1 0x00000036b7c01310 in
_dlerror_run
() from /lib64/libdl.so.2</font>
<br>
<font face="sans-serif" size="2">#2 0x00000036b7c0107a in dlsym
() from /lib64/libdl.so.2</font>
<br>
<font face="sans-serif" size="2">#3 0x00007f50f98c3487 in __xstat
(vers=1, path=0x7f50f966ffac "/dev/ummunotify",
buf=0x7fff6b6f08c0)
at lib/darshan-posix.c:711</font>
<br>
<font face="sans-serif" size="2">#4 0x00007f50f9661a64 in
opal_memory_linux_malloc_init_hook
() at hooks.c:756</font>
<br>
<font face="sans-serif" size="2">#5 0x00000036b7875b63 in
ptmalloc_init
() from /lib64/libc.so.6</font>
<br>
<font face="sans-serif" size="2">#6 0x00000036b7879987 in
malloc_hook_ini
() from /lib64/libc.so.6</font>
<br>
<font face="sans-serif" size="2">#7 0x00000036b78a6da1 in
__alloc_dir
() from /lib64/libc.so.6</font>
<br>
<font face="sans-serif" size="2">#8 0x00000036b94053cd in ?? ()
from /usr/lib64/libnuma.so.1</font>
<br>
<font face="sans-serif" size="2">#9 0x00000036b740e515 in
_dl_init_internal
() from /lib64/ld-linux-x86-64.so.2</font>
<br>
<font face="sans-serif" size="2">#10 0x00000036b7400b3a in
_dl_start_user
() from /lib64/ld-linux-x86-64.so.2</font>
<br>
<font face="sans-serif" size="2">#11 0x0000000000000002 in ?? ()</font>
<br>
<font face="sans-serif" size="2">#12 0x00007fff6b6f1a43 in ?? ()</font>
<br>
<font face="sans-serif" size="2">#13 0x00007fff6b6f1a9e in ?? ()</font>
<br>
<font face="sans-serif" size="2">#14 0x0000000000000000 in ?? ()</font>
<br>
<font face="sans-serif" size="2">(gdb)</font>
<br>
<br>
<font face="sans-serif" size="2">Can someone help me to understand
the
issue?</font>
<br>
<font face="sans-serif" size="2">Thanks,</font>
<br>
<font face="sans-serif" size="2">Myriam.</font>
<br>
<br>
<br>
<br>
<font face="sans-serif" size="2">HERE IS the environment:</font>
<br>
<br>
<font face="sans-serif" size="2">LD_PRELOAD=/home_nfs/botallam/install/darshan.4/lib/libdarshan.so</font>
<br>
<br>
<font face="sans-serif" size="2">The Darshan library was
configured and
generated with environment variable OMPI_CC=gcc</font>
<br>
<font face="sans-serif" size="2">using mpicc: Open MPI 1.6.2
(Language:
C)</font>
<br>
<br>
<font face="sans-serif" size="2"># which mpicc</font>
<br>
<font face="sans-serif" size="2">/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc</font>
<br>
<font face="sans-serif" size="2"># CC=mpicc CFLAGS=-g ./configure
--prefix=/home_nfs/botallam/install/darshan.4
--with-mem-align=16 --with-log-path-by-env=DARSHAN_LOGPATH
--with-jobid-env=SLURM_JOBID</font>
<br>
<br>
<font face="sans-serif" size="2"># ldd
/home_nfs/botallam/install/darshan.4/lib/libdarshan.so</font>
<br>
<font face="sans-serif" size="2"> linux-vdso.so.1
=> (0x00007fffe9e93000)</font>
<br>
<font face="sans-serif" size="2"> libdl.so.2
=> /lib64/libdl.so.2 (0x00007f8a753bd000)</font>
<br>
<font face="sans-serif" size="2"> libpthread.so.0
=> /lib64/libpthread.so.0 (0x00007f8a751a0000)</font>
<br>
<font face="sans-serif" size="2"> librt.so.1
=> /lib64/librt.so.1 (0x00007f8a74f98000)</font>
<br>
<font face="sans-serif" size="2"> libz.so.1
=> /lib64/libz.so.1 (0x00007f8a74d83000)</font>
<br>
<font face="sans-serif" size="2"> libmpi.so.1
=>
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1(0x00007f8a74967000)</font>
<br>
<font face="sans-serif" size="2"> libm.so.6
=> /lib64/libm.so.6 (0x00007f8a746e3000)</font>
<br>
<font face="sans-serif" size="2"> libnuma.so.1
=> /usr/lib64/libnuma.so.1 (0x00007f8a744db000)</font>
<br>
<font face="sans-serif" size="2"> libnsl.so.1
=> /lib64/libnsl.so.1 (0x00007f8a742c1000)</font>
<br>
<font face="sans-serif" size="2"> libutil.so.1
=> /lib64/libutil.so.1 (0x00007f8a740be000)</font>
<br>
<font face="sans-serif" size="2"> libc.so.6
=> /lib64/libc.so.6 (0x00007f8a73d2e000)</font>
<br>
<font face="sans-serif" size="2">
/lib64/ld-linux-x86-64.so.2
(0x00000036b7400000)</font>
<br>
<font face="sans-serif" size="2"> libimf.so
=>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libimf.so
(0x00007f8a73871000)</font>
<br>
<font face="sans-serif" size="2"> libsvml.so
=>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libsvml.so
(0x00007f8a72fa3000)</font>
<br>
<font face="sans-serif" size="2"> libgcc_s.so.1
=> /lib64/libgcc_s.so.1 (0x00007f8a72d8d000)</font>
<br>
<font face="sans-serif" size="2"> libintlc.so.5
=>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libintlc.so.5
(0x00007f8a72b3e000)</font>
<br>
<font face="sans-serif" size="2">#</font>
<br>
<font face="sans-serif" size="2">#</font>
<br>
<font face="sans-serif" size="2"># nm
/home_nfs/botallam/install/darshan.4/lib/libdarshan.so|grep
mpi</font>
<br>
<font face="sans-serif" size="2">0000000000227020 B
__real_ncmpi_close</font>
<br>
<font face="sans-serif" size="2">0000000000227010 B
__real_ncmpi_create</font>
<br>
<font face="sans-serif" size="2">0000000000227018 B
__real_ncmpi_open</font>
<br>
<font face="sans-serif" size="2">00000000000048cc T
darshan_mpi_initialize</font>
<br>
<font face="sans-serif" size="2">00000000000179d2 T ncmpi_close</font>
<br>
<font face="sans-serif" size="2">000000000001763c T ncmpi_create</font>
<br>
<font face="sans-serif" size="2">0000000000017807 T ncmpi_open</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_byte</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_char</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_comm_world</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_double</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_info_null</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_int</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_long</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_op_land</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_op_lor</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_op_max</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_op_null</font>
<br>
<font face="sans-serif" size="2">
U ompi_mpi_op_sum</font>
<br>
<font face="sans-serif" size="2">0000000000016444 T
resolve_mpi_symbols</font>
<br>
<font face="sans-serif" size="2">#</font>
<br>
<br>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Darshan-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.gov</a>
<a class="moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a>
</pre>
</blockquote>
<br>
</body>
</html>