[mpich-discuss] Hydra invalid pointer when HYDRA_BOOTSTRAP_EXEC set

[Steve Sutphen]

I am trying to get mpich2-1.3.1 working so that I can set system wide
environment variables that make it easier for the user community to use
the system.  To that end I would like to set HYDRA_BOOTSTRAP_EXEC=rsh
(which works on our cluster) but if I set that environment variable
then mpiexec fails on exit (i.e. it does the computation and 
then catches an invalid pointer being passed to munmap_chunk).
If I unset the HYDRA_BOOTSTRAP_EXEC variable and add "-bootstrap rsh"
to the mpiexec parameters it exits clean.

I also downloaded and built mpich2-1.3.2rc1 to see if the problem was 
fixed, but I got the same result.

The system is an older Fedora Core release 5 (Bordeaux), but the 
build did not show any errors so I did not expect that there would be
a problem.

Thank you,
	steve.

Here is the output that I see when it fails:

:mpich2-1.3.1# mpiexec -n 6 ./examples/cpi
Process 2 of 6 is on eureka002
Process 4 of 6 is on eureka003
Process 0 of 6 is on eureka001
Process 1 of 6 is on eureka001
Process 5 of 6 is on eureka003
Process 3 of 6 is on eureka002
pi is approximately 3.1415926544231243, Error is 0.0000000008333312
wall clock time = 0.043777
*** glibc detected *** mpiexec: munmap_chunk(): invalid pointer: 0x00007fffb0b89d3a ***
======= Backtrace: =========
/lib64/libc.so.6(__libc_free+0x17a)[0x3f6ed6d58a]
mpiexec[0x424abb]
mpiexec[0x409000]
mpiexec[0x4058b3]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3f6ed1c784]
mpiexec[0x4042d9]
======= Memory map: ========
00400000-0044e000 r-xp 00000000 03:03 377538       /opt/Mpich2/bin/mpiexec.hydra
0054e000-00550000 rw-p 0004e000 03:03 377538       /opt/Mpich2/bin/mpiexec.hydra
00550000-005bb000 rw-p 00550000 00:00 0            [heap]
3f6eb00000-3f6eb1a000 r-xp 00000000 03:01 367489   /lib64/ld-2.4.so
3f6ec19000-3f6ec1a000 r--p 00019000 03:01 367489   /lib64/ld-2.4.so
3f6ec1a000-3f6ec1b000 rw-p 0001a000 03:01 367489   /lib64/ld-2.4.so
3f6ed00000-3f6ee3f000 r-xp 00000000 03:01 367490   /lib64/libc-2.4.so
3f6ee3f000-3f6ef3e000 ---p 0013f000 03:01 367490   /lib64/libc-2.4.so
3f6ef3e000-3f6ef42000 r--p 0013e000 03:01 367490   /lib64/libc-2.4.so
3f6ef42000-3f6ef43000 rw-p 00142000 03:01 367490   /lib64/libc-2.4.so
3f6ef43000-3f6ef48000 rw-p 3f6ef43000 00:00 0 
3f6f000000-3f6f080000 r-xp 00000000 03:01 367491   /lib64/libm-2.4.so
3f6f080000-3f6f17f000 ---p 00080000 03:01 367491   /lib64/libm-2.4.so
3f6f17f000-3f6f180000 r--p 0007f000 03:01 367491   /lib64/libm-2.4.so
3f6f180000-3f6f181000 rw-p 00080000 03:01 367491   /lib64/libm-2.4.so
3f6f200000-3f6f204000 r-xp 00000000 03:01 99847    /usr/lib64/libnuma.so.1
3f6f204000-3f6f304000 ---p 00004000 03:01 99847    /usr/lib64/libnuma.so.1
3f6f304000-3f6f305000 rw-p 00004000 03:01 99847    /usr/lib64/libnuma.so.1
3f6fc00000-3f6fc14000 r-xp 00000000 03:01 49089    /usr/lib64/libz.so.1.2.3
3f6fc14000-3f6fd13000 ---p 00014000 03:01 49089    /usr/lib64/libz.so.1.2.3
3f6fd13000-3f6fd14000 rw-p 00013000 03:01 49089    /usr/lib64/libz.so.1.2.3
3f6fe00000-3f6fe12000 r-xp 00000000 03:01 368614   /lib64/libpthread-2.4.so
3f6fe12000-3f6ff12000 ---p 00012000 03:01 368614   /lib64/libpthread-2.4.so
3f6ff12000-3f6ff13000 r--p 00012000 03:01 368614   /lib64/libpthread-2.4.so
3f6ff13000-3f6ff14000 rw-p 00013000 03:01 368614   /lib64/libpthread-2.4.so
3f6ff14000-3f6ff18000 rw-p 3f6ff14000 00:00 0 
3f71600000-3f7160d000 r-xp 00000000 03:01 367492   /lib64/libgcc_s-4.1.1-20060525.so.1
3f7160d000-3f7170d000 ---p 0000d000 03:01 367492   /lib64/libgcc_s-4.1.1-20060525.so.1
3f7170d000-3f7170e000 rw-p 0000d000 03:01 367492   /lib64/libgcc_s-4.1.1-20060525.so.1
3f74000000-3f74013000 r-xp 00000000 03:01 368552   /lib64/libnsl-2.4.so
3f74013000-3f74113000 ---p 00013000 03:01 368552   /lib64/libnsl-2.4.so
3f74113000-3f74114000 r--p 00013000 03:01 368552   /lib64/libnsl-2.4.so
3f74114000-3f74115000 rw-p 00014000 03:01 368552   /lib64/libnsl-2.4.so
3f74115000-3f74117000 rw-p 3f74115000 00:00 0 
3f75100000-3f7522b000 r-xp 00000000 03:01 66445    /usr/lib64/libxml2.so.2.6.23
3f7522b000-3f7532a000 ---p 0012b000 03:01 66445    /usr/lib64/libxml2.so.2.6.23
3f7532a000-3f75337000 rw-p 0012a000 03:01 66445    /usr/lib64/libxml2.so.2.6.23
3f75337000-3f75338000 rw-p 3f75337000 00:00 0 
3f75800000-3f75807000 r-xp 00000000 03:01 368615   /lib64/librt-2.4.so
3f75807000-3f75906000 ---p 00007000 03:01 368615   /lib64/librt-2.4.so
3f75906000-3f75907000 r--p 00006000 03:01 368615   /lib64/librt-2.4.so
3f75907000-3f75908000 rw-p 00007000 03:01 368615   /lib64/librt-2.4.so
2aaaaaaab000-2aaaaaaac000 rw-p 2aaaaaaab000 00:00 0 
2aaaaaacd000-2aaaaaad0000 rw-p 2aaaaaacd000 00:00 0 
2aaaaaad0000-2aaaaaad2000 r-xp 00000000 03:01 367648 /lib64/libdl-2.4.so
2aaaaaad2000-2aaaaabd2000 ---p 00002000 03:01 367648 /lib64/libdl-2.4.so
2aaaaabd2000-2aaaaabd3000 r--p 00002000 03:01 367648 /lib64/libdl-2.4.so
2aaaaabd3000-2aaaaabd4000 rw-p 00003000 03:01 367648 /lib64/libdl-2.4.so
2aaaaabd4000-2aaaaabd6000 rw-p 2aaaaabd4000 00:00 0 
2aaaaabd6000-2aaaaabe0000 r-xp 00000000 03:01 367886 /lib64/libnss_files-2.4.so
2aaaaabe0000-2aaaaacdf000 ---p 0000a000 03:01 367886 /lib64/libnss_files-2.4.so
2aaaaacdf000-2aaaaace0000 r--p 00009000 03:01 367886 /lib64/libnss_files-2.4.so
2aaaaace0000-2aaaaace1000 rw-p 0000a000 03:01 367886 /lib64/libnss_files-2.4.so
7fffb0b75000-7fffb0b8b000 rw-p 7fffb0b75000 00:00 0  [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
Aborted