Microsoft (R) Windows Debugger Version 6.2.9200.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\mpiexec.dmp]
User Mini Dump File with Full Memory: Only application data is available

0:000> ~*k

.  0  Id: 864.718 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0013fdb8 000007f9`6f00159b ntdll!NtClose+0xa
00000000`0013fdc0 00000001`4000195c KERNELBASE!CloseHandle+0x17
00000000`0013fdf0 00000001`400a92b5 mpiexec+0x195c
00000000`0013ff10 000007f9`702c167e mpiexec+0xa92b5
00000000`0013ff60 000007f9`71e93501 kernel32!BaseThreadInitThunk+0x1a
00000000`0013ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   1  Id: 864.4c0 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`01417d68 000007f9`6f005848 ntdll!NtReadFile+0xa
00000000`01417d70 00000001`4004feed KERNELBASE!ReadFile+0x74
00000000`01417df0 000007f9`702c167e mpiexec+0x4feed
00000000`0141ff60 000007f9`71e93501 kernel32!BaseThreadInitThunk+0x1a
00000000`0141ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

#  2  Id: 864.d0c Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0131ff28 000007f9`71f290c0 ntdll!DbgBreakPoint
00000000`0131ff30 000007f9`702c167e ntdll!DbgUiRemoteBreakin+0x34
00000000`0131ff60 000007f9`71e93501 kernel32!BaseThreadInitThunk+0x1a
00000000`0131ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d


------------------------------------------------------------------------------
There are only 3 threads running on mpiexec, out of which the first 2
are of interest.

The first thread calls CloseHandle(), the second calls ReadFile().
------------------------------------------------------------------------------

0:000> lm vmmpiexec
start             end                 module name
00000001`40000000 00000001`4016e000   mpiexec    (export symbols)       mpiexec.exe
    Loaded symbol image file: mpiexec.exe
    Image path: C:\Program Files\MPICH2\bin\mpiexec.exe
    Image name: mpiexec.exe
    Timestamp:        Tue Jun 21 06:28:15 2011 (4E001DDF)
    CheckSum:         001321F5
    ImageSize:        0016E000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        0.0 Unknown
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

------------------------------------------------------------------------------
This is mpiexec.exe compiled in 2011, June 21.
It belongs to mpich2-1.4-win-x86-64.msi.
------------------------------------------------------------------------------

0:002> ~0s
ntdll!NtClose+0xa:
000007f9`71e72cda c3              ret
0:000> k
Child-SP          RetAddr           Call Site
00000000`0013fdb8 000007f9`6f00159b ntdll!NtClose+0xa
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mpiexec.exe - 
00000000`0013fdc0 00000001`4000195c KERNELBASE!CloseHandle+0x17
00000000`0013fdf0 00000001`400a92b5 mpiexec+0x195c
00000000`0013ff10 000007f9`702c167e mpiexec+0xa92b5
00000000`0013ff60 000007f9`71e93501 kernel32!BaseThreadInitThunk+0x1a
00000000`0013ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:000> u 00000001`4000195c
mpiexec+0x195c:
00000001`4000195c 488b0d95741100  mov     rcx,qword ptr [mpiexec+0x118df8 (00000001`40118df8)]
00000001`40001963 bab80b0000      mov     edx,0BB8h
00000001`40001968 ff15da280c00    call    qword ptr [mpiexec+0xc4248 (00000001`400c4248)]
00000001`4000196e 85c0            test    eax,eax
00000001`40001970 7412            je      mpiexec+0x1984 (00000001`40001984)
00000001`40001972 488b0d7f741100  mov     rcx,qword ptr [mpiexec+0x118df8 (00000001`40118df8)]

0:000> ln poi(00000001`400c4248)
(000007f9`702c102c)   kernel32!WaitForSingleObject   |  (000007f9`702c1bc8)   kernel32!WaitForSingleObjectEx
Exact matches:
    kernel32!WaitForSingleObject (<no parameter info>)
0:000> ? bb8
Evaluate expression: 3000 = 00000000`00000bb8

-----------------------------------------------------------------------------
We switch the context to the first thread and look at what follows
after CloseHandle(). WaitForSingleObject(, timeout = 3000) is next.

We can also find out what kind of handle is passed to CloseHandle():
-----------------------------------------------------------------------------

0:000> ub 00000001`4000195c
mpiexec+0x1932:
00000001`40001932 ff1528290c00    call    qword ptr [mpiexec+0xc4260 (00000001`400c4260)]
00000001`40001938 488b0db1741100  mov     rcx,qword ptr [mpiexec+0x118df0 (00000001`40118df0)]
00000001`4000193f 4c3925b2741100  cmp     qword ptr [mpiexec+0x118df8 (00000001`40118df8)],r12
00000001`40001946 7450            je      mpiexec+0x1998 (00000001`40001998)
00000001`40001948 b9f6ffffff      mov     ecx,0FFFFFFF6h
00000001`4000194d ff1505290c00    call    qword ptr [mpiexec+0xc4258 (00000001`400c4258)]
00000001`40001953 488bc8          mov     rcx,rax
00000001`40001956 ff15f4280c00    call    qword ptr [mpiexec+0xc4250 (00000001`400c4250)]
0:000> ln poi(00000001`400c4258)
(000007f9`702c1d98)   kernel32!GetStdHandleStub   |  (000007f9`702c3054)   kernel32!GetProcessIdStub
Exact matches:
    kernel32!GetStdHandleStub (<no parameter info>)
0:000> .formats -0n10
Evaluate expression:
  Hex:     ffffffff`fffffff6
  Decimal: -10
  Octal:   1777777777777777777766
  Binary:  11111111 11111111 11111111 11111111 11111111 11111111 11111111 11110110
  Chars:   ........
  Time:    ***** Invalid
  Float:   low -1.#QNAN high -1.#QNAN
  Double:  -1.#QNAN

----------------------------------------------------------------------------
Based on these traces, CloseHandle(handle = GetStdHandle(-10 = STD_INPUT_HANDLE)

Let's move to thread 1 and mine the data:
----------------------------------------------------------------------------

0:000> ~1s
ntdll!NtReadFile+0xa:
000007f9`71e72c4a c3              ret
0:001> k
Child-SP          RetAddr           Call Site
00000000`01417d68 000007f9`6f005848 ntdll!NtReadFile+0xa
00000000`01417d70 00000001`4004feed KERNELBASE!ReadFile+0x74
00000000`01417df0 000007f9`702c167e mpiexec+0x4feed
00000000`0141ff60 000007f9`71e93501 kernel32!BaseThreadInitThunk+0x1a
00000000`0141ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:001> ub 00000001`4004feed l0n30
mpiexec+0x4fe66:
00000001`4004fe66 ff15d4440700    call    qword ptr [mpiexec+0xc4340 (00000001`400c4340)]
00000001`4004fe6c 85c0            test    eax,eax
00000001`4004fe6e 7535            jne     mpiexec+0x4fea5 (00000001`4004fea5)
00000001`4004fe70 ff15fa430700    call    qword ptr [mpiexec+0xc4270 (00000001`400c4270)]
00000001`4004fe76 488d542440      lea     rdx,[rsp+40h]
00000001`4004fe7b 4533c9          xor     r9d,r9d
00000001`4004fe7e 8bc8            mov     ecx,eax
00000001`4004fe80 41b800010000    mov     r8d,100h
00000001`4004fe86 8bd8            mov     ebx,eax
00000001`4004fe88 e8f373ffff      call    mpiexec+0x47280 (00000001`40047280)
00000001`4004fe8d 4c8d442440      lea     r8,[rsp+40h]
00000001`4004fe92 488d0d07010a00  lea     rcx,[mpiexec+0xeffa0 (00000001`400effa0)]
00000001`4004fe99 8bd3            mov     edx,ebx
00000001`4004fe9b e8a078ffff      call    mpiexec+0x47740 (00000001`40047740)
00000001`4004fea0 e9f3010000      jmp     mpiexec+0x50098 (00000001`40050098)
00000001`4004fea5 4885ed          test    rbp,rbp
00000001`4004fea8 0f84de010000    je      mpiexec+0x5008c (00000001`4005008c)
00000001`4004feae 4883fdff        cmp     rbp,0FFFFFFFFFFFFFFFFh
00000001`4004feb2 0f84d4010000    je      mpiexec+0x5008c (00000001`4005008c)
00000001`4004feb8 4533e4          xor     r12d,r12d
00000001`4004febb 4c8d4c2430      lea     r9,[rsp+30h]
00000001`4004fec0 488d942440010000 lea     rdx,[rsp+140h]
00000001`4004fec8 458d442401      lea     r8d,[r12+1]
00000001`4004fecd 418bdc          mov     ebx,r12d
00000001`4004fed0 488bcd          mov     rcx,rbp
00000001`4004fed3 4489642430      mov     dword ptr [rsp+30h],r12d
00000001`4004fed8 498bfc          mov     rdi,r12
00000001`4004fedb 4c89642420      mov     qword ptr [rsp+20h],r12
00000001`4004fee0 889c2440010000  mov     byte ptr [rsp+140h],bl
00000001`4004fee7 ff1573440700    call    qword ptr [mpiexec+0xc4360 (00000001`400c4360)]
0:001> da 00000001`400effa0
00000001`400effa0  "smpd_stdin_thread:ReleaseMutex(h"
00000001`400effc0  "LaunchProcessMutex) failed: Erro"
00000001`400effe0  "r %d, %s."

--------------------------------------------------------------------------
ReadFile(rcx = rbp, rdx = rsp+140, r8 = 1, r9 = rsp+30, qword ptr[rsp+20] = 0);

We have found a string in the function "smpd_stdin_thread:ReleaseMutex(hLaunchProcessMutex) failed: Error %d, %s."
which will be a clue in the source code.

We disassemble further back to find out the value of rbp. rbp represents
the handle passed to ReadFile().
--------------------------------------------------------------------------

0:001> ub 00000001`4004feed l0n40
mpiexec+0x4fe31:
00000001`4004fe31 4c8d442440      lea     r8,[rsp+40h]
00000001`4004fe36 488d0db3010a00  lea     rcx,[mpiexec+0xefff0 (00000001`400efff0)]
00000001`4004fe3d 8bd3            mov     edx,ebx
00000001`4004fe3f e8fc78ffff      call    mpiexec+0x47740 (00000001`40047740)
00000001`4004fe44 e957020000      jmp     mpiexec+0x500a0 (00000001`400500a0)
00000001`4004fe49 b9f6ffffff      mov     ecx,0FFFFFFF6h
00000001`4004fe4e 4889ac2480810000 mov     qword ptr [rsp+8180h],rbp
00000001`4004fe56 ff15fc430700    call    qword ptr [mpiexec+0xc4258 (00000001`400c4258)]
00000001`4004fe5c 488b0d4d900c00  mov     rcx,qword ptr [mpiexec+0x118eb0 (00000001`40118eb0)]
00000001`4004fe63 488be8          mov     rbp,rax
.
.
.
0:001> ln poi(00000001`400c4258)
(000007f9`702c1d98)   kernel32!GetStdHandleStub   |  (000007f9`702c3054)   kernel32!GetProcessIdStub
Exact matches:
    kernel32!GetStdHandleStub (<no parameter info>)

--------------------------------------------------------------------------
This is a race between:
CloseHandle(STD_INPUT_HANDLE) and ReadFile(STD_INPUT_HANDLE).