<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I paste below my email thread with Pavan on this problem. What you
are seeing looks very similar to me.<br>
<br>
Except that in my case mpiexec returned an error; in your case its
hanging ???<br>
<br>
- Mike<br>
<br>
<br>
<div class="moz-forward-container">
-------- Original Message --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Subject:
</th>
<td>Re: Need help with MPICH2 launch problem</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Date: </th>
<td>Fri, 30 Nov 2012 02:53:11 -0600</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">From: </th>
<td>Pavan Balaji <a class="moz-txt-link-rfc2396E" href="mailto:balaji@mcs.anl.gov"><balaji@mcs.anl.gov></a></td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">To: </th>
<td>Michael Wilde <a class="moz-txt-link-rfc2396E" href="mailto:wilde@mcs.anl.gov"><wilde@mcs.anl.gov></a></td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">CC: </th>
<td>Justin M Wozniak <a class="moz-txt-link-rfc2396E" href="mailto:wozniak@mcs.anl.gov"><wozniak@mcs.anl.gov></a>, Mihael
Hategan <a class="moz-txt-link-rfc2396E" href="mailto:hategan@mcs.anl.gov"><hategan@mcs.anl.gov></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>Ok, I figured out a way to do this. I've committed a few changes into
the mpich trunk to cover this case. The test program you sent works
correctly now. Can you try the latest trunk version (>= r10701)?
-- Pavan
On 11/30/2012 02:17 AM US Central Time, Pavan Balaji wrote:
> Hi Mike,
>
> I'm able to reproduce this error.
>
> The issue is that, within mpiexec, I don't have a good way of knowing
> that STDIN has been closed (in which case I should disable stdin
> forwarding). I can check STDIN_FILENO, but there's no guarantee that
> it'll point to stdin, if stdin is closed. In this case, a new socket
> can have the same file descriptor.
>
> I'll look into this some more tomorrow, but I don't see an easy way to
> do this.
>
> -- Pavan
>
> On 11/29/2012 05:43 PM US Central Time, Michael Wilde wrote:
>>
>>> Hydra should automatically detect a closed fd 0; you shouldn't have to
>>> do anything special in Swift for this.
>>
>> I *thought* it did, but only on mpich2 on login.mcs.anl.gov. But now my latest test is failing there too (I kept distilling down to a tinier test, so maybe I perturbed something?)
>>
>> It seems to fail consistently on two local mvapich2's (fusion and midway) and likely on eureka (Im unsure which mpich I used there).
>>
>>> Do you have a test program that
>>> demonstrates this problem?
>>
>> Yes, attached.
>>
>> tar xf mpifd0.tar
>> cd mpifd0.tar
>> ./runme.sh
>>
>> Should give these results (all on login hosts, not cluster nodes):
>>
>> === login.mcs.anl.gov
>>
>> mcs$ which mpiexec
>> /usr/bin/mpiexec
>> mcs$ mpiexec --version | head -3
>> HYDRA build details:
>> Version: 1.4.1
>> Release Date: Wed Aug 24 14:40:04 CDT 2011
>> mcs$ ./runme.sh
>>
>> testing with fd 0 open:
>>
>> ./tinympi: rank is 1
>> ./tinympi: rank is 0
>>
>> Testing with fd 0 closed:
>>
>> [mpiexec@login4] HYDT_dmx_register_fd (./tools/demux/demux.c:98): registering duplicate fd 0
>> [mpiexec@login4] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:561): unable to register fd
>> [mpiexec@login4] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec@login4] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>> [mpiexec@login4] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>> mcs$
>>
>> === midway.rcc.uchicago.edu:
>>
>> mid$ which mpiexec
>> /software/mvapich2-1.8-el6-x86_64/bin/mpiexec
>> mid$ ./runme.sh
>>
>> testing with fd 0 open:
>>
>> ./tinympi: rank is 0
>> ./tinympi: rank is 1
>>
>> Testing with fd 0 closed:
>>
>> [mpiexec@midway-login1] HYDT_dmx_register_fd (./tools/demux/demux.c:98): registering duplicate fd 0
>> [mpiexec@midway-login1] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:563): unable to register fd
>> [mpiexec@midway-login1] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec@midway-login1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>> [mpiexec@midway-login1] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>> mid$
>>
>> === fusion.lcrc.anl.gov:
>>
>> fusion$ which mpiexec
>> /soft/mvapich2/1.4.1-gcc-4.1.2-r2/bin/mpiexec
>> fusion$ ./runme.sh
>>
>> testing with fd 0 open:
>>
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
>> ./tinympi: rank is 1
>> ./tinympi: rank is 0
>>
>> Testing with fd 0 closed:
>>
>> [mpiexec@flogin2] HYDT_dmx_register_fd (./tools/demux/demux.c:98): registering duplicate fd 0
>> [mpiexec@flogin2] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:487): unable to register fd
>> [mpiexec@flogin2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec@flogin2] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:206): error waiting for event
>> [mpiexec@flogin2] main (./ui/mpich/mpiexec.c:404): process manager error waiting for completion
>> fusion$
>>
>>
>>
>>
>>
>>>
>>> -- Pavan
>>>
>>> On 11/29/2012 04:44 PM US Central Time, Michael Wilde wrote:
>>>> Hi Pavan,
>>>>
>>>> I think Ive resolved this. It seems indeed due to the fact that
>>>> Swift was closing stdin (fd 0) in one of its execution modes
>>>> ("coasters"). I thought this was the problem early on, but got
>>>> misled because I commented out the close in the wrong swift module.
>>>>
>>>> The problem seems similar to this issue:
>>>>
>>>> <a class="moz-txt-link-freetext" href="http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-February/009018.html">http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-February/009018.html</a>
>>>> <a class="moz-txt-link-freetext" href="http://trac.mpich.org/projects/mpich/ticket/1029">http://trac.mpich.org/projects/mpich/ticket/1029</a>
>>>>
>>>> When I leave fd 0 open to /dev/null before exec()'ing mpiexec, (as
>>>> you suggest in the posting above) my mpiexec works fine.
>>>>
>>>> Without this (ie with fd 0 closed) it fails on eureka, on uchicago
>>>> and fusion mvapich2, but works on the mpich2 running on mcs login
>>>> (which looks like hydra 1.4.1)
>>>>
>>>> I think we're OK now, with the above fix in Swift, but I still need
>>>> to test on Eureka when it comes back up.
>>>>
>>>> - Mike
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Pavan Balaji" <a class="moz-txt-link-rfc2396E" href="mailto:balaji@mcs.anl.gov"><balaji@mcs.anl.gov></a>
>>>>> To: "Michael Wilde" <a class="moz-txt-link-rfc2396E" href="mailto:wilde@mcs.anl.gov"><wilde@mcs.anl.gov></a>
>>>>> Cc: "Justin M Wozniak" <a class="moz-txt-link-rfc2396E" href="mailto:wozniak@mcs.anl.gov"><wozniak@mcs.anl.gov></a>, "Sheri Mickelson"
>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:mickelso@mcs.anl.gov"><mickelso@mcs.anl.gov></a>, "Jayesh Krishna"
>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:jayesh@mcs.anl.gov"><jayesh@mcs.anl.gov></a>, "Robert Jacob" <a class="moz-txt-link-rfc2396E" href="mailto:jacob@mcs.anl.gov"><jacob@mcs.anl.gov></a>, "Mihael
>>>>> Hategan" <a class="moz-txt-link-rfc2396E" href="mailto:hategan@mcs.anl.gov"><hategan@mcs.anl.gov></a>
>>>>> Sent: Monday, November 26, 2012 12:37:00 PM
>>>>> Subject: Re: Need help with MPICH2 launch problem
>>>>> Hi Mike,
>>>>>
>>>>> I'm at Argonne this week. Did you want to sync up on this?
>>>>>
>>>>> -- Pavan
>>>>>
>>>>> On 11/11/2012 09:53 AM US Central Time, Michael Wilde wrote:
>>>>>> OK, thanks, Pavan - this can wait till you get back. I'll try to
>>>>>> turn on some MPICH debugging and isolate it to a simple test case
>>>>>> that you or anyone on the MPI team can reproduce.
>>>>>>
>>>>>> - Mike
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Pavan Balaji" <a class="moz-txt-link-rfc2396E" href="mailto:balaji@mcs.anl.gov"><balaji@mcs.anl.gov></a>
>>>>>>> To: "Michael Wilde" <a class="moz-txt-link-rfc2396E" href="mailto:wilde@mcs.anl.gov"><wilde@mcs.anl.gov></a>
>>>>>>> Cc: "Justin M Wozniak" <a class="moz-txt-link-rfc2396E" href="mailto:wozniak@mcs.anl.gov"><wozniak@mcs.anl.gov></a>, "Sheri Mickelson"
>>>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:mickelso@mcs.anl.gov"><mickelso@mcs.anl.gov></a>, "Jayesh Krishna"
>>>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:jayesh@mcs.anl.gov"><jayesh@mcs.anl.gov></a>, "Robert Jacob" <a class="moz-txt-link-rfc2396E" href="mailto:jacob@mcs.anl.gov"><jacob@mcs.anl.gov></a>, "Mihael
>>>>>>> Hategan" <a class="moz-txt-link-rfc2396E" href="mailto:hategan@mcs.anl.gov"><hategan@mcs.anl.gov></a>
>>>>>>> Sent: Saturday, November 10, 2012 3:56:32 PM
>>>>>>> Subject: Re: Need help with MPICH2 launch problem
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> Unfortunately, this is too little information for me to tell
>>>>>>> what's
>>>>>>> going on. Perhaps it's best to sit together and look at it. I'm
>>>>>>> at
>>>>>>> SC.
>>>>>>>
>>>>>>> -- Pavan
>>>>>>>
>>>>>>> On 11/09/2012 02:14 PM, Michael Wilde wrote:
>>>>>>>>
>>>>>>>> Hi Pavan,
>>>>>>>>
>>>>>>>> We are having a problem running the MPI app "Pagoda" (a parallel
>>>>>>>> netcdf processing tool) under Swift for Rob Jacob's ParVis
>>>>>>>> project.
>>>>>>>>
>>>>>>>> It seems that when we do an MPICH2 launch on Eureka under
>>>>>>>> Swift's
>>>>>>>> worker-node agents, we break something that MPICH is expecting
>>>>>>>> in
>>>>>>>> the environment (in the general sense) in which mpiexec is being
>>>>>>>> run.
>>>>>>>>
>>>>>>>> Ive check the env variables, and *think* that we are passing
>>>>>>>> everything on to mpiexec without damage. Im more suspicious of
>>>>>>>> having damaged a file descriptor or done something to break ssh
>>>>>>>> connectivity, etc.
>>>>>>>>
>>>>>>>> What I get from MPICH2 mpiexec is:
>>>>>>>>
>>>>>>>>>> [mpiexec@vs37] HYDT_dmx_register_fd
>>>>>>>>>> (./tools/demux/demux.c:82):
>>>>>>>>>> registering duplicate fd 0
>>>>>>>>>> [mpiexec@vs37] HYDT_bscd_external_launch_procs
>>>>>>>>>> (./tools/bootstrap/external/external_launch.c:295): demux
>>>>>>>>>> returned
>>>>>>>>>> error registering fd
>>>>>>>>>> [mpiexec@vs37] HYDT_bsci_launch_procs
>>>>>>>>>> (./tools/bootstrap/src/bsci_launch.c:21): bootstrap device
>>>>>>>>>> returned
>>>>>>>>>> error while launching processes
>>>>>>>>>> [mpiexec@vs37] HYD_pmci_launch_procs
>>>>>>>>>> (./pm/pmiserv/pmiserv_pmci.c:298): bootstrap server cannot
>>>>>>>>>> launch
>>>>>>>>>> processes
>>>>>>>>>> [mpiexec@vs37] main (./ui/mpich/mpiexec.c:298): process
>>>>>>>>>> manager
>>>>>>>>>> returned error launching processes
>>>>>>>>
>>>>>>>> I think the failure is related to something that either perl, or
>>>>>>>> our
>>>>>>>> worker.pl perl code (which is Swift's worker-node execution
>>>>>>>> agent,
>>>>>>>> soft of like Swift's "Hydra"), is doing when it forks the job. I
>>>>>>>> thought the culprit was that worker.pl closes STDIN, but
>>>>>>>> commenting
>>>>>>>> out that close doesnt correct the problem.
>>>>>>>>
>>>>>>>> I'll continue to isolate what the difference is between running
>>>>>>>> mpiexec under Swift vs running under, say, multiple levels of
>>>>>>>> nested
>>>>>>>> shell on the "lead" worker node, but if you have some
>>>>>>>> suggestions,
>>>>>>>> Pavan, as to what the above error is telling us that MPI doesnt
>>>>>>>> like, that would be a great help!
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> - Mike
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pavan Balaji
>>>>>>> <a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a>
>>>>>>
>>>>>
>>>>> --
>>>>> Pavan Balaji
>>>>> <a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a>
>>>>
>>>
>>> --
>>> Pavan Balaji
>>> <a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a>
>>
>
--
Pavan Balaji
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a>
</pre>
<br>
</div>
<br>
<br>
<div class="moz-cite-prefix">On 5/8/14, 8:33 PM, Michael Wilde
wrote:<br>
</div>
<blockquote cite="mid:536C3084.2090607@anl.gov" type="cite">Perhaps
try redirecting stdin on the $turbine call with "< /dev/null"
<br>
<br>
Some older versions of mpiexec had a bug when stdin was not a tty
or ptty.
<br>
<br>
Not sure if thats the case here, but easy to try.
<br>
<br>
- Mike
<br>
<br>
On 5/8/14, 5:10 PM, Ketan Maheshwari wrote:
<br>
<blockquote type="cite">Hi,
<br>
<br>
Trying invoke turbine from Galaxy as follows:
<br>
<br>
$turbine -V -n $n $wdir/script.tcl "${swiftargs}"
<br>
<br>
Each of the variables being set in previous lines.
<br>
<br>
The process tree shows mpiexec as defunct processes:
<br>
PID TTY TIME CMD
<br>
389 ? 00:00:00 sshd
<br>
390 pts/39 00:00:00 bash
<br>
804 pts/39 00:00:00 sh
<br>
990 pts/39 00:00:38 python
<br>
1189 pts/39 00:00:00 sh
<br>
1190 pts/39 00:00:00 bash
<br>
1330 pts/39 00:00:00 turbine
<br>
1332 pts/39 00:00:00 mpiexec <defunct>
<br>
1547 pts/39 00:00:00 sh
<br>
1548 pts/39 00:00:00 bash
<br>
1683 pts/39 00:00:00 turbine
<br>
1686 pts/39 00:00:00 mpiexec <defunct>
<br>
2046 pts/39 00:00:00 sh
<br>
2047 pts/39 00:00:00 bash
<br>
2299 pts/39 00:00:00 turbine
<br>
2302 pts/39 00:00:00 mpiexec <defunct>
<br>
<br>
<br>
Not sure how to debug this. The same call works outside of
Galaxy.
<br>
<br>
Any suggestions?
<br>
<br>
Thanks,
<br>
Ketan
<br>
<br>
<br>
<br>
_______________________________________________
<br>
ExM-user mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:ExM-user@lists.mcs.anl.gov">ExM-user@lists.mcs.anl.gov</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/exm-user">https://lists.mcs.anl.gov/mailman/listinfo/exm-user</a>
<br>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
ExM-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ExM-user@lists.mcs.anl.gov">ExM-user@lists.mcs.anl.gov</a>
<a class="moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/exm-user">https://lists.mcs.anl.gov/mailman/listinfo/exm-user</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</body>
</html>