[mpich-discuss] Hydra issues

[Pavan Balaji]

> Ok, I was not patient enough. If I let it run, it eventually starts. The 
> actual walltime is nearly the same as when I use proxies, but the stdout 
> is delayed until the application completes.

Hydra itself does not do any buffering of stdout/stderr as it comes. The 
problem is that hydra never gets the stdout/stderr events till the 
application process flushes stdout/stderr (or libc's internally 
buffering is full). In fact, this is the same problem with mpd as well. 
For example, you can try a simple program which does

for (5 loops) { printf("foo\n"); sleep(1); }

... and you'll notice that the stdout doesn't get printed till the end 
(both with hydra and mpd). Instead, if you change the above test to:

for (5 loops) { printf("foo\n"); fflush(stdout); sleep(1); }

... this should show the output as it appears.

We, obviously, can't expect users to go around adding fflush() in their 
code, so this is not a good solution. Guillaume @ INRIA and I had tried 
some experiments with this, but there wasn't any solution that we could 
use to work around it. Maybe it's time to look into this some more.

  -- Pavan

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji