[mpich-discuss] MPD daemons disconnect/die unexpectedly

[Michael Ahlmann]

I am running a 4-node cluster using Centos 5 and dual-core Athlon 
chips.  I installed MPICH2 for parallel computations.  I can use MPDBOOT 
to setup my daemons, and an MPDTRACE confirms that all nodes are 
present.  MPDRINGTEST also runs as expected.  However, during some jobs, 
the MPD's die unexpectedly.  I originally thought this was caused by a 
network connectivity (maybe firewall?) issue, but if I run a single 
daemon on the head node (using MPD &), the same thing happens (but only 
sometimes).  The problem seems rather sporadic as on some occasions the 
cluster runs the same job for days on end without problems, and at other 
times, the problem occurs within minutes.  Any help in figuring out how 
to debug this problem would be highly appreciated.  Thanks!

-Michael