[mpich-discuss] disable-auto-cleanup send/receive example

[Rob Stewart]

On 03/11/11 03:09, Pavan Balaji wrote:

> Here are some examples:

Thanks. I'm finding it difficult to find many canonical examples of 
fault tolerant mpich2 programs. The 3 examples don't explicitly deal 
with faults.

Pavan: you have recommended I create new communicators. Take a task farm 
example. I'm struggling to identify how exactly I would do this.

Imagine a scenario where a master was distributing work in a round robin 
fashion, and at some point, a remote process becomes unavailable. I'd 
want to handle the MPI_Send gracefully, remove knowledge of this failed 
rank, retry sending the work to another worker, and all future round 
robin work distributions, avoiding this failed rank e.g.

init(){
MPI_Comm_dup(MPI_COMM_WORLD, &comm);
}

send(){
err = MPI_Send(xs, x, MPI_INT, ranki, 0, comm);
  if (err == MPI_SUCCESS){
     /* What to do? */
   send(); // try again
}

In the "what to do", I want to remove the failed rank, "ranki", from the 
communication group, and so next time "send()" is called, the failed 
rank will not be hit by a round robin send in the future.

Darius: Thanks for elaborating on MPI_IProbe. It does work in my example 
as you say, but I cannot find any documentation to say exactly what the 
behaviour of MPI_IProbe is in mpich2. When you say "the MPICH library 
will detect and handle the failed process only when", *how* exactly does 
it handle the failed rank? Remove it from MPI_COMM_WORLD? What else 
could it do?

What is better? To use MPI_IProbe, or to recreate communication groups?

Also, is this bug fixed, which probably relates to my problem
http://trac.mcs.anl.gov/projects/mpich2/ticket/1163

Darius: Lastly, I am somewhat alarmed by your earlier comment: "all 
wildcard (i.e., MPI_ANY_SOURCE) receives will also complete with an 
error". I read this to mean that, for fault tolerant MPICH2 programs, 
one cannot use unspecified senders.

Unspecified senders is fairly common. Load balancing work for instance, 
or task farms, divide and conquer parallelism, map reduce etc etc... And 
for my intended use of mpich2, there is no way for me to be able to 
specify known ranks for sending an receiving. Is there a way to turn 
this behaviour off ?

--
Rob Stewart