[mpich-discuss] MPI and threaded communication

[Pavan Balaji]

Joey,

The problem you are trying to solve is identical to what another 
application developer (for mpiBLAST) was attempting some time ago. The 
solution used was something like this:

Thread 1:
while (1) {
	MPI_Recv(ANY_SOURCE, ANY_TAG);
	if (TAG == SPECIAL_TAG) /* This is a cancel message */
		quit();
	else
		enqueue_msg();
}

Thread 2:
MPI_Irecv(SPECIAL_TAG);
while (1) {
	dequeue_msg();
	MPI_Isend();
	MPI_Waitall();
	if (tag == SPECIAL_TAG)
		quit();
}

Thread 3:
/* When it wants to kill the two threads */
for i = 0 to 1 {
	MPI_Send(SELF, SPECIAL_TAG);
	/* Each thread will receive one of these two messages */
}

There might be small errors in the logic (it's 5am here, and I've run 
out of coffee!), but the general idea should be there.

  -- Pavan

Joseph Gonzalez wrote, On 02/03/2009 04:22 AM:
> Hi Again,
> 
> Thanks for addressing my earlier question concerning thread and
> MPICH2.  As part of our multithreaded system we are creating a
> callback based communicator with buffered send and receive queues.
> This seems like an obvious goal however I am having trouble
> terminating pending receives at the end of execution.
> 
> All in the same MPI Process (pseudo-code) :
> 
> Thread 1
> while(alive) {
>   msg <- MPI_Recv(Any_Origin, Any_type)
>   enqueue msg to handlers
> }
> 
> 
> Thread 2
> while(alive) {
>   msg <- dequeue outbound message
>   MPI_Send(msg, msg.dest, msg.type)
> }
> 
> At some point an additional thread sets alive to false and would like
> to force the blocking Send and Recv to return (maybe with an error).
> How can I terminate the blocking sends and receives? I tried using
> MPI_ISend and MPI_IRecv followed with a MPI_Wait and again neither
> awake from the MPI_Wait when MPI_Cancel is invoked.  From the
> specification it is clear how MPI_Cancel behaves before MPI_Wait is
> invoked but not after.  We are introducing this abstraction because
> our algorithm will dynamically determine which nodes it needs to
> communicate with at runtime and it will be difficult to directly
> interleave algorithm and communication events.
> 
> Thank you,
> Joey
> 
> 
> 
> On Sun, Feb 1, 2009 at 6:15 PM, Joseph Gonzalez <jegonzal at cs.cmu.edu> wrote:
>> Hi,
>>
>> We are building a threaded, event driven C++ wrapper for MPICH2.  We would
>> like to be able to simultaneously send and receive messages from multiple
>> threads in the same process.  If two threads simultaneous try to send to two
>> different rank destinations (on some remote machine) will they both transmit
>> simultaneously or is their an internal mutex that will prohibit this from
>> happening?
>>
>> Thanks,
>> Joey
>>
> 
> 
> 

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji