[mpich-discuss] "unexpected messages" Question

[Hiatt, Dave M]

Yes, this helps a lot.  And I was thinking in terms of the Barrier, the issue is within a collective operation for sure.  And I did labor under a misconception about how that was being handled.  Thank you.

-----Original Message-----
From: mpich-discuss-bounces at mcs.anl.gov
[mailto:mpich-discuss-bounces at mcs.anl.gov]On Behalf Of Darius Buntinas
Sent: Thursday, January 07, 2010 12:51 PM
To: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] "unexpected messages" Question


Hi Dave,

Receiving too many unexpected messages is a common bug people hit.  This
is often caused by some processes "running ahead" of others.

First, an unexpected message is a message which has been received by the
mpi library for which a receive hasn't been posted (i.e., the program
has not called a receive function like MPI_Recv or MPI_Irecv).  What
happens is that for small messages ("small" being determined by the
particular mpi library and/or interconnect you're using) the library
stores a copy of the message locally.  If you receive enough of these,
you run out of memory.

So let's say you have a program where one process receives a message
then does some computation on it repeatedly in a loop, and another
process sends messages to the other process, also in a loop.  Because
the first process spends some time processing each message it'll
probably run slower than second, and the second process will end up
sending messages faster than the first process can receive them.

You then end up with all of these unexpected messages because the
receiver hasn't been able to post the receives fast enough.  Note that
this can happen even if you're using blocking sends.  Remember that
MPI_Send() returns when the send buffer is free to be reused, and not
necessarily when the receiver has received the message.

Another place where you can run into this problem of unexpected messages
is with collectives.  People often thing of collective operations as
synchronizing operations.  This is not always the case.  Consider
reduce.  A reduction operation is typically performed in a tree fashion,
where each process receives messages from it's children performs the
operation and sends the result to it's parent.  If a process which
happens to be a leaf of the tree calls MPI_Reduce before it's parent
process does, it will result in an unexpected message at the parent.
Note also that the leaf process may return from MPI_Reduce before it's
parent even calls MPI_Reduce.  Now look what happens if you have a loop
with MPI_Reduce in it.  Because the non-leaf nodes have to receive
messages from several children perform a calculation and send the
result, they will run slower than the leaf nodes which only have to send
a single message, so you may end up with a "unexpected message storm"
similar to the one above.

So how can you fix the problem?  See if you can rearrange your code to
get rid of loops like the ones described above.  Otherwise, you'll
probably have to introduce some synchronization between the processes
which may affect performance.  For loops with collectives, you can add
an MPI_Barrier in the loop.  For loops with sends/receives you can use
synchronous sends (MPI_Ssend, and friends) or have the sender wait for
an explicit ack message from the receiver.  Of course you could optimize
this where you're not doing the synchronization in every iteration of
the loop, e.g., call MPI_Barrier every 100th iteration.

I hope this helps some.

Darius



On 01/07/2010 11:27 AM, Hiatt, Dave M wrote:
> I'm following up on an earlier question.  I'm auditing the number of Bcast and Sends I do versus an exception message that is thrown during processing.  The message is saying "261894 unexpected messages queued".  This number is dramatically different that what appears to be the counts of messages the app is sending (I'm counting a Bcast as 1 message). and counting messages being received and sent between node 0 and the compute nodes.  This cluster has 496 total nodes.  When I run on a 60 node cluster I never see any hit of a problem like this.  And the network utilization does not indicate some kind of large congestion, but clearly something is happening.  So I'm assuming it's my app.  To that end a few questions if I might ask:
>
> First question - Is a BCast considered 1 message or will it be N messages where N is the number of active nodes in terms of this kind of count?
> Second question - What constitutes an "unexpected message"?  I am assuming any Send or BCast is expected.  Am I confused on this nomenclature?
> Third question  - I've assumed that the message count being stated in this queue translated directly to the number of calls to MPI::Send and MPI::Bcast calls I make.
>
> I have not been able so far to duplicate this problem on my test clusters (albeit they are much smaller, typically 60 nodes).  And I have no indication of being able to create some kind of "message storm" as it were in some kind of race condition.
>
> Thanks
> dave
>
> "Consequences, Schmonsequences, as long as I'm rich". - Daffy Duck
> Dave Hiatt
> Market Risk Systems Integration
> CitiMortgage, Inc.
> 1000 Technology Dr.
> Third Floor East, M.S. 55
> O'Fallon, MO 63368-2240
>
> Phone:  636-261-1408
> Mobile: 314-452-9165
> FAX:    636-261-1312
> Email:     Dave.M.Hiatt at citigroup.com
>
>
>
>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
_______________________________________________
mpich-discuss mailing list
mpich-discuss at mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss