<html><head><base href="x-msg://1095/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">[moving Edric's message over to mpich-discuss@ before responding]<br><div><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>From: </b></span><span style="font-family:'Helvetica'; font-size:medium;">Edric Ellis <<a href="mailto:Edric.Ellis@mathworks.co.uk">Edric.Ellis@mathworks.co.uk</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>Date: </b></span><span style="font-family:'Helvetica'; font-size:medium;">August 17, 2010 10:41:08 AM CDT<br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>To: </b></span><span style="font-family:'Helvetica'; font-size:medium;">"<a href="mailto:mpich2-maint@mcs.anl.gov">mpich2-maint@mcs.anl.gov</a>" <<a href="mailto:mpich2-maint@mcs.anl.gov">mpich2-maint@mcs.anl.gov</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>Subject: </b></span><span style="font-family:'Helvetica'; font-size:medium;"><b>[mpich2-maint] Relative ordering of MPI_Iprobe()s and MPI_Barrier()s</b><br></span></div><br><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-GB" link="blue" vlink="purple"><div class="Section1" style="page: Section1; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">Hi mpich2-maint,<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">We’re in the process of moving to MPICH2-1.2.1p1 using the SMPD/nemesis variant (on Linux only - on Windows, we’re waiting for a fix to ticket #895), and we’ve found a discrepancy in the behaviour compared to the sock variant. I’m not sure if this is a real bug, or if I’ve missed something in the MPI standard. Our test for our wrapper around MPI_Barrier() essentially proceeds as follows (see attached for a C test case which usually shows this problem when running with 10 processes). Each process does this:<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 36pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -18pt; "><span>1.<span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span>Call MPI_Send() to each other process in turn with a tiny payload (assuming that this will be sent in the “eager” mode).<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 36pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -18pt; "><span>2.<span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span>MPI_Barrier()<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 36pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -18pt; "><span>3.<span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span>Check that MPI_Iprobe() indicates a message ready to receive from each other process<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">With the sock variant, this works as I expect - each process gets a return from MPI_Iprobe indicating that there is indeed a message waiting from each other process. With nemesis, this isn’t always the case - sometimes multiple calls to MPI_Iprobe are required. (Could this be related to ticket #1062?).<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">I couldn’t see in the MPI standard where the “expected” behaviour of the above might be specified, but it’s possible that I’ve missed something.<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">I can see several options for where a problem might exist:<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 36pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -18pt; "><span>1.<span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span>MPI doesn’t actually specify that these MPI_Iprobe()s should definitely return “true”<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 36pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -18pt; "><span>2.<span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span>The nemesis channel isn’t preserving the ordering between MPI_Barrier() and pt2pt communications in the way I expect<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">As it happens, our usage of MPI_Iprobe() is basically restricted to our test code, so we could modify our tests not to rely on the old behaviour, but we’d like to understand better where the problem is.<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">Cheers,<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; ">Edric.<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div></div></div></span></blockquote></div></body></html>