<div dir="ltr">Min,<div> I suggest MPICH add tests to play with the maximal MPI tag (through attribute MPI_TAG_UB). </div><div> PETSc uses tags from the maximal and downwards. I guess MPICH tests use small tags. That is why the bug only showed up with PETSc.<br><div class="gmail_extra"><br clear="all"><div><div class="m_-2405801840177781509gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 3:58 PM, Min Si <span dir="ltr"><<a href="mailto:msi@anl.gov" target="_blank">msi@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
Thanks for narrowing down the problem. I checked the MPICH code and believe this is a bug in MPICH. I just created a PR to fix it:<br>
<a href="https://github.com/pmodels/mpich/pull/3097" rel="noreferrer" target="_blank">https://github.com/pmodels/mpi<wbr>ch/pull/3097</a><br>
<br>
It should be merged into MPICH master branch soon.<br>
<br>
Thanks,<br>
Min<div class="m_-2405801840177781509HOEnZb"><div class="m_-2405801840177781509h5"><br>
<br>
On 2018/04/17 14:10, Eric Chamberland wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
are we talking about the "tag" passed to MPI_Isend for example?<br>
<br>
but does that mean there is something to change for any MPI call which involves tags usage or is it only a PETSc "bad" tag usage?<br>
<br>
thanks Satish for your finding!<br>
<br>
Eric<br>
<br>
On 16/04/18 11:31 PM, Satish Balay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, 13 Mar 2018, Eric Chamberland wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
each night we are testing mpich/master with our petsc-based code. I don't<br>
know if PETSc team is doing the same thing with mpich/master? (Maybe it is a<br>
good idea?)<br>
<br>
Everything was fine (except the issue<br>
<a href="https://github.com/pmodels/mpich/issues/2892" rel="noreferrer" target="_blank">https://github.com/pmodels/mpi<wbr>ch/issues/2892</a>) up to commit 7b8d64debd, but<br>
since commit mpich:a8a2b30fd21), I have a segfault on a any parallel nightly<br>
test.<br>
</blockquote>
<br>
I attempted a bisect of the above range of commits - and narrowed down to:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
db11d4c4a70e39a28be88ed32f0054<wbr>2301699e08 is the first bad commit<br>
<<<<<<<<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
balay@asterix /home/balay/soft/build/mpich ((db11d4c4a...)|BISECTING)<br>
$ git show db11d4c4a70e39a28be88ed32f0054<wbr>2301699e08<br>
commit db11d4c4a70e39a28be88ed32f0054<wbr>2301699e08 (HEAD, refs/bisect/bad)<br>
Author: Ken Raffenetti <<a href="mailto:raffenet@mcs.anl.gov" target="_blank">raffenet@mcs.anl.gov</a>><br>
Date: Thu Feb 15 11:37:59 2018 -0600<br>
<br>
init: Fix tag upper limit initialization<br>
The starting point for this value is equivalent to the usable tag bits<br>
macro. This value should be set before device initialization,<br>
otherwise devices will assume they have more bits than are actually<br>
available.<br>
Signed-off-by: Wesley Bland <<a href="mailto:wesley.bland@intel.com" target="_blank">wesley.bland@intel.com</a>><br>
<br>
diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c<br>
index cbc41f4d5..b31ae2f07 100644<br>
--- a/src/mpi/init/initthread.c<br>
+++ b/src/mpi/init/initthread.c<br>
@@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int required, int *provided)<br>
MPIR_Process.attrs.host = MPI_PROC_NULL;<br>
<a href="http://MPIR_Process.attrs.io" rel="noreferrer" target="_blank">MPIR_Process.attrs.io</a> = MPI_PROC_NULL;<br>
MPIR_Process.attrs.lastusedcod<wbr>e = MPI_ERR_LASTCODE;<br>
- MPIR_Process.attrs.tag_ub = 0;<br>
+ MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS;<br>
MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET;<br>
MPIR_Process.attrs.wtime_is_gl<wbr>obal = 0;<br>
@@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, int required, int *provided)<br>
MPIR_Assert(((unsigned) MPIR_Process.<br>
attrs.tag_ub & ((unsigned) MPIR_Process.attrs.tag_ub + 1)) == 0);<br>
- /* Set aside tag space for tagged collectives and failure notification */<br>
-#ifdef HAVE_TAG_ERROR_BITS<br>
- MPIR_Process.attrs.tag_ub >>= 3;<br>
-#else<br>
- MPIR_Process.attrs.tag_ub >>= 1;<br>
-#endif<br>
-<br>
/* Assert: tag_ub is at least the minimum asked for in the MPI spec */<br>
MPIR_Assert(MPIR_Process.attrs<wbr>.tag_ub >= 32767);<br>
<<<<<<<<<<<<<<<<<<br>
<br>
Reverthing this patch gets mpich-3.3b2 working with petsc<br>
<br>
Satish<br>
<br>
</blockquote></blockquote>
<br>
</div></div></blockquote></div><br></div></div></div>