<div dir="ltr"><div style="line-height:21px">From my reading, the code actually does not need multiple tags. You can just let _get_tags() return a constant (say 0), or use your modulo MPI_TAG_UB approach. <div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre"><span style="color:rgb(175,0,219)"><br></span></div><div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre"><span style="color:rgb(175,0,219)">541 for</span> (i = <span style="color:rgb(9,134,88)">0</span>; i < np; ++i) <span style="color:rgb(121,94,38)">PetscCallMPI</span>(<span style="color:rgb(121,94,38)">MPI_Isend</span>(&<span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">messages_to_be_sent</span>[i], <span style="color:rgb(9,134,88)">1</span>, MPIU_INT, <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">neighbour_procs</span>[i], <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">send_tags</span>[i], <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">comm</span>, &<span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">_requests</span>[i]));</div><div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre"><span style="color:rgb(175,0,219)">542 for</span> (i = <span style="color:rgb(9,134,88)">0</span>; i < np; ++i) <span style="color:rgb(121,94,38)">PetscCallMPI</span>(<span style="color:rgb(121,94,38)">MPI_Irecv</span>(&<span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">messages_to_be_recvieved</span>[i], <span style="color:rgb(9,134,88)">1</span>, MPIU_INT, <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">neighbour_procs</span>[i], <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">recv_tags</span>[i], <span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">comm</span>, &<span style="color:rgb(0,16,128)">de</span>-><span style="color:rgb(0,16,128)">_requests</span>[np + i]));</div><div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre"><br></div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 22, 2022 at 11:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Tue, Nov 22, 2022 at 11:23 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I don't understand why you need so many tags. Is the communication pattern actually MPI_Alltoallv, but you implemented it in MPI_Send/Recv?</div></blockquote><div><br></div><div>I am preserving the original design from Dave until we do a more thorough rewrite. I think he is using a different tag for each pair of processes to</div><div>make debugging easier.</div><div><br></div><div>I don't think Alltoallv is appropriate most of the time. If you had a lot of particles with a huge spread of velocities then you could get that, but most</div><div>scenarios I think look close to nearest neighbor.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 21, 2022 at 2:37 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In data_ex.c, Swarm uses a distinct tag for each pair of processes. If the number of processes exceeds 1024, there are > 1024^2 tags which exceeds MPI_TAG_UB on Intel MPI.<div><br></div><div>My solution is going to be to use that process pair number modulo MPI_TAG_UB. Does anyone have a slicker suggestion?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt<br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>