<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">If ranks are uniquely listed in neighbour_procs, then you only need 1 fresh tag per communication round from PetscCommGetNewTag<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 23, 2022, at 9:08 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" class="">dave.mayhem23@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 23 Nov 2022 at 08:57, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="line-height:21px" class="">From my reading, the code actually does not need multiple tags. You can just let _get_tags() return a constant (say 0), or use your modulo MPI_TAG_UB approach.</div></div></blockquote><div class=""><br class=""></div><div class="">Yes I believe that is correct.</div><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="line-height:21px" class=""> <div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre-wrap;" class=""><span style="color:rgb(175,0,219)" class=""><br class=""></span></div><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre-wrap;" class=""><span style="color:rgb(175,0,219)" class="">541 for</span> (i = <span style="color:rgb(9,134,88)" class="">0</span>; i < np; ++i) <span style="color:rgb(121,94,38)" class="">PetscCallMPI</span>(<span style="color:rgb(121,94,38)" class="">MPI_Isend</span>(&<span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">messages_to_be_sent</span>[i], <span style="color:rgb(9,134,88)" class="">1</span>, MPIU_INT, <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">neighbour_procs</span>[i], <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">send_tags</span>[i], <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">comm</span>, &<span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">_requests</span>[i]));</div><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre-wrap;" class=""><span style="color:rgb(175,0,219)" class="">542 for</span> (i = <span style="color:rgb(9,134,88)" class="">0</span>; i < np; ++i) <span style="color:rgb(121,94,38)" class="">PetscCallMPI</span>(<span style="color:rgb(121,94,38)" class="">MPI_Irecv</span>(&<span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">messages_to_be_recvieved</span>[i], <span style="color:rgb(9,134,88)" class="">1</span>, MPIU_INT, <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">neighbour_procs</span>[i], <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">recv_tags</span>[i], <span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">comm</span>, &<span style="color:rgb(0,16,128)" class="">de</span>-><span style="color:rgb(0,16,128)" class="">_requests</span>[np + i]));</div><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre-wrap;" class=""><br class=""></div></div><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 22, 2022 at 11:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Tue, Nov 22, 2022 at 11:23 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I don't understand why you need so many tags. Is the communication pattern actually MPI_Alltoallv, but you implemented it in MPI_Send/Recv?</div></blockquote><div class=""><br class=""></div><div class="">I am preserving the original design from Dave until we do a more thorough rewrite. I think he is using a different tag for each pair of processes to</div><div class="">make debugging easier.</div><div class=""><br class=""></div><div class="">I don't think Alltoallv is appropriate most of the time. If you had a lot of particles with a huge spread of velocities then you could get that, but most</div><div class="">scenarios I think look close to nearest neighbor.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 21, 2022 at 2:37 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">In data_ex.c, Swarm uses a distinct tag for each pair of processes. If the number of processes exceeds 1024, there are > 1024^2 tags which exceeds MPI_TAG_UB on Intel MPI.<div class=""><br class=""></div><div class="">My solution is going to be to use that process pair number modulo MPI_TAG_UB. Does anyone have a slicker suggestion?</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt<br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></body></html>