<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Junchao,<div class=""><br class=""></div><div class="">I have no objection if you want to add the test code to the petsc repository.</div><div class=""><br class=""></div><div class="">In the meantime, we will check out the changes you’ve made and let you know what we find.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Randy M.</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 27, 2020, at 9:59 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Randy,<div class=""> You are absolutely right. The AOApplicationToPetsc could not be removed. Since the excessive communication is inevitable, I made two changes in petsc to ease that. One is I skewed the communication to let each rank send to ranks greater than itself first. The other is an option, -max_pending_isend, to control number of pending isends. Current default is 512.</div><div class=""> I have an MR at <a href="https://gitlab.com/petsc/petsc/-/merge_requests/2757" class="">https://gitlab.com/petsc/petsc/-/merge_requests/2757</a>. I tested it dozens of times with your example at 5120 ranks. It worked fine.</div><div class=""> Please try it in your environment and let me know the result. Since the failure is random, you may need to run multiple times.</div><div class=""><br class=""></div><div class=""> BTW, if no objection, I'd like to add your excellent example to petsc repo.</div><div class=""><br class=""></div><div class=""> Thanks</div><div class=""><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 24, 2020 at 5:32 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Hi Junchao,<div class=""><br class=""></div><div class="">I tested by commenting out the AOApplicationToPetsc calls as you suggest, but it doesn’t work because it doesn’t maintain the proper order of the elements in the scattered vectors.</div><div class=""><br class=""></div><div class="">I attach a modified version of the test code where I put elements into the global vector, then carry out the scatter, and check on the subcomms that they are correct.</div><div class=""><br class=""></div><div class="">You can see everything is fine with the AOApplicationToPetsc calls, but the comparison fails when those are commented out.</div><div class=""><br class=""></div><div class="">If there is some way I can achieve the right VecScatters without those calls, I would be happy to know how to do that.</div><div class=""><br class=""></div><div class="">Thank you again for your help.</div><div class=""><br class=""></div><div class="">Randy</div><div class=""><br class=""></div><div class="">ps. I suggest you run this test with nx=ny=nz=10 and only a couple subcomms and maybe 4 processes to demonstrate the behavior</div></div><div style="overflow-wrap: break-word;" class=""><div class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 20, 2020, at 2:45 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hello, Randy,<div class=""> I further looked at the problem and believe it was due to overwhelming traffic. The code sometimes fails at MPI_Waitall. I printed out MPI error strings of bad MPI Statuses. One of them is like "MPID_nem_tcp_connpoll(1845): Communication error with rank 25: Connection reset by peer", which is a tcp error and has nothing to do with petsc.</div><div class=""> Further investigation shows in the case of 5120 ranks with 320 sub communicators, during VecScatterSetUp<span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:13.3333px" class="">, </span>each rank has around 640 isends/irecvs neighbors, and quite a few ranks has 1280 isends neighbors. I guess these overwhelming isends occasionally crashed the connection.</div><div class=""> The piece of code in VecScatterSetUp is to calculate the communication pattern. With index sets "having good locality", the calculate itself incurs less traffic. Here good locality means indices in an index set mostly point to local entries. However, the AOApplicationToPetsc() call in your code unnecessarily ruined the good petsc ordering. If we remove AOApplicationToPetsc() (the vecscatter result is still correct) , then each rank uniformly has around 320 isends/irecvs.</div><div class=""> So, test with this modification and see if it really works in your environment. If not applicable, we can provide options in petsc to carry out the communication in phases to avoid flooding the network (though it is better done by MPI). </div><div class=""><br clear="all" class=""><div class=""><div dir="ltr" class=""><div class=""> Thanks.</div><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 17, 2020 at 10:47 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Junchao,<div class=""><br class=""></div><div class="">Thank you for your efforts.</div><div class="">We tried petsc-3.13.0 but it made no difference.</div><div class="">We think now the issue are with sysctl parameters, and increasing those seemed to have cleared up the problem.</div><div class="">This also most likely explains how different clusters had different behaviors with our test code.</div><div class=""><br class=""></div><div class="">We are now running our code and will report back once we are sure that there are no further issues.</div><div class=""><br class=""></div><div class="">Thanks again for your help.</div><div class=""><br class=""></div><div class="">Randy M.</div><div class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 17, 2020, at 8:09 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 16, 2020 at 11:13 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Randy,<div class=""> I reproduced your error with petsc-3.12.4 and 5120 mpi ranks. I also found the error went away with petsc-3.13. However, I have not figured out what is the bug and which commit fixed it :).</div><div class=""> So at your side, it is better to use the latest petsc.<br clear="all" class=""></div></div></blockquote><div class="">I want to add that even with petsc-3.12.4 the error is random. I was only able to reproduce the error once, so I can not claim petsc-3.13 actually fixed it (or, the bug is really in petsc).</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 16, 2020 at 9:06 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Randy,<div class=""> Up to now I could not reproduce your error, even with the biggest mpirun -n 5120 ./test -nsubs 320 -nx 100 -ny 100 -nz 100</div><div class=""> While I continue doing test, you can try other options. It looks you want to duplicate a vector to subcomms. I don't think you need the two lines:</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">call AOApplicationToPetsc(aoParent,nis,ind1,ierr)<br class="">call AOApplicationToPetsc(aoSub,nis,ind2,ierr)</blockquote><div style="line-height:21px" class=""><div class=""> In addition, you can use simpler and more memory-efficient index sets. There is a petsc example for this task, see case 3 in <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c" target="_blank" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c</a></div></div><div class=""> BTW, it is good to use petsc master so we are on the same page.</div><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 15, 2020 at 10:28 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Junchao,<div class=""><br class=""></div><div class="">So I was able to create a small test code that duplicates the issue we have been having, and it is attached to this email in a zip file.</div><div class="">Included is the test.F90 code, the commands to duplicate crash and to duplicate a successful run, output errors, and our petsc configuration.</div><div class=""><br class=""></div><div class="">Our findings to date include:</div><div class=""><br class=""></div><div class=""><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">The error is reproducible in a very short time with this script</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:11pt" class="">It is related to nproc*nsubs and (although to a less extent) to DM grid size</span></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class=""><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">It happens regardless of MPI implementation (mpich, intel mpi 2018, 2019, openmpi) or compiler (gfortran/gcc , intel 2018)<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">No effect changing vecscatter_type to mpi1 or mpi3. Mpi1 seems to slightly increase the limit, but still fails on the full machine set.<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">Nothing looks interesting on valgrind</div><div class=""><br class=""></div><div class="">Our initial tests were carried out on an Azure cluster, but we also tested on our smaller cluster, and we found the following:</div><div class=""><br class=""></div><div class=""><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">Works:<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">$PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1280 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100<span style="background-color:black" class=""><u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><span style="color:rgb(31,73,125)" class=""> </span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><span style="color:rgb(31,73,125)" class="">Crashes (this works on Azure)<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">$PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2560 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100</div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">So it looks like it may also be related to the physical number of nodes as well.</div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">In any case, even with 2560 processes on 192 cores the memory does not go above 3.5 Gbyes so you don’t need a huge cluster to test.</div></div><div class=""><br class=""></div><div class="">Thanks,</div></div><div class=""><br class=""></div><div class="">Randy M.</div><div class=""><br class=""></div><div class=""></div></div><div class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 14, 2020, at 12:23 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">There is an MPI_Allreduce in PetscGatherNumberOfMessages, that is why I doubted it was the problem. Even if users configure petsc with 64-bit indices, we use PetscMPIInt in MPI calls. So it is not a problem.<div class="">Try -vecscatter_type mpi1 to restore to the original VecScatter implementation. If the problem still remains, could you provide a test example for me to debug?</div><div class=""><div class=""><br class=""></div><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 14, 2020 at 12:13 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Junchao,<div class=""><br class=""></div><div class="">We have tried your two suggestions but the problem remains.</div><div class="">And the problem seems to be on the MPI_Isend line 117 in PetscGatherMessageLengths and not MPI_AllReduce.</div><div class=""><br class=""></div><div class="">We have now tried Intel MPI, Mpich, and OpenMPI, and so are thinking the problem must be elsewhere and not MPI.</div><div class=""><br class=""></div><div class="">Give that this is a 64 bit indices build of PETSc, is there some possible incompatibility between PETSc and MPI calls?</div><div class=""><br class=""></div><div class="">We are open to any other possible suggestions to try as other than valgrind on thousands of processes we seem to have run out of ideas.</div><div class=""><br class=""></div><div class="">Thanks, Randy M.</div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 13, 2020, at 8:54 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 10:53 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Randy,</div><div class=""> Someone reported similar problem before. It turned out an Intel MPI MPI_Allreduce bug. A workaround is setting the environment variable I_MPI_ADJUST_ALLREDUCE=1.arr</div></div></blockquote><div class=""> Correct: I_MPI_ADJUST_ALLREDUCE=1</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""> But you mentioned mpich also had the error. So maybe the problem is not the same. So let's try the workaround first. If it doesn't work, add another petsc option -build_twosided allreduce, which is a workaround for Intel MPI_Ibarrier bugs we met.</div> Thanks.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 10:38 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Dear PETSc users,<div class=""><br class=""></div><div class="">We are trying to understand an issue that has come up in running our code on a large cloud cluster with a large number of processes and subcomms.</div><div class="">This is code that we use daily on multiple clusters without problems, and that runs valgrind clean for small test problems.</div><div class=""><br class=""></div><div class="">The run generates the following messages, but doesn’t crash, just seems to hang with all processes continuing to show activity:</div><div class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #1 PetscGatherMessageLengths() line 117 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/sys/utils/mpimesg.c<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #2 VecScatterSetUp_SF() line 658 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/impls/sf/vscatsf.c<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #3 VecScatterSetUp() line 209 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscatfce.c<u class=""></u><u class=""></u></span></div><div class=""><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt" class="">[492]PETSC ERROR: #4 VecScatterCreate() line 282 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscreate.c</span></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Looking at line 117 in PetscGatherMessageLengths we find the offending statement is the MPI_Isend:</div><div class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="color:rgb(31,73,125)" class=""> </span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> /* Post the Isends with the message length-info */</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> for (i=0,j=0; i<size; ++i) {</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> if (ilengths[i]) {</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> ierr = MPI_Isend((void*)(ilengths+i),1,MPI_INT,i,tag,comm,s_waits+j);CHKERRQ(ierr);</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> j++;</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> }</span><u class=""></u><u class=""></u></div><div class=""><span style="color:rgb(31,73,125);font-family:"Courier New";font-size:11pt" class=""> }</span> </div><div class=""><br class=""></div><div class="">We have tried this with Intel MPI 2018, 2019, and mpich, all giving the same problem.</div><div class=""><br class=""></div><div class="">We suspect there is some limit being set on this cloud cluster on the number of file connections or something, but we don’t know.</div><div class=""><br class=""></div><div class="">Anyone have any ideas? We are sort of grasping for straws at this point.</div><div class=""><br class=""></div><div class="">Thanks, Randy M.</div></div></blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></body></html>