<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Junchao,<div class=""><br class=""></div><div class="">Thank you for your efforts.</div><div class="">We tried petsc-3.13.0 but it made no difference.</div><div class="">We think now the issue are with sysctl parameters, and increasing those seemed to have cleared up the problem.</div><div class="">This also most likely explains how different clusters had different behaviors with our test code.</div><div class=""><br class=""></div><div class="">We are now running our code and will report back once we are sure that there are no further issues.</div><div class=""><br class=""></div><div class="">Thanks again for your help.</div><div class=""><br class=""></div><div class="">Randy M.</div><div class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 17, 2020, at 8:09 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 16, 2020 at 11:13 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Randy,<div class=""> I reproduced your error with petsc-3.12.4 and 5120 mpi ranks. I also found the error went away with petsc-3.13. However, I have not figured out what is the bug and which commit fixed it :).</div><div class=""> So at your side, it is better to use the latest petsc.<br clear="all" class=""></div></div></blockquote><div class="">I want to add that even with petsc-3.12.4 the error is random. I was only able to reproduce the error once, so I can not claim petsc-3.13 actually fixed it (or, the bug is really in petsc).</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 16, 2020 at 9:06 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Randy,<div class=""> Up to now I could not reproduce your error, even with the biggest mpirun -n 5120 ./test -nsubs 320 -nx 100 -ny 100 -nz 100</div><div class=""> While I continue doing test, you can try other options. It looks you want to duplicate a vector to subcomms. I don't think you need the two lines:</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">call AOApplicationToPetsc(aoParent,nis,ind1,ierr)<br class="">call AOApplicationToPetsc(aoSub,nis,ind2,ierr)</blockquote><div style="line-height:21px" class=""><div class=""> In addition, you can use simpler and more memory-efficient index sets. There is a petsc example for this task, see case 3 in <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c" target="_blank" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c</a></div></div><div class=""> BTW, it is good to use petsc master so we are on the same page.</div><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 15, 2020 at 10:28 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Junchao,<div class=""><br class=""></div><div class="">So I was able to create a small test code that duplicates the issue we have been having, and it is attached to this email in a zip file.</div><div class="">Included is the test.F90 code, the commands to duplicate crash and to duplicate a successful run, output errors, and our petsc configuration.</div><div class=""><br class=""></div><div class="">Our findings to date include:</div><div class=""><br class=""></div><div class=""><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">The error is reproducible in a very short time with this script</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:11pt" class="">It is related to nproc*nsubs and (although to a less extent) to DM grid size</span></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class=""><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">It happens regardless of MPI implementation (mpich, intel mpi 2018, 2019, openmpi) or compiler (gfortran/gcc , intel 2018)<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">No effect changing vecscatter_type to mpi1 or mpi3. Mpi1 seems to slightly increase the limit, but still fails on the full machine set.<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif" class="">Nothing looks interesting on valgrind</div><div class=""><br class=""></div><div class="">Our initial tests were carried out on an Azure cluster, but we also tested on our smaller cluster, and we found the following:</div><div class=""><br class=""></div><div class=""><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">Works:<u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">$PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1280 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100<span style="background-color:black" class=""><u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><span style="color:rgb(31,73,125)" class=""> </span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><span style="color:rgb(31,73,125)" class="">Crashes (this works on Azure)<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">$PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2560 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100</div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">So it looks like it may also be related to the physical number of nodes as well.</div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt" class="">In any case, even with 2560 processes on 192 cores the memory does not go above 3.5 Gbyes so you don’t need a huge cluster to test.</div></div><div class=""><br class=""></div><div class="">Thanks,</div></div><div class=""><br class=""></div><div class="">Randy M.</div><div class=""><br class=""></div><div class=""></div></div><div class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 14, 2020, at 12:23 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">There is an MPI_Allreduce in PetscGatherNumberOfMessages, that is why I doubted it was the problem. Even if users configure petsc with 64-bit indices, we use PetscMPIInt in MPI calls. So it is not a problem.<div class="">Try -vecscatter_type mpi1 to restore to the original VecScatter implementation. If the problem still remains, could you provide a test example for me to debug?</div><div class=""><div class=""><br class=""></div><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 14, 2020 at 12:13 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Junchao,<div class=""><br class=""></div><div class="">We have tried your two suggestions but the problem remains.</div><div class="">And the problem seems to be on the MPI_Isend line 117 in PetscGatherMessageLengths and not MPI_AllReduce.</div><div class=""><br class=""></div><div class="">We have now tried Intel MPI, Mpich, and OpenMPI, and so are thinking the problem must be elsewhere and not MPI.</div><div class=""><br class=""></div><div class="">Give that this is a 64 bit indices build of PETSc, is there some possible incompatibility between PETSc and MPI calls?</div><div class=""><br class=""></div><div class="">We are open to any other possible suggestions to try as other than valgrind on thousands of processes we seem to have run out of ideas.</div><div class=""><br class=""></div><div class="">Thanks, Randy M.</div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 13, 2020, at 8:54 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 10:53 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Randy,</div><div class=""> Someone reported similar problem before. It turned out an Intel MPI MPI_Allreduce bug. A workaround is setting the environment variable I_MPI_ADJUST_ALLREDUCE=1.arr</div></div></blockquote><div class=""> Correct: I_MPI_ADJUST_ALLREDUCE=1</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""> But you mentioned mpich also had the error. So maybe the problem is not the same. So let's try the workaround first. If it doesn't work, add another petsc option -build_twosided allreduce, which is a workaround for Intel MPI_Ibarrier bugs we met.</div> Thanks.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 10:38 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Dear PETSc users,<div class=""><br class=""></div><div class="">We are trying to understand an issue that has come up in running our code on a large cloud cluster with a large number of processes and subcomms.</div><div class="">This is code that we use daily on multiple clusters without problems, and that runs valgrind clean for small test problems.</div><div class=""><br class=""></div><div class="">The run generates the following messages, but doesn’t crash, just seems to hang with all processes continuing to show activity:</div><div class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #1 PetscGatherMessageLengths() line 117 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/sys/utils/mpimesg.c<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #2 VecScatterSetUp_SF() line 658 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/impls/sf/vscatsf.c<u class=""></u><u class=""></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif" class=""><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)" class="">[492]PETSC ERROR: #3 VecScatterSetUp() line 209 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscatfce.c<u class=""></u><u class=""></u></span></div><div class=""><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt" class="">[492]PETSC ERROR: #4 VecScatterCreate() line 282 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscreate.c</span></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Looking at line 117 in PetscGatherMessageLengths we find the offending statement is the MPI_Isend:</div><div class=""><br class=""></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="color:rgb(31,73,125)" class=""> </span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> /* Post the Isends with the message length-info */</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> for (i=0,j=0; i<size; ++i) {</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> if (ilengths[i]) {</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> ierr = MPI_Isend((void*)(ilengths+i),1,MPI_INT,i,tag,comm,s_waits+j);CHKERRQ(ierr);</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> j++;</span><u class=""></u><u class=""></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif" class=""><span style="font-size:11pt;font-family:"Courier New";color:rgb(31,73,125)" class=""> }</span><u class=""></u><u class=""></u></div><div class=""><span style="color:rgb(31,73,125);font-family:"Courier New";font-size:11pt" class=""> }</span> </div><div class=""><br class=""></div><div class="">We have tried this with Intel MPI 2018, 2019, and mpich, all giving the same problem.</div><div class=""><br class=""></div><div class="">We suspect there is some limit being set on this cloud cluster on the number of file connections or something, but we don’t know.</div><div class=""><br class=""></div><div class="">Anyone have any ideas? We are sort of grasping for straws at this point.</div><div class=""><br class=""></div><div class="">Thanks, Randy M.</div></div></blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></body></html>