<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> I May 2019 Lisandro changed the versions of Metis and ParMetis that PETSc uses to use a portable machine independent random number generator so if you are having PETSc install Metis then its random number generator should generate the exact same random numbers on repeated identical runs on the same machine or different machines. If this does not appear to be the case please let us know. PETSc doesn't use random numbers much but when it does they are all portable and machine independent and produce identical results for identical runs.<div class=""><br class=""></div><div class=""> Due to the non-commutativity of floating point arithmetic with parallelism (numbers arrive in different orders on identical runs) the numerical results will differ and hence decisions (iteration convergence etc) will differ so the "results" can vary a great deal in different identical runs. </div><div class=""><br class=""></div><div class=""> We did have a user getting "random" errors only after very long runs that were due to "hardware errors" perhaps due to overheating so it is also possible your problems are not due directly to memory corruption, PETSc or MPI or even OS software. This may be too complicated for your work flow but perhaps if you restricted your jobs to shorter times with a restart mechanism these problems would not appear.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""> Truly random "hardware" errors often produce just wildly crazy numbers, in your report the numbers are off but not absurdly large which points to possible software errors.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 25, 2020, at 12:02 PM, Mark McClure <<a href="mailto:mark@resfrac.com" class="">mark@resfrac.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hello,<div class=""><br class=""></div><div class="">I work with Chris, and have a few comments, hopefully helpful. Thank you all, for your help.</div><div class=""><br class=""></div><div class="">Our program is unfortunately behaving a little bit nondeterministically. I am not sure why because for the OpenMP parts, I test it for race conditions using Intel Inspector and see none. We are using Metis, not Parmetis. Do Petsc or Metis have any behavior that is nondeterministic? We will continue to investigate the cause of the nondeterminism. But that is a challenge for us reproducing this problem because we can run the same simulation 10 times, and we do not get precisely the same result each time. In fact, for this bug, Chris did run it 10 times and did not reproduce.</div><div class=""><br class=""></div><div class="">Every month, 1000s of hours of this simulator are being run on the Azure cluster. This crash has been occurring for months, but at infrequent intervals, and have never been able to reproduce it. As such, we can't generate an error dump, unless we gave all users a Petsc build with no optimization and waited weeks until a problem cropped up. </div><div class=""><br class=""></div><div class="">Here's what we'll do - we'll internally make a build with debug and then run a large number of simulations until the problem reproduces and we get the error dump. That will take us some time, but we'll do it. </div><div class=""><br class=""></div><div class="">Here is a bit more background that might be helpful.
At first, we used OpenMPI 2.1.1-8 with Petsc 3.13.2. With that combo, we observed a memory leak, and simulations failed when the node ran out of RAM. Then we switched to MPICH3.3a2 and Petsc 3.13.3. The memory leak went away. That's when we started seeing this bug "greater than largest key allowed". It was unreproducible, but happening relatively more often than it is now (I think) - I was getting a couple user reports a week. Then, we updated to MPICH 3.3.2 and the same Petsc version (3.13.3). The problem seems to be less common - hadn't happened for the past month. But then it happened four times this week. </div><div class=""><br class=""></div><div class="">Other background to note - our linear system very frequently changes size/shape. There will be 10,000s of Petsc solves with different matrices (different positions of nonzero values) over the course of a simulation. As far as we can tell, the crash only occurs only after the simulator has run for a long time, 10+ hours. Having said that, it does not appear to be a memory leak - at least, the node has plenty of remaining RAM when these crashes are occurring.</div><div class=""><br class=""></div><div class="">Mark</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 24, 2020 at 4:41 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">You might add code here like:<div class=""><br class=""></div><div class="">if (ierr) {</div><div class=""><div class=""> for (; i<aij->B->rmap->n; i++) {</div><div class=""> for ( j<B->ilen[i]; j++) {</div><div class=""> PetscInt data,gid1 = aj[B->i[i] + j] + 1; // don't use ierr</div></div><div class=""> print rank, gid1;</div><div class="">}</div><div class="">CHKERRQ(ierr);<br class=""></div><div class=""><br class=""></div><div class="">I am guessing that somehow you have a table that is bad and too small. It failed on an index not much bigger than the largest key allowed. Maybe just compute the max and see if it goes much larger than the largest key allowed.</div><div class=""><br class=""></div><div class="">If your mesh just changed to you know if it got bigger or smaller...</div><div class=""><br class=""></div><div class="">Anyway just some thoughts,</div><div class="">Mark</div><div class=""><br class=""></div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 24, 2020 at 4:18 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Sep 24, 2020, at 2:47 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Sep 24, 2020 at 3:42 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> The stack is listed below. It crashes inside <span style="color:rgb(14,16,26)" class="">MatPtAP(). </span></div></blockquote><div class=""><br class=""></div><div class="">What about just checking that the column indices that PtAP receives are valid? Are we not doing that?</div></div></div></div></blockquote><div class=""><br class=""></div> The code that checks for column too large in MatSetValues_MPIXXAIJ() is turned off for optimized builds, I am making a MR to always have it on. But I doubt this is the problem, other, more harsh crashes, are likely if the column index is too large.</div><div class=""><br class=""></div><div class=""> This is difficult to debug because all we get is a stack trace. It would be good if we produce some information about the current state of the objects when the error is detected. We should think about what light-weight stuff we could report when errors are detected.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Barry</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><font color="#0e101a" class=""> It is possible there is some subtle bug in the rather complex PETSc code for </font><span style="color:rgb(14,16,26)" class="">MatPtAP</span>() <span style="color:rgb(14,16,26)" class="">but I am included to blame MPI again. </span></div><div class=""><font color="#0e101a" class=""><br class=""></font></div><div class=""><font color="#0e101a" class=""> I think we should add some simple low-overhead always on communication error detecting code to PetscSF where some check sums are also communicated at the highest level of PetscSF(). </font></div><div class=""><font color="#0e101a" class=""><br class=""></font></div><div class=""><font color="#0e101a" class=""> I don't know how but perhaps when the data is packed per destination rank a checksum is computed and when unpacked the checksum is compared using extra space at the end of the communicated packed array to store and send the checksum. Yes, it is kind of odd for a high level library like PETSc to not trust the communication channel but MPI implementations have proven themselves to not be trustworthy and adding this to PetscSF is not intrusive to the PETSc API or user. Plus it gives a definitive yes or no as to the problem being from an error in the communication.</font></div><div class=""><font color="#0e101a" class=""><br class=""></font></div><div class=""><font color="#0e101a" class=""> Barry</font></div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Sep 24, 2020, at 12:35 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Sep 24, 2020 at 1:22 PM Chris Hewson <<a href="mailto:chris@resfrac.com" target="_blank" class="">chris@resfrac.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Guys,<div class=""><br class=""></div><div class="">Thanks for all of the prompt responses, very helpful and appreciated.</div><div class=""><br class=""></div><div class="">By "when debugging", did you mean when configure petsc --with-debugging=1 COPTFLAGS=-O0 -g etc or when you attached a debugger?<br class=""></div><div class="">- Both, I have run with a debugger attached and detached, all compiled with the following flags "--with-debugging=1 COPTFLAGS=-O0 -g"</div><div class=""><br class=""></div><div class="">1) Try OpenMPI (probably won't help, but worth trying)</div><div class="">- Worth a try for sure</div><div class=""><br class="">2) Find which part of the simulation makes it non-deterministic. Is it the mesh partitioning (parmetis)? Then try to make it deterministic. <br class=""></div><div class="">- Good tip, it is the mesh partitioning and along the lines of a question from Barry, the matrix size is changing. I will make this deterministic and give it a try</div><div class=""><br class=""></div><div class="">3) Dump matrices, vectors, etc and see when it fails, you can quickly reproduce the error by reading in the intermediate data.</div><div class="">- Also a great suggestion, will give it a try</div><div class=""><br class=""></div><div class="">The full stack would be really useful here. I am guessing this happens on MatMult(), but I do not know.</div><div class="">- Agreed, I am currently running it so that the full stack will be produced, but waiting for it to fail, had compiled with all available optimizations on, but downside is of course if there is a failure.</div><div class="">As a general question, roughly what's the performance impact on petsc with -o1/-o2/-o0 as opposed to -o3? Performance impact of --with-debugging = 1?</div><div class="">Obviously problem/machine dependant, wondering on guidance more for this than anything</div><div class=""><br class=""></div><div class=""><span style="color:rgb(255,38,0)" class="">Is the nonzero structure of your matrices changing or is it fixed for the entire simulation?</span></div><div class=""><font class="">The non-zero structure is changing, although the structures are reformed when this happens and this happens thousands of time before the failure has occured.</font></div></div></blockquote><div class=""><br class=""></div><div class="">Okay, this is the most likely spot for a bug. How are you changing the matrix? It should be impossible to put in an invalid column index when using MatSetValues()</div><div class="">because we check all the inputs. However, I do not think we check when you just yank out the arrays.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><span style="color:rgb(255,38,0)" class="">Does this particular run always crash at the same place? Similar place? Doesn't always crash?</span></div><div class=""><font class="">Doesn't always crash, but other similar runs have crashed in different spots, which makes it difficult to resolve. I am going to try out a few of the strategies suggested above and will let you know what comes of that. </font><br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class=""><br class=""></b></div><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 24, 2020 at 11:05 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""> Chris,</div><div class=""><br class=""></div><div class=""> We realize how frustrating this type of problem is to deal with. </div><div class=""><br class=""></div><div class=""> Here is the code:</div><div class=""><br class=""></div> ierr = PetscTableCreate(aij->B->rmap->n,mat->cmap->N+1,&gid1_lid1);CHKERRQ(ierr);<div class=""> for (i=0; i<aij->B->rmap->n; i++) {</div><div class=""> for (j=0; j<B->ilen[i]; j++) {</div><div class=""> PetscInt data,gid1 = aj[B->i[i] + j] + 1;</div><div class=""> ierr = PetscTableFind(gid1_lid1,gid1,&data);CHKERRQ(ierr);</div><div class=""> if (!data) {</div><div class=""> /* one based table */</div><div class=""> ierr = PetscTableAdd(gid1_lid1,gid1,++ec,INSERT_VALUES);CHKERRQ(ierr);</div><div class=""> }</div><div class=""> }</div><div class=""> }</div><div class=""><br class=""></div><div class=""> It is simply looping over the rows of the sparse matrix putting the columns it finds into the hash table. </div><div class=""><br class=""></div><div class=""> aj[B->i[i] + j] are the column entries, the number of columns in the matrix is mat->cmap->N so the column entries should always be </div><div class="">less than the number of columns. The code is crashing when column entry 24443 which is larger than the number of columns 23988.</div><div class=""><br class=""></div><div class="">So either the aj[B->i[i] + j] + 1 are incorrect or the mat->cmap->N is incorrect.</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div style="color:rgb(14,16,26);margin-top:0pt;margin-bottom:0pt" class=""><span style="margin-top:0pt;margin-bottom:0pt" class="">640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c</span></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></blockquote></div></blockquote><br class=""></div><div class=""><div class=""> if (!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) {</div><div class=""> ierr = MatSetUpMultiply_MPIAIJ(mat);CHKERRQ(ierr);</div><div class=""> }</div></div><div class=""><br class=""></div><div class="">Seems to indicate it is setting up a new multiple because it is either the first time into the algorithm or the nonzero structure changed on some rank requiring a new assembly process. </div><div class=""><br class=""></div><div class=""> <font color="#ff2600" class="">Is the nonzero structure of your matrices changing or is it fixed for the entire simulation? </font></div><div class=""><br class=""></div><div class="">Since the code has been running for a very long time already I have to conclude that this is not the first time through and so something has changed in the matrix?</div><div class=""><br class=""></div><div class="">I think we have to put more diagnostics into the library to provide more information before or at the time of the error detection. </div><div class=""><br class=""></div><div class=""> <font color="#ff2600" class="">Does this particular run always crash at the same place? Similar place? Doesn't always crash?</font></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Sep 24, 2020, at 8:46 AM, Chris Hewson <<a href="mailto:chris@resfrac.com" target="_blank" class="">chris@resfrac.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">After about a month of not having this issue pop up, it has come up again</div><div class=""><br class=""></div>We have been struggling with a similar PETSc Error for awhile now, the error message is as follows:<br class=""><br class="">[7]PETSC ERROR: PetscTableFind() line 132 in /home/chewson/petsc-3.13.3/include/petscctable.h key 24443 is greater than largest key allowed 23988<br class=""><br class="">It is a particularly nasty bug as it doesn't reproduce itself when debugging and doesn't happen all the time with the same inputs either. The problem occurs after a long runtime of the code (12-40 hours) and we are using a ksp solve with KSPBCGS. <br class=""><br class="">The PETSc compilation options that are used are:<br class=""><br class="">--download-metis<br class=""> --download-mpich<br class=""> --download-mumps<br class=""> --download-parmetis<br class=""> --download-ptscotch<br class=""> --download-scalapack<br class=""> --download-suitesparse<br class=""> --prefix=/opt/anl/petsc-3.13.3<br class=""> --with-debugging=0<br class=""> --with-mpi=1<br class=""> COPTFLAGS=-O3 -march=haswell -mtune=haswell<br class=""> CXXOPTFLAGS=-O3 -march=haswell -mtune=haswell<br class=""> FOPTFLAGS=-O3 -march=haswell -mtune=haswell<br class=""><br class="">This is being run across 8 processes and is failing consistently on the rank 7 process. We also use openmp outside of PETSC and the linear solve portion of the code. The rank 0 process is always using compute, during this the slave processes use an MPI_Wait call to wait for the collective parts of the code to be called again. All PETSC calls are done across all of the processes.<br class=""><br class="">We are using mpich version 3.3.2, downloaded with the petsc 3.13.3 package.<br class=""><br class="">At every PETSC call we are checking the error return from the function collectively to ensure that no errors have been returned from PETSC.<br class=""><br class="">Some possible causes that I can think of are as follows:<br class="">1. Memory leak causing a corruption either in our program or in petsc or with one of the petsc objects. This seems unlikely as we have checked runs with the option -malloc_dump for PETSc and using valgrind.<br class=""><br class="">2. Optimization flags set for petsc compilation are causing variables that go out of scope to be optimized out. <br class=""><br class="">3. We are giving the wrong number of elements for a process or the value is changing during the simulation. This seems unlikely as there is nothing overly unique about these simulations and it's not reproducing itself.<br class=""><br class="">4. An MPI channel or socket error causing an error in the collective values for PETSc.<br class=""><br class="">Any input on this issue would be greatly appreciated.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class=""><br class=""></b></div><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 13, 2020 at 4:21 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">That is a great idea. I'll figure it out.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 13, 2020 at 4:31 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Junchao,<div class=""><br class=""></div><div class=""> Any way in the PETSc configure to warn that MPICH version is "bad" or "untrustworthy" or even the vague "we have suspicians about this version and recommend avoiding it"? A lot of time could be saved if others don't deal with the same problem.</div><div class=""><br class=""></div><div class=""> Maybe add arrays of suspect_versions for OpenMPI, MPICH, etc and always check against that list and print a boxed warning at configure time? Better you could somehow generalize it and put it in package.py for use by all packages, then any package can included lists of "suspect" versions. (There are definitely HDF5 versions that should be avoided :-)).</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 13, 2020, at 12:14 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Thanks for the update. Let's assume it is a bug in MPI :)<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 13, 2020 at 11:15 AM Chris Hewson <<a href="mailto:chris@resfrac.com" target="_blank" class="">chris@resfrac.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Just as an update to this, I can confirm that using the mpich version (3.3.2) downloaded with the petsc download solved this issue on my end.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class=""><br class=""></b></div><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 23, 2020 at 5:58 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">On Mon, Jul 20, 2020 at 7:05 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Is there a comprehensive MPI test suite (perhaps from MPICH)? Is there any way to run this full test suite under the problematic MPI and see if it detects any problems? <div class=""><br class=""></div><div class=""> Is so, could someone add it to the FAQ in the debugging section?</div></div></blockquote><div class="">MPICH does have a test suite. It is at the subdir test/mpi of downloaded <a href="http://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz" target="_blank" class="">mpich</a>. It annoyed me since it is not user-friendly. It might be helpful in catching bugs at very small scale. But say if I want to test allreduce on 1024 ranks on 100 doubles, I have to hack the test suite.</div><div class="">Anyway, the instructions are here.<br class=""></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class="gmail_quote"><div class="">For the purpose of petsc, under test/mpi one can configure it with</div></div><div class="gmail_quote"><div class="">$./configure CC=mpicc CXX=mpicxx FC=mpifort --enable-strictmpi --enable-threads=funneled --enable-fortran=f77,f90 --enable-fast --disable-spawn --disable-cxx --disable-ft-tests // It is weird I disabled cxx but I had to set CXX!</div></div><div class="gmail_quote"><div class="">$make -k -j8 // -k is to keep going and ignore compilation errors, e.g., when building tests for MPICH extensions not in MPI standard, but your MPI is OpenMPI.</div></div><div class="gmail_quote"><div class="">$ // edit testlist, remove lines mpi_t, rma, f77, impls. Those are sub-dirs containing tests for MPI routines Petsc does not rely on. </div></div><div class="gmail_quote"><div class="">$ make testings or directly './runtests -tests=testlist'</div></div><div class="gmail_quote"><div class=""><br class=""></div></div><div class="gmail_quote"><div class="">On a batch system, </div></div><div class="gmail_quote"><div class="">$export MPITEST_BATCHDIR=`pwd`/btest // specify a batch dir, say btest, </div></div><div class="gmail_quote"><div class="">$./runtests -batch -mpiexec=mpirun -np=1024 -tests=testlist // Use 1024 ranks if a test does no specify the number of processes.</div></div><div class="gmail_quote"><div class="">$ // It copies test binaries to the batch dir and generates a script runtests.batch there. Edit the script to fit your batch system and then submit a job and wait for its finish. </div></div><div class="gmail_quote"><div class="">$ cd btest && ../checktests --ignorebogus</div></div></blockquote><div class="gmail_quote"><div class=""><br class=""></div><div class="">PS: Fande, changing an MPI fixed your problem does not necessarily mean the old MPI has bugs. It is complicated. It could be a petsc bug. You need to provide us a code to reproduce your error. It does not matter if the code is big. </div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class=""> Thanks</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jul 20, 2020, at 12:16 AM, Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank" class="">fdkong.jd@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">Trace could look like this:<div class=""><br class=""></div><div class=""><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: Argument out of range</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: See </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" style="background-color:transparent;margin-top:0pt;margin-bottom:0pt;color:rgb(74,110,224)" target="_blank" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</span></a><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""> for trouble shooting.</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: Petsc Release Version 3.13.3, unknown </span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by wangy2 Sun Jul 19 17:14:28 2020</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #1 PetscTableFind() line 132 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #8 MatProductNumeric() line 759 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #9 MatPtAP() line 9199 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #10 MatGalerkin() line 10236 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #11 PCSetUp_MG() line 745 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #13 PCSetUp() line 898 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #14 KSPSetUp() line 376 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #15 KSPSolve_Private() line 633 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #16 KSPSolve() line 853 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c</span></div><div style="color:rgb(14,16,26);background-color:transparent;margin-top:0pt;margin-bottom:0pt" class=""><span style="background-color:transparent;margin-top:0pt;margin-bottom:0pt" class="">[640]PETSC ERROR: #18 SNESSolve() line 4519 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c</span></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 19, 2020 at 11:13 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank" class="">fdkong.jd@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">I am not entirely sure what is happening, but we encountered similar issues recently. It was not reproducible. It might occur at different stages, and errors could be weird other than "ctable stuff." Our code was Valgrind clean since every PR in moose needs to go through rigorous Valgrind checks before it reaches the devel branch. The errors happened when we used mvapich.</div><div class=""><br class=""></div><div class="">We changed to use HPE-MPT (a vendor stalled MPI), then everything was smooth. May you try a different MPI? It is better to try a system carried one. </div><div class=""><br class=""></div><div class="">We did not get the bottom of this problem yet, but we at least know this is kind of MPI-related. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Fande,</div><div class=""><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson <<a href="mailto:chris@resfrac.com" target="_blank" class="">chris@resfrac.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi,<div class=""><br class=""></div><div class="">I am having a bug that is occurring in PETSC with the return string:</div><div class=""><br class=""></div><div class="">[7]PETSC ERROR: PetscTableFind() line 132 in /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than largest key allowed 5693</div><div class=""><br class=""></div><div class="">This is using petsc-3.13.2, compiled and running using mpich with -O3 and debugging turned off tuned to the haswell architecture and occurring either before or during a KSPBCGS solve/setup or during a MUMPS factorization solve (I haven't been able to replicate this issue with the same set of instructions etc.).</div><div class=""><br class=""></div><div class="">This is a terrible way to ask a question, I know, and not very helpful from your side, but this is what I have from a user's run and can't reproduce on my end (either with the optimization compilation or with debugging turned on). This happens when the code has run for quite some time and is happening somewhat rarely.</div><div class=""><br class=""></div><div class="">More than likely I am using a static variable (code is written in c++) that I'm not updating when the matrix size is changing or something silly like that, but any help or guidance on this would be appreciated. </div><div class=""><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>