<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 7, 2015 at 12:56 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote: <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Run with -info<br></blockquote><div> <br></div><div>This timed out printing from 128K cores. I will try again with this run below.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Note that in maint all your verbose printf used %d which gives nonsense for 64 bit integers so is unreliable. For example N=0<br>
<br></blockquote><div><br></div><div>Is there a way to fix this? %ld?<br><br></div><div>And I ran a variant that does just a MaTransposeMat, and get 128Mb of this:<br><br>MPICH2 ERROR [Rank 63070] [job id 11889451] [Sun Apr 19 08:13:54 2015] [c6-1c1s2n2] [nid02762] - MPID_nem_gni_check_localCQ(): GNI_CQ_EVENT_TY<br>PE_POST had error (SOURCE_SSID:AT_MDD_INV:CPLTN_SRSP)<br>[63070]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1760 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpimatmatmult.<br>c<br>[63070]PETSC ERROR: #2 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 881 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c<br>[63070]PETSC ERROR: #3 MatTransposeMatMult() line 9275 in /global/u2/m/madams/petsc/src/mat/interface/matrix.c<br>[63070]PETSC ERROR: #4 PCGAMGCoarsen_AGG() line 952 in /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c<br>[63070]PETSC ERROR: #5 PCSetUp_GAMG() line 567 in /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c<br>[63070]PETSC ERROR: #6 PCSetUp() line 918 in /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c<br>[63070]PETSC ERROR: #7 KSPSetUp() line 330 in /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c<br>[63070]PETSC ERROR: #8 KSPSolve() line 542 in /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c<br>MPICH2 ERROR [Rank 63633] [job id 11889451] [Sun Apr 19 08:13:54 2015] [c6-1c1s8n2] [nid02786] - MPID_nem_gni_check_localCQ(): GNI_CQ_EVENT_TY<br>PE_POST had error (SOURCE_SSID:AT_MDD_INV:CPLTN_SRSP)<br>[63633]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1760 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpimatmatmult.<br>c<br>[63633]PETSC ERROR: #2 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 881 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c<br>[63633]PETSC ERROR: #3 MatTransposeMatMult() line 9275 in /global/u2/m/madams/petsc/src/mat/interface/matrix.c<br>[63633]PETSC ERROR: #4 PCGAMGCoarsen_AGG() line 952 in /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c<br>[63633]PETSC ERROR: #5 PCSetUp_GAMG() line 567 in /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c<br>[63633]PETSC ERROR: #6 PCSetUp() line 918 in /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c<br>[63633]PETSC ERROR: #7 KSPSetUp() line 330 in /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c<br>[63633]PETSC ERROR: #8 KSPSolve() line 542 in /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c<br><br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I am wondering if it is possible there is overflow in one of the PetscMPIInt variables or arrays.<br>
<span class=""><font color="#888888"><br>
Barry<br>
</font></span><div class=""><div class="h5"><br>
> On Apr 6, 2015, at 11:20 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
><br>
> This is with 'maint'. I will retry with master.<br>
><br>
> On Mon, Apr 6, 2015 at 1:44 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> Mark,<br>
><br>
> Exactly what git commit is this? Should be somewhere in the error message, just send the entire error message.<br>
><br>
> Barry<br>
><br>
> > On Apr 6, 2015, at 9:05 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> ><br>
> > And added squaring the matrix (A^T * A), which uses a matrix matrix product. And get the error. (recall this did run without the square graph, which is where it dies here). I can run this about every two weeks if you want me to run with some other branch or parameters (-info ?).<br>
> > Mark<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > [0]PCSetFromOptions_GAMG threshold set -5.000000e-03<br>
> > [0]PCSetUp_GAMG level 0 N=0, n data rows=1, n data cols=1, nnz/row (ave)=26, np=131072<br>
> > [0]PCGAMGFilterGraph 100% nnz after filtering, with threshold -0.005, 26.9649 nnz ave. (N=0)<br>
> > [0]PCGAMGCoarsen_AGG square graph<br>
> > MPICH2 ERROR [Rank 48173] [job id 11540278] [Sun Apr 5 22:11:13 2015] [c0-1c0s10n3] [nid01579] - MPID_nem_gni_check_localCQ(): GNI_CQ_EVENT_TYPE_POST had error (SOURCE_SSID:AT_MDD_INV:CPLTN_SRSP)<br>
> > [48173]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1757 in /global/u2/m/madams/petsc_maint/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
> > [48173]PETSC ERROR: #2 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 881 in /global/u2/m/madams/petsc_maint/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
> > [48173]PETSC ERROR: #3 MatTransposeMatMult() line 8977 in /global/u2/m/madams/petsc_maint/src/mat/interface/matrix.c<br>
> > [48173]PETSC ERROR: #4 PCGAMGCoarsen_AGG() line 991 in /global/u2/m/madams/petsc_maint/src/ksp/pc/impls/gamg/agg.c<br>
> > [48173]PETSC ERROR: #5 PCSetUp_GAMG() line 596 in /global/u2/m/madams/petsc_maint/src/ksp/pc/impls/gamg/gamg.c<br>
> > [48173]PETSC ERROR: #6 PCSetUp() line 902 in /global/u2/m/madams/petsc_maint/src/ksp/pc/interface/precon.c<br>
> > [48173]PETSC ERROR: #7 KSPSetUp() line 306 in /global/u2/m/madams/petsc_maint/src/ksp/ksp/interface/itfunc.c<br>
> > [48<br>
> ><br>
> > On Sat, Mar 28, 2015 at 1:21 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> > It works for a stripped down GAMG solve (no smoothing and no square graph). I will try adding the square graph back in ...<br>
> ><br>
> > On Wed, Mar 25, 2015 at 7:24 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> ><br>
> ><br>
> > On Sat, Mar 7, 2015 at 5:49 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > > On Mar 7, 2015, at 4:27 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Sat, Mar 7, 2015 at 3:11 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > > Hmm, my first guess is a mixup between PetscMPIInt and PetscInt arrays or MPIU_INT somewhere. (But compilers catch most of these)<br>
> > ><br>
> > > I have a run with ~9K eq/core 128K cores many times but have never gotten the 32K/core (4B eq.) run to work. So it looks like an int overflow issue.<br>
> ><br>
> > Well in theory if we have done everything write with 64 bit indices there should never be an integer overflow (though of course there could be mistake somewhere but generally the compiler will detect if we are trying to stick a 64 bit int into a 32 bit slot.<br>
> ><br>
> > Can you try the same example without GAMG (say with hypre instead); if it goes through ok it might indicate an int issue either in gamg or the code that gamg calls?<br>
> ><br>
> ><br>
> > Good idea, Jacobi works. I will try a stripped down GAMG next.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Barry<br>
> ><br>
> > ><br>
> > ><br>
> > > Another possibility is bug in the MPI for many large messages; any chance you can run the same thing on a very different system? Mira?<br>
> > ><br>
> > > Not soon, but I have Chombo and PETSc built on Mira and it would not be hard to get this code setup and try it.<br>
> > ><br>
> > > This is for SCE15 so I will turn the PETSc test off unless someone has any ideas on something to try. I am using maint perhaps I should use master.<br>
> > ><br>
> > > Mark<br>
> > ><br>
> > ><br>
> > > Barry<br>
> > ><br>
> > > > On Mar 7, 2015, at 1:21 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> > > ><br>
> > > > I seem to be getting this error on Edison with 128K cores and ~4 Billion equations. I've seen this error several time. I've attached a recent output from this. I wonder if it is an integer overflow. This built with 64 bit integers, but I notice that GAMG prints out N and I see N=0 for the finest level.<br>
> > > ><br>
> > > > Mark<br>
> > > > <out.131072.uniform.txt><br>
> > ><br>
> > ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>