[petsc-dev] asm / gasm
Barry Smith
bsmith at mcs.anl.gov
Fri Jun 24 11:34:44 CDT 2016
Ok in master using src/snes/examples/tutorials/ex56.c I get
$ petscmpiexec -n 2 ./ex56 -ne 2 -alpha 1.e-3 -two_solves true -dm_refine 1 -petscspace_poly_tensor -petscspace_order 1 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor -mat_block_size 3 -petscpartitioner_type parmetis -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm -ksp_view
[0]main 300/100 eqs in vector
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Vector wrong size 120 for scatter 1 (scatter forward and vector from != ctx from size)
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT Date: 2016-06-24 09:01:50 -0500
[0]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Fri Jun 24 11:30:00 2016
[0]PETSC ERROR: Configure options --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic --download-metis --download-parmetis
[0]PETSC ERROR: #1 VecScatterBegin() line 1773 in /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c
[0]PETSC ERROR: #2 PCApply_GASM() line 686 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c
[0]PETSC ERROR: #3 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #5 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
[0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: #8 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Nonconforming object sizes
[1]PETSC ERROR: Vector wrong size 180 for scatter 300 (scatter forward and vector from != ctx from size)
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT Date: 2016-06-24 09:01:50 -0500
[1]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Fri Jun 24 11:30:00 2016
[1]PETSC ERROR: #10 PCApply_MG() line 343 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #11 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #12 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #13 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
Configure options --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic --download-metis --download-parmetis
[1]PETSC ERROR: #1 VecScatterBegin() line 1773 in /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c
[1]PETSC ERROR: #2 PCApply_GASM() line 686 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c
[1]PETSC ERROR: #3 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #14 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #15 main() line 408 in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -alpha 1.e-3
[0]PETSC ERROR: -dm_refine 1
[0]PETSC ERROR: -ksp_converged_reason
[0]PETSC ERROR: -ksp_monitor_short
[0]PETSC ERROR: -ksp_rtol 1.e-8
[0]PETSC ERROR: -ksp_type cg
[0]PETSC ERROR: [1]PETSC ERROR: #4 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
[1]PETSC ERROR: #5 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
[1]PETSC ERROR: #6 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c
[1]PETSC ERROR: #8 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
-ksp_view
[0]PETSC ERROR: -malloc_test
[0]PETSC ERROR: -mat_block_size 3
[0]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10
[0]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[0]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[0]PETSC ERROR: -mg_levels_ksp_max_it 2
[1]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
[1]PETSC ERROR: #10 PCApply_MG() line 343 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: -mg_levels_ksp_type chebyshev
[0]PETSC ERROR: -mg_levels_pc_type gasm
[0]PETSC ERROR: -ne 2
[0]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[0]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[0]PETSC ERROR: [1]PETSC ERROR: #11 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #12 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
[1]PETSC ERROR: #13 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
[1]PETSC ERROR: #14 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
-pc_gamg_reuse_interpolation true
[0]PETSC ERROR: -pc_gamg_square_graph 1
[0]PETSC ERROR: -pc_gamg_threshold 0.0
[1]PETSC ERROR: #15 main() line 408 in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: [0]PETSC ERROR: -pc_gamg_type agg
[0]PETSC ERROR: -pc_gamg_use_agg_gasm
[0]PETSC ERROR: -pc_type gamg
[0]PETSC ERROR: -petscpartitioner_type parmetis
[0]PETSC ERROR: -petscspace_order 1
[0]PETSC ERROR: -petscspace_poly_tensor
-alpha 1.e-3
[1]PETSC ERROR: -dm_refine 1
[1]PETSC ERROR: -ksp_converged_reason
[1]PETSC ERROR: -ksp_monitor_short
[0]PETSC ERROR: -two_solves true
[0]PETSC ERROR: -use_mat_nearnullspace true
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: -ksp_rtol 1.e-8
[1]PETSC ERROR: -ksp_type cg
[1]PETSC ERROR: -ksp_view
[1]PETSC ERROR: -malloc_test
[1]PETSC ERROR: -mat_block_size 3
[1]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10
[1]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[1]PETSC ERROR: -mg_levels_ksp_max_it 2
[1]PETSC ERROR: -mg_levels_ksp_type chebyshev
[1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 60) - process 0
-mg_levels_pc_type gasm
[1]PETSC ERROR: -ne 2
[1]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[1]PETSC ERROR: -pc_gamg_reuse_interpolation true
[1]PETSC ERROR: -pc_gamg_square_graph 1
[1]PETSC ERROR: -pc_gamg_threshold 0.0
[1]PETSC ERROR: -pc_gamg_type agg
[1]PETSC ERROR: -pc_gamg_use_agg_gasm
[1]PETSC ERROR: -pc_type gamg
[1]PETSC ERROR: -petscpartitioner_type parmetis
[1]PETSC ERROR: -petscspace_order 1
[1]PETSC ERROR: -petscspace_poly_tensor
[1]PETSC ERROR: -two_solves true
[1]PETSC ERROR: -use_mat_nearnullspace true
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1
I'll have to start poking around to see what is going on. Could be do to changes in GAMG or in GASM. Bisect here I come.
Barry
> On Jun 24, 2016, at 9:52 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
> OK, but how should I proceed?
>
> On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfadams at lbl.gov> wrote:
> >
> > Barry,
> >
> > ASM worked before GASM existed. I've never heard of GASM until yesterday, other than seeing the directory in the source.
>
> Pretty impressive you never heard of GASM and yet YOU put the GASM code into GAMG
>
> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 604)
> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 605) /* set defaults */
> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother, KSPCHEBYSHEV);CHKERRQ(ierr);
> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-03-27 19:09:54 -0400 607)
> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that uses the 'aggregates' */
> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) {
> 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay 2012-03-29 12:58:42 -0500 610) PetscInt sz;
> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 611) IS *is;
> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 612)
> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level];
> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level];
> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc, PCGASM);CHKERRQ(ierr);
> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc, 0);CHKERRQ(ierr);
> 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2015-08-11 09:52:40 -0400 617) if (!sz) {
> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 618) IS is;
> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk;
> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 620) ierr = MatGetOwnershipRange(Aarr[level], &my0
>
> I think you put in the GASM but then forgot and thought you put in ASM.
>
> > It has been so long, I could probably hunt for a working version, but who cares, we are going to roll back 5 years?
> >
> >
> > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56':
> >
> > I don't care about this! This is where you have tried to change from GASM to ASM which I told you is non-trivial.
> > Give me the example and command line where the GASM version in master (or maint) doesn't work where the error message includes ** Max-trans not allowed because matrix is distributed
> >
> > Why do you want to debug GASM? I thought we wanted to use ASM. It is not clear why converting from GASM to ASM is non-trivial, I could have done this from the command line. This use aggregate-asm stuff just does what a clever user might do.
> >
> >
> > We are not communicating very well, you jumped from stating GASM crashed to monkeying with ASM and now refuse to tell me how to reproduce the GASM crash.
> >
> > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with:
> >
> > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm
> >
> > and remove the existing -mg_levels_pc_type sor, you should see an error that Garth and I saw (eg, appended).
> >
> >
> > We have to start by fixing the current code to work with GASM (if it ever worked) and then move on to using ASM (which is just an optimization of the GASM usage.)
> >
> >
> > Are you sure? I am deep into debugging asm with DDT. I can switch to GASM by just calling the GASM set block method and use gasm in the PC, and continue with DDT debugging.
> >
> > Let me know.
> >
> >
> > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make runex56
> > 1,4c1,85
> > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01
> > ---
> > > [1]PETSC ERROR: ------------------------------------------------------------------------
> > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > [1]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------
> > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > [6]PETSC ERROR: ------------------------------------------------------------------------
> > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > [4]PETSC ERROR: ------------------------------------------------------------------------
> > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > [6]PETSC ERROR: likely location of problem given in stack below
> > > [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> > > [6]PETSC ERROR: INSTEAD the line number of the start of the function
> > > [6]PETSC ERROR: is given.
> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805 /Users/markadams/Codes/petsc/src/mat/interface/matrix.c
> > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357 /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c
> > > [6]PETSC ERROR: [6] PCSetUp line 930 /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c
> > > [6]PETSC ERROR: [6] KSPSetUp line 305 /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c
> > > [6]PETSC ERROR: [6] PCSetUp_MG line 561 /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c
> > > [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > [6]PETSC ERROR: Signal received
>
>
More information about the petsc-dev
mailing list