[petsc-dev] asm / gasm
Mark Adams
mfadams at lbl.gov
Fri Jun 24 12:19:34 CDT 2016
Well, SNES ex56 is completely broken. It is not calling the solvers at all.
I will take a look at it.
On Fri, Jun 24, 2016 at 6:34 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Ok in master using src/snes/examples/tutorials/ex56.c I get
>
> $ petscmpiexec -n 2 ./ex56 -ne 2 -alpha 1.e-3 -two_solves true -dm_refine
> 1 -petscspace_poly_tensor -petscspace_order 1 -ksp_type cg
> -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg
> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100
> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
> -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true
> -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev
> -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor
> -mat_block_size 3 -petscpartitioner_type parmetis -pc_gamg_use_agg_gasm
> -mg_levels_pc_type gasm -ksp_view
> [0]main 300/100 eqs in vector
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Nonconforming object sizes
> [0]PETSC ERROR: Vector wrong size 120 for scatter 1 (scatter forward and
> vector from != ctx from size)
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT
> Date: 2016-06-24 09:01:50 -0500
> [0]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by
> barrysmith Fri Jun 24 11:30:00 2016
> [0]PETSC ERROR: Configure options
> --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic
> --download-metis --download-parmetis
> [0]PETSC ERROR: #1 VecScatterBegin() line 1773 in
> /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: #2 PCApply_GASM() line 686 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c
> [0]PETSC ERROR: #3 PCApply() line 482 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSP_PCApply() line 244 in
> /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #5 KSPSolve_CG() line 137 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
> [0]PETSC ERROR: #6 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #8 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Nonconforming object sizes
> [1]PETSC ERROR: Vector wrong size 180 for scatter 300 (scatter forward and
> vector from != ctx from size)
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT
> Date: 2016-06-24 09:01:50 -0500
> [1]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by
> barrysmith Fri Jun 24 11:30:00 2016
> [1]PETSC ERROR: #10 PCApply_MG() line 343 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #11 PCApply() line 482 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #12 KSP_PCApply() line 244 in
> /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #13 KSPSolve_CG() line 137 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
> Configure options --with-mpi-dir=/Users/barrysmith/libraries
> PETSC_ARCH=arch-basic --download-metis --download-parmetis
> [1]PETSC ERROR: #1 VecScatterBegin() line 1773 in
> /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c
> [1]PETSC ERROR: #2 PCApply_GASM() line 686 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c
> [1]PETSC ERROR: #3 PCApply() line 482 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #14 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #15 main() line 408 in
> /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -alpha 1.e-3
> [0]PETSC ERROR: -dm_refine 1
> [0]PETSC ERROR: -ksp_converged_reason
> [0]PETSC ERROR: -ksp_monitor_short
> [0]PETSC ERROR: -ksp_rtol 1.e-8
> [0]PETSC ERROR: -ksp_type cg
> [0]PETSC ERROR: [1]PETSC ERROR: #4 KSP_PCApply() line 244 in
> /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
> [1]PETSC ERROR: #5 KSPSolve_CG() line 137 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: #6 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [1]PETSC ERROR: #8 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> -ksp_view
> [0]PETSC ERROR: -malloc_test
> [0]PETSC ERROR: -mat_block_size 3
> [0]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10
> [0]PETSC ERROR: -mg_levels_esteig_ksp_type cg
> [0]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
> [0]PETSC ERROR: -mg_levels_ksp_max_it 2
> [1]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
> [1]PETSC ERROR: #10 PCApply_MG() line 343 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: -mg_levels_ksp_type chebyshev
> [0]PETSC ERROR: -mg_levels_pc_type gasm
> [0]PETSC ERROR: -ne 2
> [0]PETSC ERROR: -pc_gamg_agg_nsmooths 1
> [0]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
> [0]PETSC ERROR: [1]PETSC ERROR: #11 PCApply() line 482 in
> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #12 KSP_PCApply() line 244 in
> /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h
> [1]PETSC ERROR: #13 KSPSolve_CG() line 137 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: #14 KSPSolve() line 656 in
> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
> -pc_gamg_reuse_interpolation true
> [0]PETSC ERROR: -pc_gamg_square_graph 1
> [0]PETSC ERROR: -pc_gamg_threshold 0.0
> [1]PETSC ERROR: #15 main() line 408 in
> /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: -pc_gamg_type agg
> [0]PETSC ERROR: -pc_gamg_use_agg_gasm
> [0]PETSC ERROR: -pc_type gamg
> [0]PETSC ERROR: -petscpartitioner_type parmetis
> [0]PETSC ERROR: -petscspace_order 1
> [0]PETSC ERROR: -petscspace_poly_tensor
> -alpha 1.e-3
> [1]PETSC ERROR: -dm_refine 1
> [1]PETSC ERROR: -ksp_converged_reason
> [1]PETSC ERROR: -ksp_monitor_short
> [0]PETSC ERROR: -two_solves true
> [0]PETSC ERROR: -use_mat_nearnullspace true
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: -ksp_rtol 1.e-8
> [1]PETSC ERROR: -ksp_type cg
> [1]PETSC ERROR: -ksp_view
> [1]PETSC ERROR: -malloc_test
> [1]PETSC ERROR: -mat_block_size 3
> [1]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10
> [1]PETSC ERROR: -mg_levels_esteig_ksp_type cg
> [1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
> [1]PETSC ERROR: -mg_levels_ksp_max_it 2
> [1]PETSC ERROR: -mg_levels_ksp_type chebyshev
> [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 60) - process
> 0
> -mg_levels_pc_type gasm
> [1]PETSC ERROR: -ne 2
> [1]PETSC ERROR: -pc_gamg_agg_nsmooths 1
> [1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
> [1]PETSC ERROR: -pc_gamg_reuse_interpolation true
> [1]PETSC ERROR: -pc_gamg_square_graph 1
> [1]PETSC ERROR: -pc_gamg_threshold 0.0
> [1]PETSC ERROR: -pc_gamg_type agg
> [1]PETSC ERROR: -pc_gamg_use_agg_gasm
> [1]PETSC ERROR: -pc_type gamg
> [1]PETSC ERROR: -petscpartitioner_type parmetis
> [1]PETSC ERROR: -petscspace_order 1
> [1]PETSC ERROR: -petscspace_poly_tensor
> [1]PETSC ERROR: -two_solves true
> [1]PETSC ERROR: -use_mat_nearnullspace true
> [1]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1
>
>
> I'll have to start poking around to see what is going on. Could be do
> to changes in GAMG or in GASM. Bisect here I come.
>
>
> Barry
>
> > On Jun 24, 2016, at 9:52 AM, Mark Adams <mfadams at lbl.gov> wrote:
> >
> > OK, but how should I proceed?
> >
> > On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > >
> > > Barry,
> > >
> > > ASM worked before GASM existed. I've never heard of GASM until
> yesterday, other than seeing the directory in the source.
> >
> > Pretty impressive you never heard of GASM and yet YOU put the GASM
> code into GAMG
> >
> > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2012-05-01 12:48:43 -0400 604)
> > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2012-05-01 12:48:43 -0400 605) /* set defaults */
> > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith
> 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother,
> KSPCHEBYSHEV);CHKERRQ(ierr);
> > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2012-03-27 19:09:54 -0400 607)
> > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that
> uses the 'aggregates' */
> > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith
> 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) {
> > 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay
> 2012-03-29 12:58:42 -0500 610) PetscInt sz;
> > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp
> 2013-01-30 22:53:06 -0600 611) IS *is;
> > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2012-05-01 12:48:43 -0400 612)
> > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp
> 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level];
> > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp
> 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level];
> > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith
> 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc,
> PCGASM);CHKERRQ(ierr);
> > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc,
> 0);CHKERRQ(ierr);
> > 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2015-08-11 09:52:40 -0400 617) if (!sz) {
> > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp
> 2013-01-30 22:53:06 -0600 618) IS is;
> > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams
> 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk;
> > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith
> 2013-01-02 02:05:45 -0600 620) ierr =
> MatGetOwnershipRange(Aarr[level], &my0
> >
> > I think you put in the GASM but then forgot and thought you put in
> ASM.
> >
> > > It has been so long, I could probably hunt for a working version, but
> who cares, we are going to roll back 5 years?
> > >
> > >
> > > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56':
> > >
> > > I don't care about this! This is where you have tried to change
> from GASM to ASM which I told you is non-trivial.
> > > Give me the example and command line where the GASM version in master
> (or maint) doesn't work where the error message includes ** Max-trans not
> allowed because matrix is distributed
> > >
> > > Why do you want to debug GASM? I thought we wanted to use ASM. It is
> not clear why converting from GASM to ASM is non-trivial, I could have done
> this from the command line. This use aggregate-asm stuff just does what a
> clever user might do.
> > >
> > >
> > > We are not communicating very well, you jumped from stating GASM
> crashed to monkeying with ASM and now refuse to tell me how to reproduce
> the GASM crash.
> > >
> > > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with:
> > >
> > > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm
> > >
> > > and remove the existing -mg_levels_pc_type sor, you should see an
> error that Garth and I saw (eg, appended).
> > >
> > >
> > > We have to start by fixing the current code to work with GASM (if it
> ever worked) and then move on to using ASM (which is just an optimization
> of the GASM usage.)
> > >
> > >
> > > Are you sure? I am deep into debugging asm with DDT. I can switch to
> GASM by just calling the GASM set block method and use gasm in the PC, and
> continue with DDT debugging.
> > >
> > > Let me know.
> > >
> > >
> > > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make
> runex56
> > > 1,4c1,85
> > > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > > < Linear solve converged due to CONVERGED_RTOL iterations 8
> > > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01
> > > ---
> > > > [1]PETSC ERROR:
> ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> > > > [1]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > > > [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > > [1]PETSC ERROR: [2]PETSC ERROR:
> ------------------------------------------------------------------------
> > > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> > > > [2]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > > > [2]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
> Mac OS X to find memory corruption errors
> > > > [6]PETSC ERROR:
> ------------------------------------------------------------------------
> > > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> > > > [6]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > > > [6]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
> Mac OS X to find memory corruption errors
> > > > [4]PETSC ERROR:
> ------------------------------------------------------------------------
> > > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> > > > [4]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > > > [4]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
> Mac OS X to find memory corruption errors
> > > > [6]PETSC ERROR: likely location of problem given in stack below
> > > > [6]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> > > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> > > > [6]PETSC ERROR: INSTEAD the line number of the start of the
> function
> > > > [6]PETSC ERROR: is given.
> > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228
> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131
> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163
> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418
> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c
> > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805
> /Users/markadams/Codes/petsc/src/mat/interface/matrix.c
> > > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357
> /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c
> > > > [6]PETSC ERROR: [6] PCSetUp line 930
> /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c
> > > > [6]PETSC ERROR: [6] KSPSetUp line 305
> /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c
> > > > [6]PETSC ERROR: [6] PCSetUp_MG line 561
> /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c
> > > > [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > > > [6]PETSC ERROR: Signal received
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20160624/6ef98f45/attachment.html>
More information about the petsc-dev
mailing list