[petsc-users] signal received error; MatNullSpaceTest; Stokes flow solver with pc fieldsplit and schur complement
Mark F. Adams
mfadams at lbl.gov
Wed Oct 16 10:50:48 CDT 2013
You might also test with Jacobi as a sanity check.
On Oct 16, 2013, at 11:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
> especially the GAMG version which you also try in the debugger with the argument -start_in_debugger
>
>
>
> On Oct 16, 2013, at 9:32 AM, Dave May <dave.mayhem23 at gmail.com> wrote:
>
>> Sounds like a memory error.
>> I'd run your code through valgrind to double check. The error could be completely unconnected to the nullspaces.
>>
>> Cheers,
>> Dave
>>
>>
>> On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:
>> Dear all,
>> I'm trying to solve a stokes flow with constant viscosity but with non-zero divergence prescribed in the rhs.
>>
>> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p respectively.
>> I have another DMDA (mDaP) of same size but of 1 dof corresponding to only p.
>> I have assigned the null space for constant pressure inside the code. I have assigned two nullspace basis: One corresponding to vector created from mDa that is assigned to outer ksp. Second corresponding to vector created from mDaP that is assigned to a ksp obtained from the fieldsplit corresponding to the schur complement.
>>
>> Now when running the code, the solver converges for up to certain size, e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is given at the end of the emal.
>> But when I double the size of the grid in each dimension, it gives me a run-time error.
>>
>> The options I've used are of the kind:
>> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
>>
>> Here are:
>> 1. Error message when using hypre for fieldsplit_0
>> 2. Error message when using gamg for fieldsplit_0
>> 3. -ksp_view of the working case using hypre for filedsplit_0
>>
>> I get following error when I use hypre :
>> 1. ******************************************************************************************************
>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [5]PETSC ERROR: Signal received!
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
>> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [5]PETSC ERROR: See docs/index.html for manual pages.
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
>> [5]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
>> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
>> [5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1 --download-metis --download-parmetis --download-superlu_dist --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> [6]PETSC ERROR: ------------------------------------------------------------------------
>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [6]PETSC ERROR: likely location of problem given in stack below
>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------
>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [6]PETSC ERROR: INSTEAD the line number of the start of the function
>> [6]PETSC ERROR: is given.
>> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> [6]PETSC ERROR: [6] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> [6]PETSC ERROR: [6] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [6]PETSC ERROR: [6] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> [6]PETSC ERROR: [6] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> [6]PETSC ERROR: [6] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>>
>>
>> 2. ****************************************************************************************************
>> Using gamg instead has errors like following:
>>
>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------
>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [5]PETSC ERROR: INSTEAD the line number of the start of the function
>> [5]PETSC ERROR: is given.
>> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202 /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
>> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> [5]PETSC ERROR: [5] MatPtAP line 8223 /tmp/petsc-3.4.3/src/mat/interface/matrix.c
>> [5]PETSC ERROR: [5] createLevel line 144 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> [5]PETSC ERROR: [5] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> [5]PETSC ERROR: [5] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [5]PETSC ERROR: [5] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> [5]PETSC ERROR: [5] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> [5]PETSC ERROR: [5] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>>
>>
>> 3. ********************************************************************************************************
>>
>> BUT, It does give me results when I use a domain of size: 91X109 X 91 (half sized in each dimension) The result along with ksp view in this case is as follows:
>>
>> Linear solve converged due to CONVERGED_RTOL iterations 2
>> KSP Object: 64 MPI processes
>> type: gmres
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>> GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=10000, initial guess is zero
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> has attached null space
>> using PRECONDITIONED norm type for convergence test
>> PC Object: 64 MPI processes
>> type: fieldsplit
>> FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL
>> Preconditioner for the Schur complement formed from user provided matrix
>> Split info:
>> Split number 0 Fields 0, 1, 2
>> Split number 1 Fields 3
>> KSP solver for A00 block
>> KSP Object: (fieldsplit_0_) 64 MPI processes
>> type: gmres
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>> GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=10000, initial guess is zero
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> using PRECONDITIONED norm type for convergence test
>> PC Object: (fieldsplit_0_) 64 MPI processes
>> type: hypre
>> HYPRE BoomerAMG preconditioning
>> HYPRE BoomerAMG: Cycle type V
>> HYPRE BoomerAMG: Maximum number of levels 25
>> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>> HYPRE BoomerAMG: Threshold for strong coupling 0.25
>> HYPRE BoomerAMG: Interpolation truncation factor 0
>> HYPRE BoomerAMG: Interpolation: max elements per row 0
>> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>> HYPRE BoomerAMG: Maximum row sums 0.9
>> HYPRE BoomerAMG: Sweeps down 1
>> HYPRE BoomerAMG: Sweeps up 1
>> HYPRE BoomerAMG: Sweeps on coarse 1
>> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi
>> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi
>> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination
>> HYPRE BoomerAMG: Relax weight (all) 1
>> HYPRE BoomerAMG: Outer relax weight (all) 1
>> HYPRE BoomerAMG: Using CF-relaxation
>> HYPRE BoomerAMG: Measure type local
>> HYPRE BoomerAMG: Coarsen type Falgout
>> HYPRE BoomerAMG: Interpolation type classical
>> linear system matrix = precond matrix:
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=2793120, cols=2793120
>> total: nonzeros=221624352, allocated nonzeros=221624352
>> total number of mallocs used during MatSetValues calls =0
>> using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>> KSP solver for S = A11 - A10 inv(A00) A01
>> KSP Object: (fieldsplit_1_) 64 MPI processes
>> type: gmres
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>> GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=10000, initial guess is zero
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> has attached null space
>> using PRECONDITIONED norm type for convergence test
>> PC Object: (fieldsplit_1_) 64 MPI processes
>> type: bjacobi
>> block Jacobi: number of blocks = 64
>> Local solve is same for all blocks, in the following KSP and PC objects:
>> KSP Object: (fieldsplit_1_sub_) 1 MPI processes
>> type: preonly
>> maximum iterations=10000, initial guess is zero
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> using NONE norm type for convergence test
>> PC Object: (fieldsplit_1_sub_) 1 MPI processes
>> type: ilu
>> ILU: out-of-place factorization
>> 0 levels of fill
>> tolerance for zero pivot 2.22045e-14
>> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>> matrix ordering: natural
>> factor fill ratio given 1, needed 1
>> Factored matrix follows:
>> Matrix Object: 1 MPI processes
>> type: seqaij
>> rows=14812, cols=14812
>> package used to perform factorization: petsc
>> total: nonzeros=368098, allocated nonzeros=368098
>> total number of mallocs used during MatSetValues calls =0
>> not using I-node routines
>> linear system matrix = precond matrix:
>> Matrix Object: 1 MPI processes
>> type: seqaij
>> rows=14812, cols=14812
>> total: nonzeros=368098, allocated nonzeros=368098
>> total number of mallocs used during MatSetValues calls =0
>> not using I-node routines
>>
>> linear system matrix followed by preconditioner matrix:
>> Matrix Object: 64 MPI processes
>> type: schurcomplement
>> rows=931040, cols=931040
>> Schur complement A11 - A10 inv(A00) A01
>> A11
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=931040, cols=931040
>> total: nonzeros=24624928, allocated nonzeros=24624928
>> total number of mallocs used during MatSetValues calls =0
>> not using I-node (on process 0) routines
>> A10
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=931040, cols=2793120
>> total: nonzeros=73874784, allocated nonzeros=73874784
>> total number of mallocs used during MatSetValues calls =0
>> not using I-node (on process 0) routines
>> KSP of A00
>> KSP Object: (fieldsplit_0_) 64 MPI processes
>> type: gmres
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>> GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=10000, initial guess is zero
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>> left preconditioning
>> using PRECONDITIONED norm type for convergence test
>> PC Object: (fieldsplit_0_) 64 MPI processes
>> type: hypre
>> HYPRE BoomerAMG preconditioning
>> HYPRE BoomerAMG: Cycle type V
>> HYPRE BoomerAMG: Maximum number of levels 25
>> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>> HYPRE BoomerAMG: Threshold for strong coupling 0.25
>> HYPRE BoomerAMG: Interpolation truncation factor 0
>> HYPRE BoomerAMG: Interpolation: max elements per row 0
>> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>> HYPRE BoomerAMG: Maximum row sums 0.9
>> HYPRE BoomerAMG: Sweeps down 1
>> HYPRE BoomerAMG: Sweeps up 1
>> HYPRE BoomerAMG: Sweeps on coarse 1
>> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi
>> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi
>> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination
>> HYPRE BoomerAMG: Relax weight (all) 1
>> HYPRE BoomerAMG: Outer relax weight (all) 1
>> HYPRE BoomerAMG: Using CF-relaxation
>> HYPRE BoomerAMG: Measure type local
>> HYPRE BoomerAMG: Coarsen type Falgout
>> HYPRE BoomerAMG: Interpolation type classical
>> linear system matrix = precond matrix:
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=2793120, cols=2793120
>> total: nonzeros=221624352, allocated nonzeros=221624352
>> total number of mallocs used during MatSetValues calls =0
>> using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>> A01
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=2793120, cols=931040
>> total: nonzeros=73874784, allocated nonzeros=73874784
>> total number of mallocs used during MatSetValues calls =0
>> using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=931040, cols=931040
>> total: nonzeros=24624928, allocated nonzeros=24624928
>> total number of mallocs used during MatSetValues calls =0
>> not using I-node (on process 0) routines
>> linear system matrix = precond matrix:
>> Matrix Object: 64 MPI processes
>> type: mpiaij
>> rows=3724160, cols=3724160, bs=4
>> total: nonzeros=393998848, allocated nonzeros=393998848
>> total number of mallocs used during MatSetValues calls =0
>>
>> ******************************************************************************************************
>> What could be going wrong here ? Is it something related to null-space setting ? But I do not know why it does not arise for smaller domain sizes!
>>
>
More information about the petsc-users
mailing list