[petsc-users] problem (Segmentation voilation) using -pc_type hypre -pc_hypre_type -pilut with multiple nodes in a cluster

Fri Aug 9 12:22:44 CDT 2013

On Aug 9, 2013, at 1:08 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> 
> 
> 
> On Fri, Aug 9, 2013 at 6:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Aug 9, 2013 at 11:52 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:
> Dear all,
> I was experimenting with my stokes problem in 3D staggered grid with high viscosity jump using -pc_fieldsplit of type schur complement. Using hypre pilut preconditioner for the ksp for A00 block seemed to be giving nice results for smaller size. Using the following options in my laptop, or in the cluster I'm using with ONE node multiple cores WORKS fine:
> -ksp_type gcr -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -ksp_converged_reason -ksp_view -fieldsplit_0_ksp_type gcr -fieldsplit_0_ksp_rtol 1.0e-5 -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type pilut
> 
> But when I try to submit jobs with multiple nodes, the process never seem to end! When using gamg instead of hypre, the same program works with multiple nodes in the same cluster. 
> But gamg gave much slower convergence than the hypre, so I wanted to use the hypre.
> 
> Did you give a near nullspace to GAMG (probably the 3 translational and 3 rotational modes for this problem)? Without these,
> convergence can be quite slow.
> I have enforced the zero velocity on the boundary, i.e. on all the faces of the cube, by changing the corresponding rows of the system matrix. With this I think the nullspace would just correspond to the constant pressure.

We are talking about the 00 block solver (velocity) and there are 6 (near) null space vectors.  GAMG does want to be told you have 3 dof/node.  

Should DM do this!!!  

If you tell GAMG there is only one (the (1,1,1) vector) then that will mess it up.  If you set the blocks size on the matrix (to 3), which should be done by PETSc, and do not set a null space, GAMG will construct the default null space: three constant functions.  This should work OK.  As Matt pointed out you want to give it all six (the 3 constant vectors and 3 rotational null space vectors) to get an optimal solver.

GAMG is an AMG method like hypre boomeramg. Boomeramg is very robust and I would recommend using it to start.  GAMG might be better for vector values problems but if hypre is probably pretty good (for low order discretizations especially).

Mark

> For this I set the nullspace using MatNullSpace, for both the outer level ksp and ksp object for Schur complement. Using MatNullSpaceTest seemed to return good (true) value for both of these set nullspaces.
> Please correct me if I'm doing sth wrong or sth not preferred!
>  
>  
> When I kill the job and look at the error file, the error it reports:
> 
> It looks like PILUT is just slow.
> 
> But  using PILUT gives me results for smaller sizes with a single node. And the failure with multiple nodes is still for the same problem size!! So if it were slower it wouldn't probably have given results in my laptop too right ?
> 
> 
>   Matt
>  
> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [8]PETSC ERROR: likely location of problem given in stack below
> [8]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [8]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [8]PETSC ERROR:       INSTEAD the line number of the start of the function
> [8]PETSC ERROR:       is given.
> [8]PETSC ERROR: [8] HYPRE_SetupXXX line 130 /tmp/petsc-3.4.1/src/ksp/pc/impls/hypre/hypre.c
> [8]PETSC ERROR: [8] PCSetUp_HYPRE line 94 /tmp/petsc-3.4.1/src/ksp/pc/impls/hypre/hypre.c
> [8]PETSC ERROR: [8] PCSetUp line 868 /tmp/petsc-3.4.1/src/ksp/pc/interface/precon.c
> [8]PETSC ERROR: [8] KSPSetUp line 192 /tmp/petsc-3.4.1/src/ksp/ksp/interface/itfunc.c
> [8]PETSC ERROR: [8] KSPSolve line 356 /tmp/petsc-3.4.1/src/ksp/ksp/interface/itfunc.c
> [8]PETSC ERROR: [8] MatMult_SchurComplement line 75 /tmp/petsc-3.4.1/src/ksp/ksp/utils/schurm.c
> [8]PETSC ERROR: [8] MatNullSpaceTest line 408 /tmp/petsc-3.4.1/src/mat/interface/matnull.c
> [8]PETSC ERROR: [8] solveModel line 133 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> [8]PETSC ERROR: --------------------- Error Message ------------------------------------
> [8]PETSC ERROR: Signal received!
> [8]PETSC ERROR: ------------------------------------------------------------------------
> [8]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
> [8]PETSC ERROR: See docs/changes/index.html for recent updates.
> [8]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [8]PETSC ERROR: See docs/index.html for manual pages.
> [8]PETSC ERROR: ------------------------------------------------------------------------
> [8]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef002 by bkhanal Fri Aug  9 18:00:22 2013
> [8]PETSC ERROR: Libraries linked from /home/bkhanal/petsc/lib
> [8]PETSC ERROR: Configure run at Mon Jul  1 13:44:30 2013
> [8]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/home/bkhanal/petsc -download-f-blas-lapack=1 --download-hypre --with-clanguage=cxx
> [8]PETSC ERROR: ------------------------------------------------------------------------
> [8]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130809/032056fa/attachment-0001.html>