[petsc-users] problem (Segmentation voilation) using -pc_type hypre -pc_hypre_type -pilut with multiple nodes in a cluster

[Matthew Knepley]

On Fri, Aug 9, 2013 at 12:08 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Fri, Aug 9, 2013 at 6:58 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Aug 9, 2013 at 11:52 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> Dear all,
>>> I was experimenting with my stokes problem in 3D staggered grid with
>>> high viscosity jump using -pc_fieldsplit of type schur complement. Using
>>> hypre pilut preconditioner for the ksp for A00 block seemed to be giving
>>> nice results for smaller size. Using the following options in my laptop, or
>>> in the cluster I'm using with ONE node multiple cores WORKS fine:
>>> -ksp_type gcr -pc_type fieldsplit -pc_fieldsplit_type schur
>>> -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>>> -pc_fieldsplit_1_fields 3 -ksp_converged_reason -ksp_view
>>> -fieldsplit_0_ksp_type gcr -fieldsplit_0_ksp_rtol 1.0e-5
>>> -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type pilut
>>>
>>> But when I try to submit jobs with multiple nodes, the process never
>>> seem to end! When using gamg instead of hypre, the same program works with
>>> multiple nodes in the same cluster.
>>> But gamg gave much slower convergence than the hypre, so I wanted to use
>>> the hypre.
>>>
>>
>> Did you give a near nullspace to GAMG (probably the 3 translational and 3
>> rotational modes for this problem)? Without these,
>> convergence can be quite slow.
>>
> I have enforced the zero velocity on the boundary, i.e. on all the faces
> of the cube, by changing the corresponding rows of the system matrix. With
> this I think the nullspace would just correspond to the constant pressure.
> For this I set the nullspace using MatNullSpace, for both the outer level
> ksp and ksp object for Schur complement. Using MatNullSpaceTest seemed to
> return good (true) value for both of these set nullspaces.
> Please correct me if I'm doing sth wrong or sth not preferred!
>

You misunderstand the term "near nullspace". A nullspace is used to filter
out modes for which A u = 0. A near nullspace is
used to build a basis on the coarse grid for AMG. They are "near" nullspace
modes since A u = \lambda u where \lambda << 1,
which is what we expect for low modes on the coarse grid. Without guidance,
GAMG will only use the constant function for this
basis which results in slow convergence for more complex problems.


>
>>
>>> When I kill the job and look at the error file, the error it reports:
>>>
>>
>> It looks like PILUT is just slow.
>>
>
> But  using PILUT gives me results for smaller sizes with a single node.
> And the failure with multiple nodes is still for the same problem size!! So
> if it were slower it wouldn't probably have given results in my laptop too
> right ?
>

I have no idea how slow their parallel implementation is.

   Matt


>
>>   Matt
>>
>>
>>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [8]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [8]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[8]PETSCERROR: or try
>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>> corruption errors
>>> [8]PETSC ERROR: likely location of problem given in stack below
>>> [8]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [8]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [8]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [8]PETSC ERROR:       is given.
>>> [8]PETSC ERROR: [8] HYPRE_SetupXXX line 130
>>> /tmp/petsc-3.4.1/src/ksp/pc/impls/hypre/hypre.c
>>> [8]PETSC ERROR: [8] PCSetUp_HYPRE line 94
>>> /tmp/petsc-3.4.1/src/ksp/pc/impls/hypre/hypre.c
>>> [8]PETSC ERROR: [8] PCSetUp line 868
>>> /tmp/petsc-3.4.1/src/ksp/pc/interface/precon.c
>>> [8]PETSC ERROR: [8] KSPSetUp line 192
>>> /tmp/petsc-3.4.1/src/ksp/ksp/interface/itfunc.c
>>> [8]PETSC ERROR: [8] KSPSolve line 356
>>> /tmp/petsc-3.4.1/src/ksp/ksp/interface/itfunc.c
>>> [8]PETSC ERROR: [8] MatMult_SchurComplement line 75
>>> /tmp/petsc-3.4.1/src/ksp/ksp/utils/schurm.c
>>> [8]PETSC ERROR: [8] MatNullSpaceTest line 408
>>> /tmp/petsc-3.4.1/src/mat/interface/matnull.c
>>> [8]PETSC ERROR: [8] solveModel line 133
>>> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>>> [8]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [8]PETSC ERROR: Signal received!
>>> [8]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [8]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
>>> [8]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [8]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [8]PETSC ERROR: See docs/index.html for manual pages.
>>> [8]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [8]PETSC ERROR:
>>> /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
>>> arch-linux2-cxx-debug named nef002 by bkhanal Fri Aug  9 18:00:22 2013
>>> [8]PETSC ERROR: Libraries linked from /home/bkhanal/petsc/lib
>>> [8]PETSC ERROR: Configure run at Mon Jul  1 13:44:30 2013
>>> [8]PETSC ERROR: Configure options
>>> --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries
>>> --prefix=/home/bkhanal/petsc -download-f-blas-lapack=1 --download-hypre
>>> --with-clanguage=cxx
>>> [8]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [8]PETSC ERROR: User provided function() line 0 in unknown directory
>>> unknown file
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130809/b2254fbc/attachment.html>