[petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace

Gil Forsyth gforsyth at gwu.edu
Wed Sep 30 15:11:14 CDT 2015


Using PETSc master branch solved the problem in serial, but I'm still
seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
runs to completion when I don't use GAMG.  Log is attached for the
following run.

$PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 $PETIBM_DIR/petibm-git/bin/petibm2d
-directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg
-poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason


Thanks again,
Gil Forsyth


On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Ah, got it.  I'll checkout the master branch and see if the behavior
> persists.
>
> Many thanks,
> Gil
>
> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>
>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>>> the behavior seems to have changed between versions.  The only difference
>>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>>> MatSetNullSpace.
>>>
>>
>> Mark made some GAMG changes which were later reversed because they had
>> unintended consequences like this.
>> I think what Barry means is, "you should get the behavior you expect
>> using the master branch from PETSc development"
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>>   Update your PETSc
>>>>
>>>>
>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>> >
>>>> > Hi Barry,
>>>> >
>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>> works with:
>>>> >
>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>> kspview3.5.4
>>>> >
>>>> > and shows that the coarse grid solver is of type:preonly
>>>> >
>>>> > running the newer version that uses MatSetNullSpace in its stead and
>>>> adding in -poisson_mg_coarse_ksp_type preonly
>>>> >
>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>> kspview3.6.1
>>>> >
>>>> > still shows
>>>> >
>>>> > KSP Object:(poisson_) 1 MPI processes
>>>> >   type: cg
>>>> >   maximum iterations=10000
>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>> >   left preconditioning
>>>> >   using nonzero initial guess
>>>> >   using PRECONDITIONED norm type for convergence test
>>>> > PC Object:(poisson_) 1 MPI processes
>>>> >   type: gamg
>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>> >       Cycles per PCApply=1
>>>> >       Using Galerkin computed coarse grid matrices
>>>> >       GAMG specific options
>>>> >         Threshold for dropping small values from graph 0
>>>> >         AGG specific options
>>>> >           Symmetric graph false
>>>> >   Coarse grid solver -- level -------------------------------
>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>> >       type: gmres
>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>> Orthogonalization with no iterative refinement
>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>> >       maximum iterations=1, initial guess is zero
>>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>> >       left preconditioning
>>>> >       using NONE norm type for convergence test
>>>> >
>>>> >
>>>> > both logs are attached.
>>>> >
>>>> >
>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> >
>>>> >    This can't work. You can't use a GMRES inside a CG.   Try changing
>>>> to -poisson_mg_coarse_ksp_type preonly
>>>> >
>>>> > KSP Object:(poisson_) 1 MPI processes
>>>> >   type: cg
>>>> >
>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>> >       type: gmres
>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>> Orthogonalization with no iterative refinement
>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>> >       maximum iterations=1, initial guess is zero
>>>> >
>>>> >
>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>> > >
>>>> > >
>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>> knepley at gmail.com> wrote:
>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>> wrote:
>>>> > > Hi all,
>>>> > >
>>>> > > I've been having some trouble with what should be a relatively
>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>> > >
>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>> test problem, 32x32 at Re 100
>>>> > >
>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>>> set the null space.  This is for a 2D Poisson system with no immersed
>>>> boundary and so the null space is the constant vector.
>>>> > >
>>>> > >   MatNullSpace nsp;
>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>>> &nsp); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>> > >
>>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>>> not be created yet.
>>>> > >
>>>> > > For questions about solvers, we HAVE to see the complete output of
>>>> -ksp_view so we
>>>> > > know what we are dealing with. Its also nice to have
>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>> > >
>>>> > >   Matt
>>>> > >
>>>> > > Yes -- sorry, those are both in inline files and are called in the
>>>> reverse order that I wrote them out.
>>>> > >
>>>> > > I've attached the output of
>>>> > >
>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>> -poisson_ksp_converged_reason > kspview.log
>>>> > >
>>>> > >
>>>> > >
>>>> > > And then setup the KSP with
>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>> > >
>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>> updated.
>>>> > >
>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>> > >
>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>> > >
>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>> > >
>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>> KSP_DIVERGED_INDEFINITE_PC
>>>> > >
>>>> > > This is weird, especially because if we change nsmooths to 2, it
>>>> runs for 264 timesteps and the returns the same error.  But we have
>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>> right?
>>>> > >
>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>> > >
>>>> > > Change nsmooths to 4 and it runs to completion.
>>>> > >
>>>> > > It seems like either gamg's behavior has changed, or that
>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>> explicitly in addition to MatSetNullSpace?
>>>> > >
>>>> > > Thanks,
>>>> > > Gil Forsyth
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > > -- Norbert Wiener
>>>> > >
>>>> > > <kspview.log>
>>>> >
>>>> >
>>>> > <kspview3.5.4><kspview3.6.1>
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/fa7dec4d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpi_n2_indefinite_pc.log
Type: text/x-log
Size: 9834 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150930/fa7dec4d/attachment-0001.bin>


More information about the petsc-users mailing list