[petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace

Gil Forsyth gforsyth at gwu.edu
Tue Sep 29 12:12:44 CDT 2015 

Ah, got it.  I'll checkout the master branch and see if the behavior
persists.

Many thanks,
Gil

On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>> the behavior seems to have changed between versions.  The only difference
>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>> MatSetNullSpace.
>>
>
> Mark made some GAMG changes which were later reversed because they had
> unintended consequences like this.
> I think what Barry means is, "you should get the behavior you expect using
> the master branch from PETSc development"
>
>   Thanks,
>
>      Matt
>
>
>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Update your PETSc
>>>
>>>
>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> >
>>> > Hi Barry,
>>> >
>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>> works with:
>>> >
>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>> kspview3.5.4
>>> >
>>> > and shows that the coarse grid solver is of type:preonly
>>> >
>>> > running the newer version that uses MatSetNullSpace in its stead and
>>> adding in -poisson_mg_coarse_ksp_type preonly
>>> >
>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>> kspview3.6.1
>>> >
>>> > still shows
>>> >
>>> > KSP Object:(poisson_) 1 MPI processes
>>> >   type: cg
>>> >   maximum iterations=10000
>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> >   left preconditioning
>>> >   using nonzero initial guess
>>> >   using PRECONDITIONED norm type for convergence test
>>> > PC Object:(poisson_) 1 MPI processes
>>> >   type: gamg
>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>> >       Cycles per PCApply=1
>>> >       Using Galerkin computed coarse grid matrices
>>> >       GAMG specific options
>>> >         Threshold for dropping small values from graph 0
>>> >         AGG specific options
>>> >           Symmetric graph false
>>> >   Coarse grid solver -- level -------------------------------
>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> >       type: gmres
>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> >         GMRES: happy breakdown tolerance 1e-30
>>> >       maximum iterations=1, initial guess is zero
>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> >       left preconditioning
>>> >       using NONE norm type for convergence test
>>> >
>>> >
>>> > both logs are attached.
>>> >
>>> >
>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >    This can't work. You can't use a GMRES inside a CG.   Try changing
>>> to -poisson_mg_coarse_ksp_type preonly
>>> >
>>> > KSP Object:(poisson_) 1 MPI processes
>>> >   type: cg
>>> >
>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> >       type: gmres
>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> >         GMRES: happy breakdown tolerance 1e-30
>>> >       maximum iterations=1, initial guess is zero
>>> >
>>> >
>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > >
>>> > >
>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>> wrote:
>>> > > Hi all,
>>> > >
>>> > > I've been having some trouble with what should be a relatively
>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>> > >
>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity test
>>> problem, 32x32 at Re 100
>>> > >
>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
>>> the null space.  This is for a 2D Poisson system with no immersed boundary
>>> and so the null space is the constant vector.
>>> > >
>>> > >   MatNullSpace nsp;
>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>> &nsp); CHKERRQ(ierr);
>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>> > >
>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>> not be created yet.
>>> > >
>>> > > For questions about solvers, we HAVE to see the complete output of
>>> -ksp_view so we
>>> > > know what we are dealing with. Its also nice to have
>>> -ksp_monitor_true_residual -ksp_converged_reason
>>> > >
>>> > >   Matt
>>> > >
>>> > > Yes -- sorry, those are both in inline files and are called in the
>>> reverse order that I wrote them out.
>>> > >
>>> > > I've attached the output of
>>> > >
>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>> -poisson_ksp_converged_reason > kspview.log
>>> > >
>>> > >
>>> > >
>>> > > And then setup the KSP with
>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>> > >
>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>> updated.
>>> > >
>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>> > >
>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>> > >
>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>> > >
>>> > > and the same code diverges after 1 timestep and returns a -8
>>> KSP_DIVERGED_INDEFINITE_PC
>>> > >
>>> > > This is weird, especially because if we change nsmooths to 2, it
>>> runs for 264 timesteps and the returns the same error.  But we have
>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>> right?
>>> > >
>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>> > >
>>> > > Change nsmooths to 4 and it runs to completion.
>>> > >
>>> > > It seems like either gamg's behavior has changed, or that
>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>> explicitly in addition to MatSetNullSpace?
>>> > >
>>> > > Thanks,
>>> > > Gil Forsyth
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > > -- Norbert Wiener
>>> > >
>>> > > <kspview.log>
>>> >
>>> >
>>> > <kspview3.5.4><kspview3.6.1>
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150929/58ba6579/attachment.html>

More information about the petsc-users mailing list