[petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace

Olivier Mesnard olivier.mesnard8 at gmail.com
Tue Oct 25 19:57:38 CDT 2016 

On 25 October 2016 at 19:59, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 25, 2016 at 6:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> > On Oct 25, 2016, at 5:39 PM, Olivier Mesnard <
>> olivier.mesnard8 at gmail.com> wrote:
>> >
>> > On 25 October 2016 at 17:51, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> >   Olivier,
>> >
>> >     In theory you do not need to change anything else. Are you using a
>> different matrix object for the velocity_ksp object than the poisson_ksp
>> object?
>> >
>> > ​The matrix is different for the velocity_ksp and the poisson_ksp​.
>> >
>> >     The code change in PETSc is very little but we have a report from
>> another CFD user who also had problems with the change so there may be some
>> subtle bug that we can't figure out causing things to not behave properly.
>> >
>> >    First run the 3.7.4 code with -poisson_ksp_view and verify that when
>> it prints the matrix information it prints something like has attached null
>> space if it does not print that it means that somehow the matrix is not
>> properly getting the matrix attached.
>> >
>> > ​When running with 3.7.4 and -poisson_ksp_view, the output shows that
>> the nullspace is not attached to the KSP (as it was with 3.5.4)​; however
>> the print statement is now under the Mat info (which is expected when
>> moving from KSPSetNullSpace to MatSetNullSpace?).
>>
>>    Good, this is how it should be.
>> >
>> >     Though older versions had MatSetNullSpace() they didn't necessarily
>> associate it with the KSP so it was not expected to work as a replacement
>> for KSPSetNullSpace() with older versions.
>> >
>> >     Because our other user had great difficulty trying to debug the
>> issue feel free to send us at petsc-maint at mcs.anl.gov your code with
>> instructions on building and running and we can try to track down the
>> problem. Better than hours and hours spent with fruitless email. We will,
>> of course, not distribute the code and will delete in when we are finished
>> with it.
>> >
>> > ​The code is open-source and hosted on GitHub (
>> https://github.com/barbagroup/PetIBM)​.
>> > I just pushed the branches `feature-compatible-petsc-3.7` and
>> `revert-compatible-petsc-3.5` that I used to observe this problem.
>> >
>>     Thanks, I'll get back to you if I discover anything
>
>
> Obviously GAMG is behaving quite differently (1 vs 2 levels and a much
> sparser coarse problem in 3.7).
>
> Could you try one thing for me before we start running it? Run with
>
>   -poisson_mg_coarse_sub_pc_type svd
>
> and see what happens on 2 procs for 3.7?
>
> ​Hi Matt,

With
-poisson_mg_coarse_sub_pc_type svd
​,​
it ran normally on 1 proc but not on 2 procs the end of the output says:

"** On entry to DGESVD parameter number  6 had an illegal value
"

I attached the log file.
​


>   Thanks,
>
>      Matt
>
>
>>
>> > PETSc (both 3.5.4 and 3.7.4) was configured as follow:
>> > export PETSC_ARCH="linux-gnu-dbg"
>> > ./configure --PETSC_ARCH=$PETSC_ARCH \
>> >       --with-cc=gcc \
>> >       --with-cxx=g++ \
>> >       --with-fc=gfortran \
>> >       --COPTFLAGS="-O0" \
>> >       --CXXOPTFLAGS="-O0" \
>> >       --FOPTFLAGS="-O0" \
>> >       --with-debugging=1 \
>> >       --download-fblaslapack \
>> >       --download-mpich \
>> >       --download-hypre \
>> >       --download-yaml \
>> >       --with-x=1
>> >
>> > Our code was built using the following commands:​
>> > mkdir petibm-build
>> > cd petibm-build
>> > ​export PETSC_DIR=<directory of PETSc>
>> > export PETSC_ARCH="linux-gnu-dbg"
>> > export PETIBM_DIR=<directory of PetIBM git repo>
>> > $PETIBM_DIR/configure --prefix=$PWD \
>> >       CXX=$PETSC_DIR/$PETSC_ARCH/bin/mpicxx \
>> >       CXXFLAGS="-g -O0 -std=c++11"​
>> > make all
>> > make install
>> >
>> > ​Then
>> > cd examples
>> > make examples​
>> >
>> > ​The example of the lid-driven cavity I was talking about can be found
>> in the folder `examples/2d/convergence/lidDrivenCavity20/20/`​
>> >
>> > To run it:
>> > mpiexec -n N <path-to-petibm-build>/bin/petibm2d -directory
>> <path-to-example>
>> >
>> > Let me know if you need more info. Thank you.
>> >
>> >    Barry
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > > On Oct 25, 2016, at 4:38 PM, Olivier Mesnard <
>> olivier.mesnard8 at gmail.com> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > We develop a CFD code using the PETSc library that solves the
>> Navier-Stokes equations using the fractional-step method from Perot (1993).
>> > > At each time-step, we solve two systems: one for the velocity field,
>> the other, a Poisson system, for the pressure field.
>> > > One of our test-cases is a 2D lid-driven cavity flow (Re=100) on a
>> 20x20 grid using 1 or 2 procs.
>> > > For the Poisson system, we usually use CG preconditioned with GAMG.
>> > >
>> > > So far, we have been using PETSc-3.5.4, and we would like to update
>> the code with the latest release: 3.7.4.
>> > >
>> > > As suggested in the changelog of 3.6, we replaced the routine
>> `KSPSetNullSpace()` with `MatSetNullSpace()`.
>> > >
>> > > Here is the list of options we use to configure the two solvers:
>> > > * Velocity solver: prefix `-velocity_`
>> > >   -velocity_ksp_type bcgs
>> > >   -velocity_ksp_rtol 1.0E-08
>> > >   -velocity_ksp_atol 0.0
>> > >   -velocity_ksp_max_it 10000
>> > >   -velocity_pc_type jacobi
>> > >   -velocity_ksp_view
>> > >   -velocity_ksp_monitor_true_residual
>> > >   -velocity_ksp_converged_reason
>> > > * Poisson solver: prefix `-poisson_`
>> > >   -poisson_ksp_type cg
>> > >   -poisson_ksp_rtol 1.0E-08
>> > >   -poisson_ksp_atol 0.0
>> > >   -poisson_ksp_max_it 20000
>> > >   -poisson_pc_type gamg
>> > >   -poisson_pc_gamg_type agg
>> > >   -poisson_pc_gamg_agg_nsmooths 1
>> > >   -poissonksp_view
>> > >   -poisson_ksp_monitor_true_residual
>> > >   -poisson_ksp_converged_reason
>> > >
>> > > With 3.5.4, the case runs normally on 1 or 2 procs.
>> > > With 3.7.4, the case runs normally on 1 proc but not on 2.
>> > > Why? The Poisson solver diverges because of an indefinite
>> preconditioner (only with 2 procs).
>> > >
>> > > We also saw that the routine `MatSetNullSpace()` was already
>> available in 3.5.4.
>> > > With 3.5.4, replacing `KSPSetNullSpace()` with `MatSetNullSpace()`
>> led to the Poisson solver diverging because of an indefinite matrix (on 1
>> and 2 procs).
>> > >
>> > > Thus, we were wondering if we needed to update something else for the
>> KSP, and not just modifying the name of the routine?
>> > >
>> > > I have attached the output files from the different cases:
>> > > * `run-petsc-3.5.4-n1.log` (3.5.4, `KSPSetNullSpace()`, n=1)
>> > > * `run-petsc-3.5.4-n2.log`
>> > > * `run-petsc-3.5.4-nsp-n1.log` (3.5.4, `MatSetNullSpace()`, n=1)
>> > > * `run-petsc-3.5.4-nsp-n2.log`
>> > > * `run-petsc-3.7.4-n1.log` (3.7.4, `MatSetNullSpace()`, n=1)
>> > > * `run-petsc-3.7.4-n2.log`
>> > >
>> > > Thank you for your help,
>> > > Olivier
>> > > <run-petsc-3.5.4-n1.log><run-petsc-3.5.4-n2.log><run-petsc-3
>> .5.4-nsp-n1.log><run-petsc-3.5.4-nsp-n2.log><run-petsc-3.7.4
>> -n1.log><run-petsc-3.7.4-n2.log>
>> >
>> >
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20161025/5a4cf47a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run-petsc-3.7.4-n2.log
Type: application/octet-stream
Size: 4114 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20161025/5a4cf47a/attachment.obj>

More information about the petsc-users mailing list