[petsc-users] Trying to apply FieldSplitPC by reading bloc matrix

Gilles Steiner gilles.steiner at epfl.ch
Thu Jan 8 06:21:33 CST 2015


Hello again.

Thank you all for the answers. Here is an update on this subject.

First, some answers to the comments/questions received :

Answer to Jed Brown :

"What FE discretization?  We don't recommend using files as part of your
workflow, but if you're just experimenting, you could start with

src/ksp/ksp/examples/tests/ex11.c which solves a Q1-P0 Stokes problem
 From Underworld by reading the blocks in as matrices.  So start there
and let us know how it goes."

I tried several FE discretizations (P1-P1-stabilized, P2-P1, 
P1Bubble-P1) on tetrahedral meshes of a cube (Driven cavity).
Thanks for the reference to ex11.c, but do you have any suggestion of 
options to apply on this to have a good behaviour of the solver ?

I compiled the example (src/ksp/ksp/examples/tests/ex11.c) and 
downloaded the input file from 
http://ftp.mcs.anl.gov/pub/petsc/matrices/underworld32.gz
but the convergence of the solver is a bit disappointing with the 
default options (simply running with mpirun -np 1 ex11 -f 
underworld32.gz) :

[0] lnu = 2046 | lnp = 1024
[0] s_u = 0 | e_u = 2046
[0] s_p = 0 | e_p = 1024
[0] is_u (offset) = 0
[0] is_p (offset) = 2046
   Residual norms for fc_ solve.
   0 KSP preconditioned resid norm 1.901623820248e+11 true resid norm 
3.163503865730e-01 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 1.232616307278e+11 true resid norm 
3.110902550317e-01 ||r(i)||/||b|| 9.833724510397e-01
   2 KSP preconditioned resid norm 8.842498650573e+10 true resid norm 
3.121703549116e-01 ||r(i)||/||b|| 9.867867028497e-01
   3 KSP preconditioned resid norm 6.376643354056e+10 true resid norm 
3.216506148074e-01 ||r(i)||/||b|| 1.016754296689e+00
   4 KSP preconditioned resid norm 4.507200049791e+10 true resid norm 
3.453307829735e-01 ||r(i)||/||b|| 1.091608537971e+00
   5 KSP preconditioned resid norm 3.165593374571e+10 true resid norm 
3.787554909641e-01 ||r(i)||/||b|| 1.197265775671e+00
   6 KSP preconditioned resid norm 1.886092467773e+10 true resid norm 
4.169668186513e-01 ||r(i)||/||b|| 1.318053766800e+00
   7 KSP preconditioned resid norm 1.022004643856e+10 true resid norm 
4.423235667291e-01 ||r(i)||/||b|| 1.398207764248e+00
   8 KSP preconditioned resid norm 5.318995340694e+09 true resid norm 
4.558356174770e-01 ||r(i)||/||b|| 1.440920058341e+00
   9 KSP preconditioned resid norm 3.160919062173e+09 true resid norm 
4.608182603598e-01 ||r(i)||/||b|| 1.456670451242e+00
  10 KSP preconditioned resid norm 2.172582250050e+09 true resid norm 
4.643714263977e-01 ||r(i)||/||b|| 1.467902193603e+00
...
184 KSP preconditioned resid norm 1.890479367808e+06 true resid norm 
3.557809738610e-01 ||r(i)||/||b|| 1.124642134044e+00
-- velocity vector values --
   Min(u)  = -0.000000 [loc=1994]
   Max(u)  = 0.000000 [loc=1310]
   Norm(u) = 0.000000
   Sum(u)  = 0.000000
-- pressure vector values --
   Min(p)  = -4.877385 [loc=1015]
   Max(p)  = 4.616608 [loc=4]
   Norm(p) = 88.261241
   Sum(p)  = 0.000000
-- Full vector values --
   Min(u,p)  = -4.877385 [loc=3061]
   Max(u,p)  = 4.616608 [loc=2050]
   Norm(u,p) = 88.261241
   Sum(u,p)  = 0.000000

The only modification I have done to the code is to add the line :   
KSPMonitorSet(ksp_A, KSPMonitorTrueResidualNorm, PETSC_NULL, PETSC_NULL);
to log the convergence.




Answer to Barry Smith and Matthew Knepley :

"In both cases the preconditioned residual is decreasing nicely but the 
unpreconditioned residual is not decreasing, so something is wrong even 
in the sequential case!
> 14 KSP preconditioned resid norm 5.169032607321e-10 true resid norm 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04

    So you need to go back to the sequential case and see what is going on. Don't even touch the parallel case until the true residual is converging properly for the sequential.  First try running with -ksp_pc_side right and watch the residuals. Next run with direct solvers everywhere you can and see what happens."

I guess I figured out the issue here. I tried to speedup iterations by 
adding the options -fieldsplit_0_ksp_max_it and -fieldsplit_1_ksp_max_it 
as suggested in src/snes/examples/tutorials/ex70.c (they are acting on 
rtol instead of max_it). But doing this seems to cause the wrong 
convergence behaviour.

example with max_it=100 :
   0 KSP preconditioned resid norm 6.492190058577e+02 true resid norm 
1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 1.969844390083e-02 true resid norm 
1.509247865473e-05 ||r(i)||/||b|| 1.372043514066e-06
   2 KSP preconditioned resid norm 4.926131518900e-07 true resid norm 
7.062431971029e-09 ||r(i)||/||b|| 6.420392700936e-10
   3 KSP preconditioned resid norm 8.183544025740e-12 true resid norm 
6.418682323277e-09 ||r(i)||/||b|| 5.835165748434e-10

same with max_it=5 :
   0 KSP preconditioned resid norm 6.191842548447e+02 true resid norm 
1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 5.474611327659e+01 true resid norm 
3.793016384570e-01 ||r(i)||/||b|| 3.448196713245e-02
   2 KSP preconditioned resid norm 9.064410031719e+00 true resid norm 
5.519394994263e-02 ||r(i)||/||b|| 5.017631812966e-03
   3 KSP preconditioned resid norm 2.162169261727e+00 true resid norm 
4.136759289559e-02 ||r(i)||/||b|| 3.760690263235e-03
   4 KSP preconditioned resid norm 4.487680249963e-01 true resid norm 
9.486438220687e-03 ||r(i)||/||b|| 8.624034746079e-04
   5 KSP preconditioned resid norm 1.100928562187e-01 true resid norm 
1.199868960902e-02 ||r(i)||/||b|| 1.090789964456e-03
   6 KSP preconditioned resid norm 2.826378652048e-02 true resid norm 
1.176242279335e-02 ||r(i)||/||b|| 1.069311163032e-03
   7 KSP preconditioned resid norm 5.075277611809e-03 true resid norm 
1.191372816799e-02 ||r(i)||/||b|| 1.083066197090e-03
   8 KSP preconditioned resid norm 1.288581278231e-03 true resid norm 
1.191807770501e-02 ||r(i)||/||b|| 1.083461609546e-03
   9 KSP preconditioned resid norm 3.446344125585e-04 true resid norm 
1.190778433764e-02 ||r(i)||/||b|| 1.082525848877e-03
  10 KSP preconditioned resid norm 7.282719316035e-05 true resid norm 
1.191201986724e-02 ||r(i)||/||b|| 1.082910897022e-03
  11 KSP preconditioned resid norm 1.714618056930e-05 true resid norm 
1.191095763273e-02 ||r(i)||/||b|| 1.082814330249e-03
  12 KSP preconditioned resid norm 4.317573472482e-06 true resid norm 
1.191112698437e-02 ||r(i)||/||b|| 1.082829725851e-03
  13 KSP preconditioned resid norm 9.546213335221e-07 true resid norm 
1.191112173116e-02 ||r(i)||/||b|| 1.082829248287e-03
  14 KSP preconditioned resid norm 2.155641210237e-07 true resid norm 
1.191111695140e-02 ||r(i)||/||b|| 1.082828813763e-03
  15 KSP preconditioned resid norm 5.936176282735e-08 true resid norm 
1.191111896030e-02 ||r(i)||/||b|| 1.082828996391e-03
  16 KSP preconditioned resid norm 1.362639176327e-08 true resid norm 
1.191111833902e-02 ||r(i)||/||b|| 1.082828939911e-03
  17 KSP preconditioned resid norm 3.042659891281e-09 true resid norm 
1.191111844668e-02 ||r(i)||/||b|| 1.082828949699e-03
  18 KSP preconditioned resid norm 8.958920024015e-10 true resid norm 
1.191111844183e-02 ||r(i)||/||b|| 1.082828949257e-03
  19 KSP preconditioned resid norm 2.059234060666e-10 true resid norm 
1.191111843406e-02 ||r(i)||/||b|| 1.082828948551e-03


I will do more tests with this but if someone has some clue on how to 
correctly run the ex11 it would be of great help.

Thanks again,
Gilles


On 12/20/2014 02:29 AM, Matthew Knepley wrote:
>
> On Dec 19, 2014 6:14 AM, "Gilles Steiner" <gilles.steiner at epfl.ch 
> <mailto:gilles.steiner at epfl.ch>> wrote:
> >
> > Hello Petsc Users,
> >
> > I have an issue trying to use FiledSplitPC in parallel.
> >
> > My goal : I want to get a linear system from petsc binary files and 
> solve this in parallel with the FieldSplitPC.
> >
> > The problem I want to solve is an FE approximation of the Stokes 
> equations.
> >
> > Skipping the details, my code looks like :
> >
> > // Reading the four blocs UU, UP, PU and PP
> > for(int i=0; i < 4; ++i)
> > {
> >       string name = matrix + to_string(i) + ".petscbin";
> >       PetscViewer    PETSC_matreader;
> >       PetscViewerBinaryOpen(PETSC_COMM_WORLD, name.c_str(), 
> FILE_MODE_READ, &PETSC_matreader);
> >       MatCreate(PETSC_COMM_WORLD,&PETSC_subA[i]);
> >       MatLoad(PETSC_subA[i],PETSC_matreader);
> >       PetscViewerDestroy(&PETSC_matreader);
> > }
> >
> > // Reading the RHS vector and duplicating it to create the solution 
> vector
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(), FILE_MODE_READ, 
> &PETSC_vecreader);
> > VecCreate(PETSC_COMM_WORLD,&PETSC_rhs);
> > VecLoad(PETSC_rhs,PETSC_vecreader);
> > PetscViewerDestroy(&PETSC_vecreader);
> > VecDuplicate(PETSC_rhs,&PETSC_sol);
> >
> > // Create global matrixwith MatCreateNest
> > MatCreateNest(PETSC_COMM_WORLD, 2, NULL, 2, NULL, PETSC_subA, &PETSC_A);
> > MatNestGetISs(PETSC_A, PETSC_isg, NULL);
> >
> > // Setting up the ksp and precond
> > KSPCreate(PETSC_COMM_WORLD,&PETSC_ksp);
> > KSPSetOperators(PETSC_ksp,PETSC_A,PETSC_A);
> > KSPSetFromOptions(PETSC_ksp);
> >
> > KSPGetPC(PETSC_ksp, &PETSC_pc);
> > PCSetType(PETSC_pc, PCFIELDSPLIT);
> > PCFieldSplitSetIS(PETSC_pc, "0", PETSC_isg[0]);
> > PCFieldSplitSetIS(PETSC_pc, "1", PETSC_isg[1]);
> > PCSetFromOptions(PETSC_pc);
> >
> > // Solving the system and writing back the solution in rhs file
> > KSPSolve(PETSC_ksp,PETSC_rhs,PETSC_sol);
> >
> > PetscViewer    PETSC_vecwriter;
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(), 
> FILE_MODE_WRITE, &PETSC_vecwriter);
> > VecView(PETSC_sol,PETSC_vecwriter);
> > PetscViewerDestroy(&PETSC_vecwriter);
> >
> > When I run it with 1 proc, everything works fine and I get the 
> correct solution :
> >
>
> When you get this behavior with true residuals, it very often arises 
> from failure to account for a pressure null space. Do you have one?
>
>    Matt
>
>   0 KSP preconditioned resid norm 1.271697253018e+03 true resid norm 
> 2.400000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 5.009545069728e+01 true resid norm 
> 9.166803391041e-02 ||r(i)||/||b|| 3.819501412934e-03
> >   2 KSP preconditioned resid norm 6.460631387766e+00 true resid norm 
> 4.995542253831e-02 ||r(i)||/||b|| 2.081475939096e-03
> >   3 KSP preconditioned resid norm 1.155895209298e+00 true resid norm 
> 1.515734830704e-02 ||r(i)||/||b|| 6.315561794600e-04
> >   4 KSP preconditioned resid norm 7.407384739634e-02 true resid norm 
> 9.992802256200e-03 ||r(i)||/||b|| 4.163667606750e-04
> >   5 KSP preconditioned resid norm 1.574456882990e-02 true resid norm 
> 9.994876664681e-03 ||r(i)||/||b|| 4.164531943617e-04
> >   6 KSP preconditioned resid norm 2.383022349902e-03 true resid norm 
> 9.990760645581e-03 ||r(i)||/||b|| 4.162816935659e-04
> >   7 KSP preconditioned resid norm 6.175379834254e-04 true resid norm 
> 9.990821066459e-03 ||r(i)||/||b|| 4.162842111025e-04
> >   8 KSP preconditioned resid norm 6.867982689960e-05 true resid norm 
> 9.990532094790e-03 ||r(i)||/||b|| 4.162721706163e-04
> >   9 KSP preconditioned resid norm 1.041091257246e-05 true resid norm 
> 9.990558069113e-03 ||r(i)||/||b|| 4.162732528797e-04
> >  10 KSP preconditioned resid norm 1.447793722489e-06 true resid norm 
> 9.990557786778e-03 ||r(i)||/||b|| 4.162732411158e-04
> >  11 KSP preconditioned resid norm 2.139317335854e-07 true resid norm 
> 9.990557262754e-03 ||r(i)||/||b|| 4.162732192814e-04
> >  12 KSP preconditioned resid norm 4.383129810322e-08 true resid norm 
> 9.990557306920e-03 ||r(i)||/||b|| 4.162732211217e-04
> >  13 KSP preconditioned resid norm 3.351461304399e-09 true resid norm 
> 9.990557311707e-03 ||r(i)||/||b|| 4.162732213211e-04
> >  14 KSP preconditioned resid norm 5.169032607321e-10 true resid norm 
> 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04
> >
> > [14:49:10::INFO   ] System Solved. Final tolerance reached is 
> 5.16903e-10 in 14 iterations.
> >
> > But if I do it with 2 procs, the resolution seems fine but the 
> solution is wrong :
> >   0 KSP preconditioned resid norm 1.247694088756e+03 true resid norm 
> 2.400000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 4.481954484303e+01 true resid norm 
> 5.277507840772e-01 ||r(i)||/||b|| 2.198961600321e-02
> >   2 KSP preconditioned resid norm 1.110647693456e+01 true resid norm 
> 4.005558168981e-02 ||r(i)||/||b|| 1.668982570409e-03
> >   3 KSP preconditioned resid norm 1.220368027409e+00 true resid norm 
> 1.877650834971e-02 ||r(i)||/||b|| 7.823545145714e-04
> >   4 KSP preconditioned resid norm 2.834261749922e-01 true resid norm 
> 1.613967205264e-02 ||r(i)||/||b|| 6.724863355265e-04
> >   5 KSP preconditioned resid norm 4.215090288154e-02 true resid norm 
> 1.562561614611e-02 ||r(i)||/||b|| 6.510673394212e-04
> >   6 KSP preconditioned resid norm 1.209476134754e-02 true resid norm 
> 1.563808960492e-02 ||r(i)||/||b|| 6.515870668718e-04
> >   7 KSP preconditioned resid norm 2.038835108629e-03 true resid norm 
> 1.564163643064e-02 ||r(i)||/||b|| 6.517348512765e-04
> >   8 KSP preconditioned resid norm 1.928844666836e-04 true resid norm 
> 1.564072761376e-02 ||r(i)||/||b|| 6.516969839065e-04
> >   9 KSP preconditioned resid norm 3.138911950605e-05 true resid norm 
> 1.564047323377e-02 ||r(i)||/||b|| 6.516863847403e-04
> >  10 KSP preconditioned resid norm 4.950062975470e-06 true resid norm 
> 1.564048216528e-02 ||r(i)||/||b|| 6.516867568865e-04
> >  11 KSP preconditioned resid norm 7.677242244159e-07 true resid norm 
> 1.564049253364e-02 ||r(i)||/||b|| 6.516871889019e-04
> >  12 KSP preconditioned resid norm 1.870521888617e-07 true resid norm 
> 1.564049269566e-02 ||r(i)||/||b|| 6.516871956526e-04
> >  13 KSP preconditioned resid norm 3.077235724319e-08 true resid norm 
> 1.564049264800e-02 ||r(i)||/||b|| 6.516871936666e-04
> >  14 KSP preconditioned resid norm 6.584409191524e-09 true resid norm 
> 1.564049264183e-02 ||r(i)||/||b|| 6.516871934095e-04
> >  15 KSP preconditioned resid norm 1.091619359913e-09 true resid norm 
> 1.564049263170e-02 ||r(i)||/||b|| 6.516871929874e-04
> >
> > [15:10:58::INFO   ] System Solved. Final tolerance reached is 
> 1.09162e-09 in 15 iterations.
> >
> > Any idea of what is wrong with this ? Is it the code or the base 
> concept ?
> >
> > Thank you.
> > Gilles
> >
>


-- 
-------------------------------------------------------------------------
| STEINER Gilles
=========================================================================
| PostDoc  @ EPFL
| Engineer @ Ycoor Systems SA
-------------------------------------------------------------------------
|         |         EPFL                 |      Ycoor Systems SA
| Unit    | SB / MATHICSE / GR-PI        |            -
| Office  | MA C2 643                    |            -
| Tel     | +41 21 693 42 46             | +41 43 535 09 30
| @       | gilles.steiner at epfl.ch       | steiner at ycoorsystems.com
| Www     | http://mathicse.epfl.ch/asn/ | http://www.ycoorsystems.com/
| Address | EPFL SB MATHICSE GR-PI       | Wallisellenstrasse 333
|         | Station 8                    | CH-8050 Zürich
|         | CH-1015 Lausanne             |
-------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150108/bd06e57c/attachment.html>


More information about the petsc-users mailing list