[petsc-users] Trying to apply FieldSplitPC by reading bloc matrix
Gilles Steiner
gilles.steiner at epfl.ch
Thu Jan 8 06:21:33 CST 2015
Hello again.
Thank you all for the answers. Here is an update on this subject.
First, some answers to the comments/questions received :
Answer to Jed Brown :
"What FE discretization? We don't recommend using files as part of your
workflow, but if you're just experimenting, you could start with
src/ksp/ksp/examples/tests/ex11.c which solves a Q1-P0 Stokes problem
From Underworld by reading the blocks in as matrices. So start there
and let us know how it goes."
I tried several FE discretizations (P1-P1-stabilized, P2-P1,
P1Bubble-P1) on tetrahedral meshes of a cube (Driven cavity).
Thanks for the reference to ex11.c, but do you have any suggestion of
options to apply on this to have a good behaviour of the solver ?
I compiled the example (src/ksp/ksp/examples/tests/ex11.c) and
downloaded the input file from
http://ftp.mcs.anl.gov/pub/petsc/matrices/underworld32.gz
but the convergence of the solver is a bit disappointing with the
default options (simply running with mpirun -np 1 ex11 -f
underworld32.gz) :
[0] lnu = 2046 | lnp = 1024
[0] s_u = 0 | e_u = 2046
[0] s_p = 0 | e_p = 1024
[0] is_u (offset) = 0
[0] is_p (offset) = 2046
Residual norms for fc_ solve.
0 KSP preconditioned resid norm 1.901623820248e+11 true resid norm
3.163503865730e-01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP preconditioned resid norm 1.232616307278e+11 true resid norm
3.110902550317e-01 ||r(i)||/||b|| 9.833724510397e-01
2 KSP preconditioned resid norm 8.842498650573e+10 true resid norm
3.121703549116e-01 ||r(i)||/||b|| 9.867867028497e-01
3 KSP preconditioned resid norm 6.376643354056e+10 true resid norm
3.216506148074e-01 ||r(i)||/||b|| 1.016754296689e+00
4 KSP preconditioned resid norm 4.507200049791e+10 true resid norm
3.453307829735e-01 ||r(i)||/||b|| 1.091608537971e+00
5 KSP preconditioned resid norm 3.165593374571e+10 true resid norm
3.787554909641e-01 ||r(i)||/||b|| 1.197265775671e+00
6 KSP preconditioned resid norm 1.886092467773e+10 true resid norm
4.169668186513e-01 ||r(i)||/||b|| 1.318053766800e+00
7 KSP preconditioned resid norm 1.022004643856e+10 true resid norm
4.423235667291e-01 ||r(i)||/||b|| 1.398207764248e+00
8 KSP preconditioned resid norm 5.318995340694e+09 true resid norm
4.558356174770e-01 ||r(i)||/||b|| 1.440920058341e+00
9 KSP preconditioned resid norm 3.160919062173e+09 true resid norm
4.608182603598e-01 ||r(i)||/||b|| 1.456670451242e+00
10 KSP preconditioned resid norm 2.172582250050e+09 true resid norm
4.643714263977e-01 ||r(i)||/||b|| 1.467902193603e+00
...
184 KSP preconditioned resid norm 1.890479367808e+06 true resid norm
3.557809738610e-01 ||r(i)||/||b|| 1.124642134044e+00
-- velocity vector values --
Min(u) = -0.000000 [loc=1994]
Max(u) = 0.000000 [loc=1310]
Norm(u) = 0.000000
Sum(u) = 0.000000
-- pressure vector values --
Min(p) = -4.877385 [loc=1015]
Max(p) = 4.616608 [loc=4]
Norm(p) = 88.261241
Sum(p) = 0.000000
-- Full vector values --
Min(u,p) = -4.877385 [loc=3061]
Max(u,p) = 4.616608 [loc=2050]
Norm(u,p) = 88.261241
Sum(u,p) = 0.000000
The only modification I have done to the code is to add the line :
KSPMonitorSet(ksp_A, KSPMonitorTrueResidualNorm, PETSC_NULL, PETSC_NULL);
to log the convergence.
Answer to Barry Smith and Matthew Knepley :
"In both cases the preconditioned residual is decreasing nicely but the
unpreconditioned residual is not decreasing, so something is wrong even
in the sequential case!
> 14 KSP preconditioned resid norm 5.169032607321e-10 true resid norm 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04
So you need to go back to the sequential case and see what is going on. Don't even touch the parallel case until the true residual is converging properly for the sequential. First try running with -ksp_pc_side right and watch the residuals. Next run with direct solvers everywhere you can and see what happens."
I guess I figured out the issue here. I tried to speedup iterations by
adding the options -fieldsplit_0_ksp_max_it and -fieldsplit_1_ksp_max_it
as suggested in src/snes/examples/tutorials/ex70.c (they are acting on
rtol instead of max_it). But doing this seems to cause the wrong
convergence behaviour.
example with max_it=100 :
0 KSP preconditioned resid norm 6.492190058577e+02 true resid norm
1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP preconditioned resid norm 1.969844390083e-02 true resid norm
1.509247865473e-05 ||r(i)||/||b|| 1.372043514066e-06
2 KSP preconditioned resid norm 4.926131518900e-07 true resid norm
7.062431971029e-09 ||r(i)||/||b|| 6.420392700936e-10
3 KSP preconditioned resid norm 8.183544025740e-12 true resid norm
6.418682323277e-09 ||r(i)||/||b|| 5.835165748434e-10
same with max_it=5 :
0 KSP preconditioned resid norm 6.191842548447e+02 true resid norm
1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP preconditioned resid norm 5.474611327659e+01 true resid norm
3.793016384570e-01 ||r(i)||/||b|| 3.448196713245e-02
2 KSP preconditioned resid norm 9.064410031719e+00 true resid norm
5.519394994263e-02 ||r(i)||/||b|| 5.017631812966e-03
3 KSP preconditioned resid norm 2.162169261727e+00 true resid norm
4.136759289559e-02 ||r(i)||/||b|| 3.760690263235e-03
4 KSP preconditioned resid norm 4.487680249963e-01 true resid norm
9.486438220687e-03 ||r(i)||/||b|| 8.624034746079e-04
5 KSP preconditioned resid norm 1.100928562187e-01 true resid norm
1.199868960902e-02 ||r(i)||/||b|| 1.090789964456e-03
6 KSP preconditioned resid norm 2.826378652048e-02 true resid norm
1.176242279335e-02 ||r(i)||/||b|| 1.069311163032e-03
7 KSP preconditioned resid norm 5.075277611809e-03 true resid norm
1.191372816799e-02 ||r(i)||/||b|| 1.083066197090e-03
8 KSP preconditioned resid norm 1.288581278231e-03 true resid norm
1.191807770501e-02 ||r(i)||/||b|| 1.083461609546e-03
9 KSP preconditioned resid norm 3.446344125585e-04 true resid norm
1.190778433764e-02 ||r(i)||/||b|| 1.082525848877e-03
10 KSP preconditioned resid norm 7.282719316035e-05 true resid norm
1.191201986724e-02 ||r(i)||/||b|| 1.082910897022e-03
11 KSP preconditioned resid norm 1.714618056930e-05 true resid norm
1.191095763273e-02 ||r(i)||/||b|| 1.082814330249e-03
12 KSP preconditioned resid norm 4.317573472482e-06 true resid norm
1.191112698437e-02 ||r(i)||/||b|| 1.082829725851e-03
13 KSP preconditioned resid norm 9.546213335221e-07 true resid norm
1.191112173116e-02 ||r(i)||/||b|| 1.082829248287e-03
14 KSP preconditioned resid norm 2.155641210237e-07 true resid norm
1.191111695140e-02 ||r(i)||/||b|| 1.082828813763e-03
15 KSP preconditioned resid norm 5.936176282735e-08 true resid norm
1.191111896030e-02 ||r(i)||/||b|| 1.082828996391e-03
16 KSP preconditioned resid norm 1.362639176327e-08 true resid norm
1.191111833902e-02 ||r(i)||/||b|| 1.082828939911e-03
17 KSP preconditioned resid norm 3.042659891281e-09 true resid norm
1.191111844668e-02 ||r(i)||/||b|| 1.082828949699e-03
18 KSP preconditioned resid norm 8.958920024015e-10 true resid norm
1.191111844183e-02 ||r(i)||/||b|| 1.082828949257e-03
19 KSP preconditioned resid norm 2.059234060666e-10 true resid norm
1.191111843406e-02 ||r(i)||/||b|| 1.082828948551e-03
I will do more tests with this but if someone has some clue on how to
correctly run the ex11 it would be of great help.
Thanks again,
Gilles
On 12/20/2014 02:29 AM, Matthew Knepley wrote:
>
> On Dec 19, 2014 6:14 AM, "Gilles Steiner" <gilles.steiner at epfl.ch
> <mailto:gilles.steiner at epfl.ch>> wrote:
> >
> > Hello Petsc Users,
> >
> > I have an issue trying to use FiledSplitPC in parallel.
> >
> > My goal : I want to get a linear system from petsc binary files and
> solve this in parallel with the FieldSplitPC.
> >
> > The problem I want to solve is an FE approximation of the Stokes
> equations.
> >
> > Skipping the details, my code looks like :
> >
> > // Reading the four blocs UU, UP, PU and PP
> > for(int i=0; i < 4; ++i)
> > {
> > string name = matrix + to_string(i) + ".petscbin";
> > PetscViewer PETSC_matreader;
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD, name.c_str(),
> FILE_MODE_READ, &PETSC_matreader);
> > MatCreate(PETSC_COMM_WORLD,&PETSC_subA[i]);
> > MatLoad(PETSC_subA[i],PETSC_matreader);
> > PetscViewerDestroy(&PETSC_matreader);
> > }
> >
> > // Reading the RHS vector and duplicating it to create the solution
> vector
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(), FILE_MODE_READ,
> &PETSC_vecreader);
> > VecCreate(PETSC_COMM_WORLD,&PETSC_rhs);
> > VecLoad(PETSC_rhs,PETSC_vecreader);
> > PetscViewerDestroy(&PETSC_vecreader);
> > VecDuplicate(PETSC_rhs,&PETSC_sol);
> >
> > // Create global matrixwith MatCreateNest
> > MatCreateNest(PETSC_COMM_WORLD, 2, NULL, 2, NULL, PETSC_subA, &PETSC_A);
> > MatNestGetISs(PETSC_A, PETSC_isg, NULL);
> >
> > // Setting up the ksp and precond
> > KSPCreate(PETSC_COMM_WORLD,&PETSC_ksp);
> > KSPSetOperators(PETSC_ksp,PETSC_A,PETSC_A);
> > KSPSetFromOptions(PETSC_ksp);
> >
> > KSPGetPC(PETSC_ksp, &PETSC_pc);
> > PCSetType(PETSC_pc, PCFIELDSPLIT);
> > PCFieldSplitSetIS(PETSC_pc, "0", PETSC_isg[0]);
> > PCFieldSplitSetIS(PETSC_pc, "1", PETSC_isg[1]);
> > PCSetFromOptions(PETSC_pc);
> >
> > // Solving the system and writing back the solution in rhs file
> > KSPSolve(PETSC_ksp,PETSC_rhs,PETSC_sol);
> >
> > PetscViewer PETSC_vecwriter;
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(),
> FILE_MODE_WRITE, &PETSC_vecwriter);
> > VecView(PETSC_sol,PETSC_vecwriter);
> > PetscViewerDestroy(&PETSC_vecwriter);
> >
> > When I run it with 1 proc, everything works fine and I get the
> correct solution :
> >
>
> When you get this behavior with true residuals, it very often arises
> from failure to account for a pressure null space. Do you have one?
>
> Matt
>
> 0 KSP preconditioned resid norm 1.271697253018e+03 true resid norm
> 2.400000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
> > 1 KSP preconditioned resid norm 5.009545069728e+01 true resid norm
> 9.166803391041e-02 ||r(i)||/||b|| 3.819501412934e-03
> > 2 KSP preconditioned resid norm 6.460631387766e+00 true resid norm
> 4.995542253831e-02 ||r(i)||/||b|| 2.081475939096e-03
> > 3 KSP preconditioned resid norm 1.155895209298e+00 true resid norm
> 1.515734830704e-02 ||r(i)||/||b|| 6.315561794600e-04
> > 4 KSP preconditioned resid norm 7.407384739634e-02 true resid norm
> 9.992802256200e-03 ||r(i)||/||b|| 4.163667606750e-04
> > 5 KSP preconditioned resid norm 1.574456882990e-02 true resid norm
> 9.994876664681e-03 ||r(i)||/||b|| 4.164531943617e-04
> > 6 KSP preconditioned resid norm 2.383022349902e-03 true resid norm
> 9.990760645581e-03 ||r(i)||/||b|| 4.162816935659e-04
> > 7 KSP preconditioned resid norm 6.175379834254e-04 true resid norm
> 9.990821066459e-03 ||r(i)||/||b|| 4.162842111025e-04
> > 8 KSP preconditioned resid norm 6.867982689960e-05 true resid norm
> 9.990532094790e-03 ||r(i)||/||b|| 4.162721706163e-04
> > 9 KSP preconditioned resid norm 1.041091257246e-05 true resid norm
> 9.990558069113e-03 ||r(i)||/||b|| 4.162732528797e-04
> > 10 KSP preconditioned resid norm 1.447793722489e-06 true resid norm
> 9.990557786778e-03 ||r(i)||/||b|| 4.162732411158e-04
> > 11 KSP preconditioned resid norm 2.139317335854e-07 true resid norm
> 9.990557262754e-03 ||r(i)||/||b|| 4.162732192814e-04
> > 12 KSP preconditioned resid norm 4.383129810322e-08 true resid norm
> 9.990557306920e-03 ||r(i)||/||b|| 4.162732211217e-04
> > 13 KSP preconditioned resid norm 3.351461304399e-09 true resid norm
> 9.990557311707e-03 ||r(i)||/||b|| 4.162732213211e-04
> > 14 KSP preconditioned resid norm 5.169032607321e-10 true resid norm
> 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04
> >
> > [14:49:10::INFO ] System Solved. Final tolerance reached is
> 5.16903e-10 in 14 iterations.
> >
> > But if I do it with 2 procs, the resolution seems fine but the
> solution is wrong :
> > 0 KSP preconditioned resid norm 1.247694088756e+03 true resid norm
> 2.400000000000e+01 ||r(i)||/||b|| 1.000000000000e+00
> > 1 KSP preconditioned resid norm 4.481954484303e+01 true resid norm
> 5.277507840772e-01 ||r(i)||/||b|| 2.198961600321e-02
> > 2 KSP preconditioned resid norm 1.110647693456e+01 true resid norm
> 4.005558168981e-02 ||r(i)||/||b|| 1.668982570409e-03
> > 3 KSP preconditioned resid norm 1.220368027409e+00 true resid norm
> 1.877650834971e-02 ||r(i)||/||b|| 7.823545145714e-04
> > 4 KSP preconditioned resid norm 2.834261749922e-01 true resid norm
> 1.613967205264e-02 ||r(i)||/||b|| 6.724863355265e-04
> > 5 KSP preconditioned resid norm 4.215090288154e-02 true resid norm
> 1.562561614611e-02 ||r(i)||/||b|| 6.510673394212e-04
> > 6 KSP preconditioned resid norm 1.209476134754e-02 true resid norm
> 1.563808960492e-02 ||r(i)||/||b|| 6.515870668718e-04
> > 7 KSP preconditioned resid norm 2.038835108629e-03 true resid norm
> 1.564163643064e-02 ||r(i)||/||b|| 6.517348512765e-04
> > 8 KSP preconditioned resid norm 1.928844666836e-04 true resid norm
> 1.564072761376e-02 ||r(i)||/||b|| 6.516969839065e-04
> > 9 KSP preconditioned resid norm 3.138911950605e-05 true resid norm
> 1.564047323377e-02 ||r(i)||/||b|| 6.516863847403e-04
> > 10 KSP preconditioned resid norm 4.950062975470e-06 true resid norm
> 1.564048216528e-02 ||r(i)||/||b|| 6.516867568865e-04
> > 11 KSP preconditioned resid norm 7.677242244159e-07 true resid norm
> 1.564049253364e-02 ||r(i)||/||b|| 6.516871889019e-04
> > 12 KSP preconditioned resid norm 1.870521888617e-07 true resid norm
> 1.564049269566e-02 ||r(i)||/||b|| 6.516871956526e-04
> > 13 KSP preconditioned resid norm 3.077235724319e-08 true resid norm
> 1.564049264800e-02 ||r(i)||/||b|| 6.516871936666e-04
> > 14 KSP preconditioned resid norm 6.584409191524e-09 true resid norm
> 1.564049264183e-02 ||r(i)||/||b|| 6.516871934095e-04
> > 15 KSP preconditioned resid norm 1.091619359913e-09 true resid norm
> 1.564049263170e-02 ||r(i)||/||b|| 6.516871929874e-04
> >
> > [15:10:58::INFO ] System Solved. Final tolerance reached is
> 1.09162e-09 in 15 iterations.
> >
> > Any idea of what is wrong with this ? Is it the code or the base
> concept ?
> >
> > Thank you.
> > Gilles
> >
>
--
-------------------------------------------------------------------------
| STEINER Gilles
=========================================================================
| PostDoc @ EPFL
| Engineer @ Ycoor Systems SA
-------------------------------------------------------------------------
| | EPFL | Ycoor Systems SA
| Unit | SB / MATHICSE / GR-PI | -
| Office | MA C2 643 | -
| Tel | +41 21 693 42 46 | +41 43 535 09 30
| @ | gilles.steiner at epfl.ch | steiner at ycoorsystems.com
| Www | http://mathicse.epfl.ch/asn/ | http://www.ycoorsystems.com/
| Address | EPFL SB MATHICSE GR-PI | Wallisellenstrasse 333
| | Station 8 | CH-8050 Zürich
| | CH-1015 Lausanne |
-------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150108/bd06e57c/attachment.html>
More information about the petsc-users
mailing list