<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hello again.<br>
<br>
Thank you all for the answers. Here is an update on this subject.<br>
<br>
First, some answers to the comments/questions received : <br>
<br>
Answer to Jed Brown : <br>
<pre wrap="">"What FE discretization? We don't recommend using files as part of your
workflow, but if you're just experimenting, you could start with
src/ksp/ksp/examples/tests/ex11.c which solves a Q1-P0 Stokes problem
>From Underworld by reading the blocks in as matrices. So start there
and let us know how it goes."
</pre>
I tried several FE discretizations (P1-P1-stabilized, P2-P1,
P1Bubble-P1) on tetrahedral meshes of a cube (Driven cavity).<br>
Thanks for the reference to ex11.c, but do you have any suggestion
of options to apply on this to have a good behaviour of the solver
? <br>
<br>
I compiled the example (src/ksp/ksp/examples/tests/ex11.c) and
downloaded the input file from
<a class="moz-txt-link-freetext" href="http://ftp.mcs.anl.gov/pub/petsc/matrices/underworld32.gz">http://ftp.mcs.anl.gov/pub/petsc/matrices/underworld32.gz</a><br>
but the convergence of the solver is a bit disappointing with the
default options (simply running with mpirun -np 1 ex11 -f
underworld32.gz) : <br>
<br>
[0] lnu = 2046 | lnp = 1024 <br>
[0] s_u = 0 | e_u = 2046 <br>
[0] s_p = 0 | e_p = 1024 <br>
[0] is_u (offset) = 0 <br>
[0] is_p (offset) = 2046 <br>
Residual norms for fc_ solve.<br>
0 KSP preconditioned resid norm 1.901623820248e+11 true resid
norm 3.163503865730e-01 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 1.232616307278e+11 true resid
norm 3.110902550317e-01 ||r(i)||/||b|| 9.833724510397e-01<br>
2 KSP preconditioned resid norm 8.842498650573e+10 true resid
norm 3.121703549116e-01 ||r(i)||/||b|| 9.867867028497e-01<br>
3 KSP preconditioned resid norm 6.376643354056e+10 true resid
norm 3.216506148074e-01 ||r(i)||/||b|| 1.016754296689e+00<br>
4 KSP preconditioned resid norm 4.507200049791e+10 true resid
norm 3.453307829735e-01 ||r(i)||/||b|| 1.091608537971e+00<br>
5 KSP preconditioned resid norm 3.165593374571e+10 true resid
norm 3.787554909641e-01 ||r(i)||/||b|| 1.197265775671e+00<br>
6 KSP preconditioned resid norm 1.886092467773e+10 true resid
norm 4.169668186513e-01 ||r(i)||/||b|| 1.318053766800e+00<br>
7 KSP preconditioned resid norm 1.022004643856e+10 true resid
norm 4.423235667291e-01 ||r(i)||/||b|| 1.398207764248e+00<br>
8 KSP preconditioned resid norm 5.318995340694e+09 true resid
norm 4.558356174770e-01 ||r(i)||/||b|| 1.440920058341e+00<br>
9 KSP preconditioned resid norm 3.160919062173e+09 true resid
norm 4.608182603598e-01 ||r(i)||/||b|| 1.456670451242e+00<br>
10 KSP preconditioned resid norm 2.172582250050e+09 true resid
norm 4.643714263977e-01 ||r(i)||/||b|| 1.467902193603e+00<br>
...<br>
184 KSP preconditioned resid norm 1.890479367808e+06 true resid
norm 3.557809738610e-01 ||r(i)||/||b|| 1.124642134044e+00<br>
-- velocity vector values --<br>
Min(u) = -0.000000 [loc=1994]<br>
Max(u) = 0.000000 [loc=1310]<br>
Norm(u) = 0.000000 <br>
Sum(u) = 0.000000 <br>
-- pressure vector values --<br>
Min(p) = -4.877385 [loc=1015]<br>
Max(p) = 4.616608 [loc=4]<br>
Norm(p) = 88.261241 <br>
Sum(p) = 0.000000 <br>
-- Full vector values --<br>
Min(u,p) = -4.877385 [loc=3061]<br>
Max(u,p) = 4.616608 [loc=2050]<br>
Norm(u,p) = 88.261241 <br>
Sum(u,p) = 0.000000 <br>
<br>
The only modification I have done to the code is to add the line :
KSPMonitorSet(ksp_A, KSPMonitorTrueResidualNorm, PETSC_NULL,
PETSC_NULL);<br>
to log the convergence.<br>
<br>
<br>
<br>
<br>
Answer to Barry Smith and Matthew Knepley : <br>
<br>
"In both cases the preconditioned residual is decreasing nicely
but the unpreconditioned residual is not decreasing, so something
is wrong even in the sequential case!
<blockquote type="cite" style="color: #000000;">
<pre wrap="">14 KSP preconditioned resid norm 5.169032607321e-10 true resid norm 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04
</pre>
</blockquote>
<pre wrap=""> So you need to go back to the sequential case and see what is going on. Don't even touch the parallel case until the true residual is converging properly for the sequential. First try running with -ksp_pc_side right and watch the residuals. Next run with direct solvers everywhere you can and see what happens."
</pre>
I guess I figured out the issue here. I tried to speedup
iterations by adding the options -fieldsplit_0_ksp_max_it and
-fieldsplit_1_ksp_max_it as suggested in
src/snes/examples/tutorials/ex70.c (they are acting on rtol
instead of max_it). But doing this seems to cause the wrong
convergence behaviour.<br>
<br>
example with max_it=100 : <br>
0 KSP preconditioned resid norm 6.492190058577e+02 true resid
norm 1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 1.969844390083e-02 true resid
norm 1.509247865473e-05 ||r(i)||/||b|| 1.372043514066e-06<br>
2 KSP preconditioned resid norm 4.926131518900e-07 true resid
norm 7.062431971029e-09 ||r(i)||/||b|| 6.420392700936e-10<br>
3 KSP preconditioned resid norm 8.183544025740e-12 true resid
norm 6.418682323277e-09 ||r(i)||/||b|| 5.835165748434e-10<br>
<br>
same with max_it=5 :<br>
0 KSP preconditioned resid norm 6.191842548447e+02 true resid
norm 1.100000000000e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 5.474611327659e+01 true resid
norm 3.793016384570e-01 ||r(i)||/||b|| 3.448196713245e-02<br>
2 KSP preconditioned resid norm 9.064410031719e+00 true resid
norm 5.519394994263e-02 ||r(i)||/||b|| 5.017631812966e-03<br>
3 KSP preconditioned resid norm 2.162169261727e+00 true resid
norm 4.136759289559e-02 ||r(i)||/||b|| 3.760690263235e-03<br>
4 KSP preconditioned resid norm 4.487680249963e-01 true resid
norm 9.486438220687e-03 ||r(i)||/||b|| 8.624034746079e-04<br>
5 KSP preconditioned resid norm 1.100928562187e-01 true resid
norm 1.199868960902e-02 ||r(i)||/||b|| 1.090789964456e-03<br>
6 KSP preconditioned resid norm 2.826378652048e-02 true resid
norm 1.176242279335e-02 ||r(i)||/||b|| 1.069311163032e-03<br>
7 KSP preconditioned resid norm 5.075277611809e-03 true resid
norm 1.191372816799e-02 ||r(i)||/||b|| 1.083066197090e-03<br>
8 KSP preconditioned resid norm 1.288581278231e-03 true resid
norm 1.191807770501e-02 ||r(i)||/||b|| 1.083461609546e-03<br>
9 KSP preconditioned resid norm 3.446344125585e-04 true resid
norm 1.190778433764e-02 ||r(i)||/||b|| 1.082525848877e-03<br>
10 KSP preconditioned resid norm 7.282719316035e-05 true resid
norm 1.191201986724e-02 ||r(i)||/||b|| 1.082910897022e-03<br>
11 KSP preconditioned resid norm 1.714618056930e-05 true resid
norm 1.191095763273e-02 ||r(i)||/||b|| 1.082814330249e-03<br>
12 KSP preconditioned resid norm 4.317573472482e-06 true resid
norm 1.191112698437e-02 ||r(i)||/||b|| 1.082829725851e-03<br>
13 KSP preconditioned resid norm 9.546213335221e-07 true resid
norm 1.191112173116e-02 ||r(i)||/||b|| 1.082829248287e-03<br>
14 KSP preconditioned resid norm 2.155641210237e-07 true resid
norm 1.191111695140e-02 ||r(i)||/||b|| 1.082828813763e-03<br>
15 KSP preconditioned resid norm 5.936176282735e-08 true resid
norm 1.191111896030e-02 ||r(i)||/||b|| 1.082828996391e-03<br>
16 KSP preconditioned resid norm 1.362639176327e-08 true resid
norm 1.191111833902e-02 ||r(i)||/||b|| 1.082828939911e-03<br>
17 KSP preconditioned resid norm 3.042659891281e-09 true resid
norm 1.191111844668e-02 ||r(i)||/||b|| 1.082828949699e-03<br>
18 KSP preconditioned resid norm 8.958920024015e-10 true resid
norm 1.191111844183e-02 ||r(i)||/||b|| 1.082828949257e-03<br>
19 KSP preconditioned resid norm 2.059234060666e-10 true resid
norm 1.191111843406e-02 ||r(i)||/||b|| 1.082828948551e-03<br>
<br>
<br>
I will do more tests with this but if someone has some clue on how
to correctly run the ex11 it would be of great help. <br>
<br>
Thanks again,<br>
Gilles<br>
<br>
<br>
On 12/20/2014 02:29 AM, Matthew Knepley wrote:<br>
</div>
<blockquote
cite="mid:CAMYG4Gk11Oy5bGpgwsB2-ORdJmwVxJcm04MYunCHefBThh9kdQ@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p dir="ltr">On Dec 19, 2014 6:14 AM, "Gilles Steiner" <<a
moz-do-not-send="true" href="mailto:gilles.steiner@epfl.ch">gilles.steiner@epfl.ch</a>>
wrote:<br>
><br>
> Hello Petsc Users,<br>
><br>
> I have an issue trying to use FiledSplitPC in parallel.<br>
><br>
> My goal : I want to get a linear system from petsc binary
files and solve this in parallel with the FieldSplitPC.<br>
><br>
> The problem I want to solve is an FE approximation of the
Stokes equations.<br>
><br>
> Skipping the details, my code looks like :<br>
><br>
> // Reading the four blocs UU, UP, PU and PP<br>
> for(int i=0; i < 4; ++i)<br>
> {<br>
> string name = matrix + to_string(i) + ".petscbin";<br>
> PetscViewer PETSC_matreader;<br>
> PetscViewerBinaryOpen(PETSC_COMM_WORLD, name.c_str(),
FILE_MODE_READ, &PETSC_matreader);<br>
> MatCreate(PETSC_COMM_WORLD,&PETSC_subA[i]);<br>
> MatLoad(PETSC_subA[i],PETSC_matreader);<br>
> PetscViewerDestroy(&PETSC_matreader);<br>
> }<br>
><br>
> // Reading the RHS vector and duplicating it to create the
solution vector<br>
> PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(),
FILE_MODE_READ, &PETSC_vecreader);<br>
> VecCreate(PETSC_COMM_WORLD,&PETSC_rhs);<br>
> VecLoad(PETSC_rhs,PETSC_vecreader);<br>
> PetscViewerDestroy(&PETSC_vecreader);<br>
> VecDuplicate(PETSC_rhs,&PETSC_sol);<br>
><br>
> // Create global matrixwith MatCreateNest<br>
> MatCreateNest(PETSC_COMM_WORLD, 2, NULL, 2, NULL,
PETSC_subA, &PETSC_A);<br>
> MatNestGetISs(PETSC_A, PETSC_isg, NULL);<br>
><br>
> // Setting up the ksp and precond<br>
> KSPCreate(PETSC_COMM_WORLD,&PETSC_ksp);<br>
> KSPSetOperators(PETSC_ksp,PETSC_A,PETSC_A);<br>
> KSPSetFromOptions(PETSC_ksp);<br>
><br>
> KSPGetPC(PETSC_ksp, &PETSC_pc);<br>
> PCSetType(PETSC_pc, PCFIELDSPLIT);<br>
> PCFieldSplitSetIS(PETSC_pc, "0", PETSC_isg[0]);<br>
> PCFieldSplitSetIS(PETSC_pc, "1", PETSC_isg[1]);<br>
> PCSetFromOptions(PETSC_pc);<br>
><br>
> // Solving the system and writing back the solution in rhs
file<br>
> KSPSolve(PETSC_ksp,PETSC_rhs,PETSC_sol);<br>
><br>
> PetscViewer PETSC_vecwriter;<br>
> PetscViewerBinaryOpen(PETSC_COMM_WORLD, rhs.c_str(),
FILE_MODE_WRITE, &PETSC_vecwriter);<br>
> VecView(PETSC_sol,PETSC_vecwriter);<br>
> PetscViewerDestroy(&PETSC_vecwriter);<br>
><br>
> When I run it with 1 proc, everything works fine and I get
the correct solution :<br>
></p>
<p dir="ltr">When you get this behavior with true residuals, it
very often arises from failure to account for a pressure null
space. Do you have one?</p>
<p dir="ltr"> Matt</p>
<p dir="ltr"> 0 KSP preconditioned resid norm 1.271697253018e+03
true resid norm 2.400000000000e+01 ||r(i)||/||b||
1.000000000000e+00<br>
> 1 KSP preconditioned resid norm 5.009545069728e+01 true
resid norm 9.166803391041e-02 ||r(i)||/||b|| 3.819501412934e-03<br>
> 2 KSP preconditioned resid norm 6.460631387766e+00 true
resid norm 4.995542253831e-02 ||r(i)||/||b|| 2.081475939096e-03<br>
> 3 KSP preconditioned resid norm 1.155895209298e+00 true
resid norm 1.515734830704e-02 ||r(i)||/||b|| 6.315561794600e-04<br>
> 4 KSP preconditioned resid norm 7.407384739634e-02 true
resid norm 9.992802256200e-03 ||r(i)||/||b|| 4.163667606750e-04<br>
> 5 KSP preconditioned resid norm 1.574456882990e-02 true
resid norm 9.994876664681e-03 ||r(i)||/||b|| 4.164531943617e-04<br>
> 6 KSP preconditioned resid norm 2.383022349902e-03 true
resid norm 9.990760645581e-03 ||r(i)||/||b|| 4.162816935659e-04<br>
> 7 KSP preconditioned resid norm 6.175379834254e-04 true
resid norm 9.990821066459e-03 ||r(i)||/||b|| 4.162842111025e-04<br>
> 8 KSP preconditioned resid norm 6.867982689960e-05 true
resid norm 9.990532094790e-03 ||r(i)||/||b|| 4.162721706163e-04<br>
> 9 KSP preconditioned resid norm 1.041091257246e-05 true
resid norm 9.990558069113e-03 ||r(i)||/||b|| 4.162732528797e-04<br>
> 10 KSP preconditioned resid norm 1.447793722489e-06 true
resid norm 9.990557786778e-03 ||r(i)||/||b|| 4.162732411158e-04<br>
> 11 KSP preconditioned resid norm 2.139317335854e-07 true
resid norm 9.990557262754e-03 ||r(i)||/||b|| 4.162732192814e-04<br>
> 12 KSP preconditioned resid norm 4.383129810322e-08 true
resid norm 9.990557306920e-03 ||r(i)||/||b|| 4.162732211217e-04<br>
> 13 KSP preconditioned resid norm 3.351461304399e-09 true
resid norm 9.990557311707e-03 ||r(i)||/||b|| 4.162732213211e-04<br>
> 14 KSP preconditioned resid norm 5.169032607321e-10 true
resid norm 9.990557312817e-03 ||r(i)||/||b|| 4.162732213674e-04<br>
><br>
> [14:49:10::INFO ] System Solved. Final tolerance reached
is 5.16903e-10 in 14 iterations.<br>
><br>
> But if I do it with 2 procs, the resolution seems fine but
the solution is wrong :<br>
> 0 KSP preconditioned resid norm 1.247694088756e+03 true
resid norm 2.400000000000e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
> 1 KSP preconditioned resid norm 4.481954484303e+01 true
resid norm 5.277507840772e-01 ||r(i)||/||b|| 2.198961600321e-02<br>
> 2 KSP preconditioned resid norm 1.110647693456e+01 true
resid norm 4.005558168981e-02 ||r(i)||/||b|| 1.668982570409e-03<br>
> 3 KSP preconditioned resid norm 1.220368027409e+00 true
resid norm 1.877650834971e-02 ||r(i)||/||b|| 7.823545145714e-04<br>
> 4 KSP preconditioned resid norm 2.834261749922e-01 true
resid norm 1.613967205264e-02 ||r(i)||/||b|| 6.724863355265e-04<br>
> 5 KSP preconditioned resid norm 4.215090288154e-02 true
resid norm 1.562561614611e-02 ||r(i)||/||b|| 6.510673394212e-04<br>
> 6 KSP preconditioned resid norm 1.209476134754e-02 true
resid norm 1.563808960492e-02 ||r(i)||/||b|| 6.515870668718e-04<br>
> 7 KSP preconditioned resid norm 2.038835108629e-03 true
resid norm 1.564163643064e-02 ||r(i)||/||b|| 6.517348512765e-04<br>
> 8 KSP preconditioned resid norm 1.928844666836e-04 true
resid norm 1.564072761376e-02 ||r(i)||/||b|| 6.516969839065e-04<br>
> 9 KSP preconditioned resid norm 3.138911950605e-05 true
resid norm 1.564047323377e-02 ||r(i)||/||b|| 6.516863847403e-04<br>
> 10 KSP preconditioned resid norm 4.950062975470e-06 true
resid norm 1.564048216528e-02 ||r(i)||/||b|| 6.516867568865e-04<br>
> 11 KSP preconditioned resid norm 7.677242244159e-07 true
resid norm 1.564049253364e-02 ||r(i)||/||b|| 6.516871889019e-04<br>
> 12 KSP preconditioned resid norm 1.870521888617e-07 true
resid norm 1.564049269566e-02 ||r(i)||/||b|| 6.516871956526e-04<br>
> 13 KSP preconditioned resid norm 3.077235724319e-08 true
resid norm 1.564049264800e-02 ||r(i)||/||b|| 6.516871936666e-04<br>
> 14 KSP preconditioned resid norm 6.584409191524e-09 true
resid norm 1.564049264183e-02 ||r(i)||/||b|| 6.516871934095e-04<br>
> 15 KSP preconditioned resid norm 1.091619359913e-09 true
resid norm 1.564049263170e-02 ||r(i)||/||b|| 6.516871929874e-04<br>
><br>
> [15:10:58::INFO ] System Solved. Final tolerance reached
is 1.09162e-09 in 15 iterations.<br>
><br>
> Any idea of what is wrong with this ? Is it the code or the
base concept ?<br>
><br>
> Thank you.<br>
> Gilles<br>
><br>
</p>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
-------------------------------------------------------------------------
| STEINER Gilles
=========================================================================
| PostDoc @ EPFL
| Engineer @ Ycoor Systems SA
-------------------------------------------------------------------------
| | EPFL | Ycoor Systems SA
| Unit | SB / MATHICSE / GR-PI | -
| Office | MA C2 643 | -
| Tel | +41 21 693 42 46 | +41 43 535 09 30
| @ | <a class="moz-txt-link-abbreviated" href="mailto:gilles.steiner@epfl.ch">gilles.steiner@epfl.ch</a> | <a class="moz-txt-link-abbreviated" href="mailto:steiner@ycoorsystems.com">steiner@ycoorsystems.com</a>
| Www | <a class="moz-txt-link-freetext" href="http://mathicse.epfl.ch/asn/">http://mathicse.epfl.ch/asn/</a> | <a class="moz-txt-link-freetext" href="http://www.ycoorsystems.com/">http://www.ycoorsystems.com/</a>
| Address | EPFL SB MATHICSE GR-PI | Wallisellenstrasse 333
| | Station 8 | CH-8050 Zürich
| | CH-1015 Lausanne |
-------------------------------------------------------------------------
</pre>
</body>
</html>