I did eventually make my test case work using the split preconditioning, first using additive schwartz as the preconditioner for the upper left (0,0) block and then using ML with asm and gmres within each multigrid level. I am posting my runtime options in case it might be helpful for someone else out there who wants to try this. The key to getting the solver to converge for me was starting with a good initial guess (the solution diverged with a zero initial guess), especially for the pressure field, using gcr as the outer ksp and, small ksp_rtol values for both of the inner ksps.<div>
<br></div><div>Max</div><div><br></div><div><div> -stokes_pc_fieldsplit_0_fields 0,1 -stokes_pc_fieldsplit_1_fields 2 \</div><div> -stokes_pc_type fieldsplit -stokes_pc_fieldsplit_type multiplicative \</div><div> -stokes_ksp_initial_guess_nonzero \</div>
<div> -stokes_fieldsplit_0_pc_type asm \</div><div> -stokes_fieldsplit_0_ksp_type gmres \</div><div> -stokes_fieldsplit_0_ksp_initial_guess_nonzero \</div><div> -stokes_fieldsplit_0_ksp_max_it 10 \</div><div> -stokes_fieldsplit_0_ksp_rtol 1.0e-9 \</div>
<div> -stokes_fieldsplit_1_pc_type jacobi \</div><div> -stokes_fieldsplit_1_ksp_type gmres \</div><div> -stokes_fieldsplit_1_ksp_max_it 10 \</div><div> -stokes_fieldsplit_1_ksp_rtol 1.0e-9 \</div><div> -stokes_ksp_type gcr \</div>
<div> -stokes_ksp_monitor_blocks \</div><div> -stokes_ksp_monitor \</div><div> -stokes_ksp_view \</div><div> -stokes_ksp_atol 1e-2 \</div><div> -stokes_ksp_rtol 0.0 \</div></div><div><br></div><div><div class="gmail_quote">
On Wed, Feb 22, 2012 at 1:29 PM, Max Rudolph <span dir="ltr"><<a href="mailto:maxwellr@gmail.com">maxwellr@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div><div class="h5">On Tue, Feb 21, 2012 at 3:36 AM, Dave May <span dir="ltr"><<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Max,<br>
<div><br>
><br>
> The test case that I am working with is isoviscous convection, benchmark<br>
> case 1a from Blankenbach 1989.<br>
><br>
<br>
</div>Okay, I know this problem.<br>
An iso viscous problem, solved on a uniform grid using dx=dy=dz discretised<br>
via FV should be super easy to precondition.<br>
<div><br>
><br>
><br>
> I think that this is the problem. The (2,2) slot in the LHS matrix is all<br>
> zero (pressure does not appear in the continuity equation), so I think that<br>
> the preconditioner is meaningless. I am still confused as to why this choice<br>
> of preconditioner was suggested in the tutorial, and what is a better choice<br>
> of preconditioner for this block? Should I be using one of the Schur<br>
> complement methods instead of the additive or multiplicative field split?<br>
><br>
<br>
</div>No, you need to define an appropriate stokes preconditioner<br>
You should assemble this matrix<br>
B = ( K,B ; B^T, -1/eta* I )<br>
as the preconditioner for stokes.<br>
Here eta* is a measure of the local viscosity within each pressure<br>
control volume.<br>
Unless you specify to use the real diagonal<br>
<br>
Pass this into the third argument in KSPSetOperators() (i.e. the Pmat variable)<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetOperators.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetOperators.html</a><br>
<br>
Not sure how you represent A and B, but if you really want to run just<br>
additive with fieldsplit, you don't need the off diagonal blocks, so<br>
B = ( K,0 ; 0, -1/eta* I )<br>
would yield the same result. Depending on your matrix representation,<br>
this may save you some memory.<br>
<br>
PCFieldsplit will use the B(1,1) and B(2,2) to build the stokes<br>
preconditioner unless you ask for it to use the real diagonal - but<br>
for the stokes operator A, this makes no sense.<br>
<br>
This is the right thing to do (as Matt states).<br>
Try it out, and let us know how it goes.<br>
<br>
<br>
Cheers,<br>
Dave<br></blockquote><div><br></div></div></div><div>Dave and Matt,</div><div>Thanks for your help. I had some time to work on this a little more. I now have a stokes operator A that looks like this:</div><div>A=(K B; B^T 0) and a matrix from which the preconditioner is generated P=(K B; B^T -1/eta*I)</div>
</div><br><div>I verified that I can solve this system using the default ksp and pc settings in 77 iterations for the first timestep (initial guess zero) and in 31 iterations for the second timestep (nonzero initial guess).</div>
<div><br></div><div>I adopted your suggestion to use the multiplicative field split as a starting point. My reading of the PETSc manual suggests to me that the preconditioner formed should then look like:</div><div><br></div>
<div>B = (ksp(K,K) 0;-B^T*ksp(K,K)*ksp(0,-1/eta*I) ksp(0,-1/eta*I))</div><div><br></div><div>My interpretation of the output suggests that the solvers within each fieldsplit are converging nicely, but the global residual is not decreasing after the first few iterations. Given the disparity in residual sizes, I think that there might be a problem with the scaling of the pressure variable (I scaled the continuity equation by eta/dx where dx is my grid spacing). I also scaled the (1,1) block in the preconditioner by this scale factor. Thanks again for all of your help.</div>
<div><br></div><div>Max</div><div><br></div><div><br></div><div>Options used:</div><div><div> -stokes_pc_fieldsplit_0_fields 0,1 -stokes_pc_fieldsplit_1_fields 2 \</div><div> -stokes_pc_type fieldsplit -stokes_pc_fieldsplit_type multiplicative \</div>
<div> -stokes_fieldsplit_0_pc_type ml \</div><div> -stokes_fieldsplit_0_ksp_type gmres \</div><div> -stokes_fieldsplit_0_ksp_monitor_true_residual \</div><div> -stokes_fieldsplit_0_ksp_norm_type UNPRECONDITIONED \</div>
<div> -stokes_fieldsplit_0_ksp_max_it 3 \</div><div class="im"><div> -stokes_fieldsplit_0_ksp_type gmres \</div><div> -stokes_fieldsplit_0_ksp_rtol 1.0e-4 \</div><div> -stokes_fieldsplit_0_mg_levels_ksp_type gmres \</div>
<div>
-stokes_fieldsplit_0_mg_levels_pc_type bjacobi \</div></div><div> -stokes_fieldsplit_0_mg_levels_ksp_max_it 4 \</div><div> -stokes_fieldsplit_1_pc_type jacobi \</div><div> -stokes_fieldsplit_1_ksp_type preonly \</div>
<div> -stokes_fieldsplit_1_ksp_max_it 3 \</div><div> -stokes_fieldsplit_1_ksp_monitor_true_residual \</div><div> -stokes_ksp_type gcr \</div><div> -stokes_ksp_monitor_blocks \</div><div> -stokes_ksp_monitor_draw \</div>
<div> -stokes_ksp_view \</div><div> -stokes_ksp_atol 1e-6 \</div><div> -stokes_ksp_rtol 1e-6 \</div><div> -stokes_ksp_max_it 100 \</div><div> -stokes_ksp_norm_type UNPRECONDITIONED \</div><div> -stokes_ksp_monitor_true_residual \</div>
</div><div><br></div><div>Output:</div><div><br></div><div><div> 0 KSP Component U,V,P residual norm [ 0.000000000000e+00, 1.165111661413e+06, 0.000000000000e+00 ]</div><div class="im"><div> Residual norms for stokes_ solve.</div>
<div>
0 KSP unpreconditioned resid norm 1.165111661413e+06 true resid norm 1.165111661413e+06 ||r(i)||/||b|| 1.000000000000e+00</div></div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div class="im"><div> 0 KSP unpreconditioned resid norm 1.165111661413e+06 true resid norm 1.165111661413e+06 ||r(i)||/||b|| 1.000000000000e+00</div>
</div><div> 1 KSP unpreconditioned resid norm 3.173622513625e+05 true resid norm 3.173622513625e+05 ||r(i)||/||b|| 2.723878421898e-01</div><div> 2 KSP unpreconditioned resid norm 5.634119635158e+04 true resid norm 1.725996376799e+05 ||r(i)||/||b|| 1.481399967026e-01</div>
<div> 3 KSP unpreconditioned resid norm 1.218418968344e+03 true resid norm 1.559727441168e+05 ||r(i)||/||b|| 1.338693528546e-01</div><div> 1 KSP Component U,V,P residual norm [ 5.763380362961e+04, 1.154490085631e+05, 3.370358145704e-12 ]</div>
<div> 1 KSP unpreconditioned resid norm 1.290353784783e+05 true resid norm 1.290353784783e+05 ||r(i)||/||b|| 1.107493665644e-01</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.290353784783e+05 true resid norm 1.290353784783e+05 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 1.655137188235e+04 true resid norm 1.655137188235e+04 ||r(i)||/||b|| 1.282700301076e-01</div><div> 2 KSP unpreconditioned resid norm 1.195941831181e+03 true resid norm 4.554417355181e+03 ||r(i)||/||b|| 3.529588093508e-02</div>
<div> 3 KSP unpreconditioned resid norm 8.479547025398e+01 true resid norm 3.817072778396e+03 ||r(i)||/||b|| 2.958159865466e-02</div><div> 2 KSP Component U,V,P residual norm [ 2.026983725663e+03, 2.531521226429e+03, 3.419060873106e-12 ]</div>
<div> 2 KSP unpreconditioned resid norm 3.243032954498e+03 true resid norm 3.243032954498e+03 ||r(i)||/||b|| 2.783452489493e-03</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 3.243032954498e+03 true resid norm 3.243032954498e+03 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 1.170090628031e+02 true resid norm 1.170090628031e+02 ||r(i)||/||b|| 3.608013376517e-02</div><div> 2 KSP unpreconditioned resid norm 9.782830529900e+00 true resid norm 1.741722174777e+01 ||r(i)||/||b|| 5.370658267168e-03</div>
<div> 3 KSP unpreconditioned resid norm 6.886950142735e-01 true resid norm 1.636749336722e+01 ||r(i)||/||b|| 5.046971028932e-03</div><div> 3 KSP Component U,V,P residual norm [ 7.515013854917e+01, 7.515663601801e+01, 3.418919176066e-12 ]</div>
<div> 3 KSP unpreconditioned resid norm 1.062829396540e+02 true resid norm 1.062829396540e+02 ||r(i)||/||b|| 9.122124786317e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.062829396540e+02 true resid norm 1.062829396540e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.373578062042e+01 true resid norm 5.373578062042e+01 ||r(i)||/||b|| 5.055917797846e-01</div><div> 2 KSP unpreconditioned resid norm 1.199305097134e+00 true resid norm 3.492111756827e+01 ||r(i)||/||b|| 3.285674792393e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.508597255523e-02 true resid norm 3.452079362567e+01 ||r(i)||/||b|| 3.248008922038e-01</div><div> 4 KSP Component U,V,P residual norm [ 7.495897679790e+01, 7.527868410560e+01, 3.418919160091e-12 ]</div>
<div> 4 KSP unpreconditioned resid norm 1.062343093509e+02 true resid norm 1.062343093509e+02 ||r(i)||/||b|| 9.117950911420e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.062343093509e+02 true resid norm 1.062343093509e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.419252803207e+01 true resid norm 5.419252803207e+01 ||r(i)||/||b|| 5.101226558840e-01</div><div> 2 KSP unpreconditioned resid norm 1.431134174522e+00 true resid norm 3.339055236737e+01 ||r(i)||/||b|| 3.143104386088e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.760479467902e-02 true resid norm 3.304522520358e+01 ||r(i)||/||b|| 3.110598205561e-01</div><div> 5 KSP Component U,V,P residual norm [ 7.491128585963e+01, 7.523275560552e+01, 3.418919008441e-12 ]</div>
<div> 5 KSP unpreconditioned resid norm 1.061681132221e+02 true resid norm 1.061681132221e+02 ||r(i)||/||b|| 9.112269384837e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.061681132221e+02 true resid norm 1.061681132221e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.343215079492e+01 true resid norm 5.343215079492e+01 ||r(i)||/||b|| 5.032787074508e-01</div><div> 2 KSP unpreconditioned resid norm 1.288069736759e+00 true resid norm 3.308925591301e+01 ||r(i)||/||b|| 3.116684935691e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.505248953960e-02 true resid norm 3.281875055845e+01 ||r(i)||/||b|| 3.091205971589e-01</div><div> 6 KSP Component U,V,P residual norm [ 7.481188568118e+01, 7.527346267608e+01, 3.418918860626e-12 ]</div>
<div> 6 KSP unpreconditioned resid norm 1.061268694649e+02 true resid norm 1.061268694649e+02 ||r(i)||/||b|| 9.108729487455e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.061268694649e+02 true resid norm 1.061268694649e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.300383444945e+01 true resid norm 5.300383444945e+01 ||r(i)||/||b|| 4.994384053416e-01</div><div> 2 KSP unpreconditioned resid norm 1.118785004087e+00 true resid norm 3.282090953364e+01 ||r(i)||/||b|| 3.092610730828e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.758015489979e-02 true resid norm 3.259718081014e+01 ||r(i)||/||b|| 3.071529479244e-01</div><div> 7 KSP Component U,V,P residual norm [ 7.475024970669e+01, 7.530858268154e+01, 3.418918784089e-12 ]</div>
<div> 7 KSP unpreconditioned resid norm 1.061083524362e+02 true resid norm 1.061083524362e+02 ||r(i)||/||b|| 9.107140195255e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.061083524362e+02 true resid norm 1.061083524362e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.296981668051e+01 true resid norm 5.296981668051e+01 ||r(i)||/||b|| 4.992049679820e-01</div><div> 2 KSP unpreconditioned resid norm 9.379451887610e-01 true resid norm 3.378466967056e+01 ||r(i)||/||b|| 3.183978348066e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.102580142867e-02 true resid norm 3.360853440947e+01 ||r(i)||/||b|| 3.167378781957e-01</div><div> 8 KSP Component U,V,P residual norm [ 7.464535615814e+01, 7.537007679541e+01, 3.418918790515e-12 ]</div>
<div> 8 KSP unpreconditioned resid norm 1.060781677449e+02 true resid norm 1.060781677449e+02 ||r(i)||/||b|| 9.104549482946e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.060781677449e+02 true resid norm 1.060781677449e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.281972737642e+01 true resid norm 5.281972737642e+01 ||r(i)||/||b|| 4.979321240109e-01</div><div> 2 KSP unpreconditioned resid norm 9.224594814880e-01 true resid norm 3.351285171891e+01 ||r(i)||/||b|| 3.159260046751e-01</div>
<div> 3 KSP unpreconditioned resid norm 9.143100662935e-02 true resid norm 3.329269756083e+01 ||r(i)||/||b|| 3.138506091177e-01</div><div> 9 KSP Component U,V,P residual norm [ 7.451688471900e+01, 7.544516987344e+01, 3.418918860847e-12 ]</div>
<div> 9 KSP unpreconditioned resid norm 1.060412172952e+02 true resid norm 1.060412172952e+02 ||r(i)||/||b|| 9.101378074496e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.060412172952e+02 true resid norm 1.060412172952e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.275132249899e+01 true resid norm 5.275132249899e+01 ||r(i)||/||b|| 4.974605520805e-01</div><div> 2 KSP unpreconditioned resid norm 7.755381284769e-01 true resid norm 3.453933285011e+01 ||r(i)||/||b|| 3.257161105001e-01</div>
<div> 3 KSP unpreconditioned resid norm 7.298768665179e-02 true resid norm 3.435179316160e+01 ||r(i)||/||b|| 3.239475558447e-01</div><div> 10 KSP Component U,V,P residual norm [ 7.451431102619e+01, 7.544762349626e+01, 3.418918857322e-12 ]</div>
<div> 10 KSP unpreconditioned resid norm 1.060411544587e+02 true resid norm 1.060411544587e+02 ||r(i)||/||b|| 9.101372681321e-05</div><div> Residual norms for stokes_fieldsplit_0_ solve.</div><div> 0 KSP unpreconditioned resid norm 1.060411544587e+02 true resid norm 1.060411544587e+02 ||r(i)||/||b|| 1.000000000000e+00</div>
<div> 1 KSP unpreconditioned resid norm 5.276103518337e+01 true resid norm 5.276103518337e+01 ||r(i)||/||b|| 4.975524403961e-01</div><div> 2 KSP unpreconditioned resid norm 7.777079890360e-01 true resid norm 3.454373663425e+01 ||r(i)||/||b|| 3.257578325186e-01</div>
<div> 3 KSP unpreconditioned resid norm 7.356028471071e-02 true resid norm 3.435584054266e+01 ||r(i)||/||b|| 3.239859158269e-01</div><div> 11 KSP Component U,V,P residual norm [ 7.438335197779e+01, 7.553731959735e+01, 3.418918856471e-12 ]</div>
<div><br></div></div><div><br></div><div><br></div><div><br></div>
</blockquote></div><br></div>