<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jan 26, 2017 at 9:36 PM, David Knezevic <span dir="ltr"><<a href="mailto:david.knezevic@akselos.com" target="_blank">david.knezevic@akselos.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jan 26, 2017 at 6:50 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
David, with Schur complement preconditioning the nesting can be rather complicated and hard to track. We need to know exactly what monitors you have turned on and the output from -ksp_view in order to understand why you are seeing this (seemingly) strange effect. Send all command line arguments and if you are running a PETSc example.<br></blockquote><div><br></div><div>Hi Barry,</div><div><br></div><div>Thanks for your comments. The command line arguments I'm using are:</div><div><br></div><div>-ksp_view -ksp_monitor -ksp_type cg -pc_type fieldsplit -pc_fieldsplit_type schur -fieldsplit_block_2_ksp_<wbr>monitor -pc_fieldsplit_type schur -fieldsplit_block_1_pc_type cholesky -fieldsplit_block_1_ksp_type preonly -fieldsplit_block_2_pc_type cholesky -fieldsplit_block_2_ksp_type minres </div></div></div></div></blockquote><div><br></div><div>preonly messes with monitors because "nothing is supposed to happen" so ksp monitors don't fire. I would start with -ksp_type richardson -ksp_max_it 1</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Note that I cleaned up my code a bit (I was doing some things via code previously, but now I'm doing it all via command line arguments) and now I can't replicate the strange effect that I reported in my previous email, so I guess that must have been a bug at my end, sorry about that! (I'm using my own example here, not a PETSc example.)</div><div><br></div><div>I've pasted the output that I get below, and it doesn't have any repeated inner iterations anymore, so it looks good to me now.</div><div><br></div><div>Thanks,</div><div>David</div><div><br></div><div><div> Residual norms for fieldsplit_block_2_ solve.</div><div> 0 KSP Residual norm 4.742303891404e+01 </div><div> 1 KSP Residual norm 2.909253505627e-01 </div><div> 2 KSP Residual norm 9.891933795043e-02 </div><div> 3 KSP Residual norm 7.147789520726e-02 </div><div> 4 KSP Residual norm 1.668752975827e-02 </div><div> 5 KSP Residual norm 5.020016959860e-03 </div><div> 6 KSP Residual norm 2.939648855177e-03 </div><div> 7 KSP Residual norm 2.847937179284e-03 </div><div> 8 KSP Residual norm 2.840502402597e-03 </div><div> 9 KSP Residual norm 2.831875522390e-03 </div><div> 10 KSP Residual norm 2.688309294147e-03 </div><div> 11 KSP Residual norm 1.364866662320e-03 </div><div> 12 KSP Residual norm 1.351460340317e-03 </div><div> 13 KSP Residual norm 9.154713357796e-06 </div><div> 0 KSP Residual norm 2.254632359666e+02 </div><div> Residual norms for fieldsplit_block_2_ solve.</div><div> 0 KSP Residual norm 1.562796164027e-05 </div><div> 1 KSP Residual norm 1.562568942204e-05 </div><div> 2 KSP Residual norm 1.559644955712e-05 </div><div> 3 KSP Residual norm 4.739731314562e-06 </div><div> 4 KSP Residual norm 4.099203626623e-06 </div><div> 5 KSP Residual norm 4.077340563505e-06 </div><div> 6 KSP Residual norm 4.069934161136e-06 </div><div> 7 KSP Residual norm 4.069081720832e-06 </div><div> 8 KSP Residual norm 4.056610946411e-06 </div><div> 9 KSP Residual norm 4.049977739107e-06 </div><div> 10 KSP Residual norm 6.930727940912e-07 </div><div> 11 KSP Residual norm 2.525753277129e-07 </div><div> 12 KSP Residual norm 2.490106038182e-07 </div><div> 13 KSP Residual norm 2.117790443270e-07 </div><div> 14 KSP Residual norm 4.278506164384e-08 </div><div> 15 KSP Residual norm 4.258945198001e-08 </div><div> 16 KSP Residual norm 4.229334447981e-08 </div><div> 17 KSP Residual norm 4.411718412575e-09 </div><div> 18 KSP Residual norm 2.538786960344e-10 </div><div> 19 KSP Residual norm 1.938707294127e-11 </div><div> 1 KSP Residual norm 8.266789276240e-01 </div><div> Residual norms for fieldsplit_block_2_ solve.</div><div> 0 KSP Residual norm 9.262098141749e-08 </div><div> 1 KSP Residual norm 5.654510775654e-10 </div><div> 2 KSP Residual norm 1.850940554946e-10 </div><div> 3 KSP Residual norm 1.215872907413e-10 </div><div> 4 KSP Residual norm 3.606362942651e-11 </div><div> 5 KSP Residual norm 9.320062914875e-12 </div><div> 6 KSP Residual norm 5.745183182943e-12 </div><div> 7 KSP Residual norm 5.567033341218e-12 </div><div> 8 KSP Residual norm 5.549943035966e-12 </div><div> 9 KSP Residual norm 5.535981844425e-12 </div><div> 10 KSP Residual norm 5.145409580427e-12 </div><div> 11 KSP Residual norm 2.666485042347e-12 </div><div> 12 KSP Residual norm 2.642698026095e-12 </div><div> 13 KSP Residual norm 2.351888597601e-14 </div><div> 2 KSP Residual norm 4.408818087377e-07 </div><div> Residual norms for fieldsplit_block_2_ solve.</div><div> 0 KSP Residual norm 6.995890260994e-14 </div><div> 1 KSP Residual norm 6.992908711883e-14 </div><div> 2 KSP Residual norm 6.975805821489e-14 </div><div> 3 KSP Residual norm 3.895698269070e-14 </div><div> 4 KSP Residual norm 1.087381803170e-14 </div><div> 5 KSP Residual norm 9.649779038703e-15 </div><div> 6 KSP Residual norm 9.639503278208e-15 </div><div> 7 KSP Residual norm 9.632063248075e-15 </div><div> 8 KSP Residual norm 9.628902454482e-15 </div><div> 9 KSP Residual norm 9.422064141948e-15 </div><div> 10 KSP Residual norm 7.978243292438e-15 </div><div> 11 KSP Residual norm 1.840364998277e-15 </div><div> 12 KSP Residual norm 1.313440209393e-15 </div><div> 13 KSP Residual norm 1.237317810391e-15 </div><div> 14 KSP Residual norm 3.450624715175e-16 </div><div> 15 KSP Residual norm 2.677943917771e-16 </div><div> 16 KSP Residual norm 2.477309550239e-16 </div><div> 17 KSP Residual norm 2.379077634512e-16 </div><div> 18 KSP Residual norm 2.195084093320e-17 </div><div> 19 KSP Residual norm 3.440267981045e-18 </div><div> 20 KSP Residual norm 1.678278636525e-19 </div><div> 3 KSP Residual norm 2.982422976816e-09 </div><div> Residual norms for fieldsplit_block_2_ solve.</div><div> 0 KSP Residual norm 7.797636927781e-16 </div><div> 1 KSP Residual norm 4.773690049123e-18 </div><div> 2 KSP Residual norm 1.562856784196e-18 </div><div> 3 KSP Residual norm 1.029604947226e-18 </div><div> 4 KSP Residual norm 3.077276285355e-19 </div><div> 5 KSP Residual norm 7.977399411645e-20 </div><div> 6 KSP Residual norm 4.916849936508e-20 </div><div> 7 KSP Residual norm 4.764440575250e-20 </div><div> 8 KSP Residual norm 4.678272883449e-20 </div><div> 9 KSP Residual norm 4.672106002328e-20 </div><div> 10 KSP Residual norm 4.395115290861e-20 </div><div> 11 KSP Residual norm 2.248590969592e-20 </div><div> 12 KSP Residual norm 2.226836892279e-20 </div><div> 13 KSP Residual norm 3.483134646974e-22 </div><div> 4 KSP Residual norm 3.713946450555e-15 </div><div>KSP Object: 1 MPI processes</div><div> type: cg</div><div> maximum iterations=1000, initial guess is zero</div><div> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div>PC Object: 1 MPI processes</div><div> type: fieldsplit</div><div> FieldSplit with Schur preconditioner, factorization FULL</div><div> Preconditioner for the Schur complement formed from A11</div><div> Split info:</div><div> Split number 0 Defined by IS</div><div> Split number 1 Defined by IS</div><div> KSP solver for A00 block</div><div> KSP Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: cholesky</div><div> Cholesky: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 5., needed 1.14458</div><div> Factored matrix follows:</div><div> Mat Object: 1 MPI processes</div><div> type: seqsbaij</div><div> rows=30, cols=30</div><div> package used to perform factorization: petsc</div><div> total: nonzeros=285, allocated nonzeros=285</div><div> total number of mallocs used during MatSetValues calls =0</div><div> block size is 1</div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: seqaij</div><div> rows=30, cols=30</div><div> total: nonzeros=468, allocated nonzeros=468</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 10 nodes, limit used is 5</div><div> KSP solver for S = A11 - A10 inv(A00) A01 </div><div> KSP Object: (fieldsplit_block_2_) 1 MPI processes</div><div> type: minres</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div> PC Object: (fieldsplit_block_2_) 1 MPI processes</div><div> type: cholesky</div><div> Cholesky: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 5., needed 23.1555</div><div> Factored matrix follows:</div><div> Mat Object: 1 MPI processes</div><div> type: seqsbaij</div><div> rows=10512, cols=10512</div><div> package used to perform factorization: petsc</div><div> total: nonzeros=4435992, allocated nonzeros=4435992</div><div> total number of mallocs used during MatSetValues calls =0</div><div> block size is 1</div><div> linear system matrix followed by preconditioner matrix:</div><div> Mat Object: (fieldsplit_block_2_) 1 MPI processes</div><div> type: schurcomplement</div><div> rows=10512, cols=10512</div><div> Schur complement A11 - A10 inv(A00) A01</div><div> A11</div><div> Mat Object: (fieldsplit_block_2_) 1 MPI processes</div><div> type: seqaij</div><div> rows=10512, cols=10512</div><div> total: nonzeros=372636, allocated nonzeros=372636</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 3504 nodes, limit used is 5</div><div> A10</div><div> Mat Object: 1 MPI processes</div><div> type: seqaij</div><div> rows=10512, cols=30</div><div> total: nonzeros=1872, allocated nonzeros=1872</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 2143 nodes, limit used is 5</div><div> KSP of A00</div><div> KSP Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: cholesky</div><div> Cholesky: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 5., needed 1.14458</div><div> Factored matrix follows:</div><div> Mat Object: 1 MPI processes</div><div> type: seqsbaij</div><div> rows=30, cols=30</div><div> package used to perform factorization: petsc</div><div> total: nonzeros=285, allocated nonzeros=285</div><div> total number of mallocs used during MatSetValues calls =0</div><div> block size is 1</div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_block_1_) 1 MPI processes</div><div> type: seqaij</div><div> rows=30, cols=30</div><div> total: nonzeros=468, allocated nonzeros=468</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 10 nodes, limit used is 5</div><div> A01</div><div> Mat Object: 1 MPI processes</div><div> type: seqaij</div><div> rows=30, cols=10512</div><div> total: nonzeros=1872, allocated nonzeros=1872</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 9 nodes, limit used is 5</div><div> Mat Object: (fieldsplit_block_2_) 1 MPI processes</div><div> type: seqaij</div><div> rows=10512, cols=10512</div><div> total: nonzeros=372636, allocated nonzeros=372636</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 3504 nodes, limit used is 5</div><div> linear system matrix = precond matrix:</div><div> Mat Object: () 1 MPI processes</div><div> type: seqaij</div><div> rows=10542, cols=10542</div><div> total: nonzeros=376848, allocated nonzeros=377064</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 3514 nodes, limit used is 5</div></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="m_-1160743629684591596gmail-HOEnZb"><div class="m_-1160743629684591596gmail-h5"><br>
<br>
<br>
> On Jan 26, 2017, at 1:26 PM, David Knezevic <<a href="mailto:david.knezevic@akselos.com" target="_blank">david.knezevic@akselos.com</a>> wrote:<br>
><br>
> I'm exploring fieldsplit with Schur (this continues from some emails I sent a few weeks ago about this topic), and I had a quick question about the inner vs outer convergence.<br>
><br>
> I've pasted the output below from "-ksp_monitor -fieldsplit_FE_split_ksp_monit<wbr>or", and I'm just wondering about why the second outer iteration has two inner iteration loops, whereas all the other outer iterations have one inner iteration loop? I assume it is something to do with a convergence tolerance, but it's not clear to me which tolerance would control that.<br>
><br>
> Thanks,<br>
> David<br>
><br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
><br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 4.742303891408e+01<br>
> 1 KSP Residual norm 2.909253505630e-01<br>
> 2 KSP Residual norm 9.891933795059e-02<br>
> 3 KSP Residual norm 7.147789520745e-02<br>
> 4 KSP Residual norm 1.668752967907e-02<br>
> 5 KSP Residual norm 5.019869896662e-03<br>
> 6 KSP Residual norm 2.848579237244e-03<br>
> 7 KSP Residual norm 2.847897269641e-03<br>
> 8 KSP Residual norm 2.840502392022e-03<br>
> 9 KSP Residual norm 2.831875522381e-03<br>
> 10 KSP Residual norm 2.688309287993e-03<br>
> 11 KSP Residual norm 1.351494303229e-03<br>
> 12 KSP Residual norm 1.350874246297e-03<br>
> 13 KSP Residual norm 9.154691604943e-06<br>
> 0 KSP Residual norm 2.254632353893e+02<br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 4.742303891408e+01<br>
> 1 KSP Residual norm 2.909253505630e-01<br>
> 2 KSP Residual norm 9.891933795059e-02<br>
> 3 KSP Residual norm 7.147789520745e-02<br>
> 4 KSP Residual norm 1.668752967907e-02<br>
> 5 KSP Residual norm 5.019869896662e-03<br>
> 6 KSP Residual norm 2.848579237244e-03<br>
> 7 KSP Residual norm 2.847897269641e-03<br>
> 8 KSP Residual norm 2.840502392022e-03<br>
> 9 KSP Residual norm 2.831875522381e-03<br>
> 10 KSP Residual norm 2.688309287993e-03<br>
> 11 KSP Residual norm 1.351494303229e-03<br>
> 12 KSP Residual norm 1.350874246297e-03<br>
> 13 KSP Residual norm 9.154691604943e-06<br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 1.554697370480e-05<br>
> 1 KSP Residual norm 1.554471967929e-05<br>
> 2 KSP Residual norm 1.551293889691e-05<br>
> 3 KSP Residual norm 8.031337431574e-06<br>
> 4 KSP Residual norm 4.137185786243e-06<br>
> 5 KSP Residual norm 4.066606123330e-06<br>
> 6 KSP Residual norm 4.051107282928e-06<br>
> 7 KSP Residual norm 4.047442850256e-06<br>
> 8 KSP Residual norm 4.047129984657e-06<br>
> 9 KSP Residual norm 4.030697964677e-06<br>
> 10 KSP Residual norm 2.882383190940e-06<br>
> 11 KSP Residual norm 3.325005138484e-07<br>
> 12 KSP Residual norm 2.107354774516e-07<br>
> 13 KSP Residual norm 2.107005548204e-07<br>
> 14 KSP Residual norm 4.399320792736e-08<br>
> 15 KSP Residual norm 4.236902403786e-08<br>
> 16 KSP Residual norm 2.932877082709e-08<br>
> 17 KSP Residual norm 3.881909203171e-09<br>
> 18 KSP Residual norm 1.107791399514e-09<br>
> 19 KSP Residual norm 2.645048006100e-11<br>
> 1 KSP Residual norm 8.266776463696e-01<br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 9.262528453386e-08<br>
> 1 KSP Residual norm 5.683232925010e-10<br>
> 2 KSP Residual norm 1.915223168286e-10<br>
> 3 KSP Residual norm 1.397893184942e-10<br>
> 4 KSP Residual norm 1.691441435404e-11<br>
> 5 KSP Residual norm 6.138315243419e-12<br>
> 6 KSP Residual norm 5.576043830003e-12<br>
> 7 KSP Residual norm 5.574440028225e-12<br>
> 8 KSP Residual norm 5.559544964428e-12<br>
> 9 KSP Residual norm 5.539862581746e-12<br>
> 10 KSP Residual norm 5.258329460152e-12<br>
> 11 KSP Residual norm 2.643581511791e-12<br>
> 12 KSP Residual norm 2.641293392449e-12<br>
> 13 KSP Residual norm 2.354608977643e-14<br>
> 2 KSP Residual norm 4.450925351013e-07<br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 6.653681330477e-14<br>
> 1 KSP Residual norm 6.650750698147e-14<br>
> 2 KSP Residual norm 6.111123464526e-14<br>
> 3 KSP Residual norm 2.026817941567e-14<br>
> 4 KSP Residual norm 9.604999144183e-15<br>
> 5 KSP Residual norm 9.208296307424e-15<br>
> 6 KSP Residual norm 9.196769686859e-15<br>
> 7 KSP Residual norm 9.185058975459e-15<br>
> 8 KSP Residual norm 9.180207477303e-15<br>
> 9 KSP Residual norm 8.991574890909e-15<br>
> 10 KSP Residual norm 8.032736869820e-15<br>
> 11 KSP Residual norm 1.536409278928e-15<br>
> 12 KSP Residual norm 1.177374264280e-15<br>
> 13 KSP Residual norm 1.175712092044e-15<br>
> 14 KSP Residual norm 2.572275406087e-16<br>
> 15 KSP Residual norm 2.548423809711e-16<br>
> 16 KSP Residual norm 8.616505207588e-17<br>
> 17 KSP Residual norm 7.563053994201e-18<br>
> 18 KSP Residual norm 6.807636198601e-18<br>
> 19 KSP Residual norm 9.747028518744e-19<br>
> 20 KSP Residual norm 2.419807103570e-21<br>
> 3 KSP Residual norm 2.986369469883e-09<br>
> Residual norms for fieldsplit_FE_split_ solve.<br>
> 0 KSP Residual norm 7.813223137340e-16<br>
> 1 KSP Residual norm 4.793103235095e-18<br>
> 2 KSP Residual norm 1.615526128222e-18<br>
> 3 KSP Residual norm 1.179102504397e-18<br>
> 4 KSP Residual norm 1.427467627551e-19<br>
> 5 KSP Residual norm 5.177440470993e-20<br>
> 6 KSP Residual norm 4.703763659148e-20<br>
> 7 KSP Residual norm 4.701953228322e-20<br>
> 8 KSP Residual norm 4.689269668869e-20<br>
> 9 KSP Residual norm 4.672625361251e-20<br>
> 10 KSP Residual norm 4.435174006113e-20<br>
> 11 KSP Residual norm 2.229156843383e-20<br>
> 12 KSP Residual norm 2.228887211080e-20<br>
> 13 KSP Residual norm 3.492936921635e-22<br>
> 4 KSP Residual norm 3.753341263086e-15<br>
<br>
</div></div></blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>