<div dir="ltr">Thanks Barry, trying with <span style="font-size:12.8px">-fieldsplit_u_type lu gives better convergence. I still used 4 procs though, probably with 1 proc it should also be the same.</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">The u block used a Nitsche-type operator to connect two non-matching domains. I don't think it will leave some rigid body motion leads to not sufficient constraints. Maybe you have other idea?<br></span><div><br></div><div><div> Residual norms for fieldsplit_u_ solve.</div><div> 0 KSP Residual norm 3.129067184300e+05 </div><div> 1 KSP Residual norm 5.906261468196e-01 </div><div> Residual norms for fieldsplit_wp_ solve.</div><div> 0 KSP Residual norm 0.000000000000e+00 </div><div> 0 KSP preconditioned resid norm 3.129067184300e+05 true resid norm 9.015150492169e+06 ||r(i)||/||b|| 1.000000000000e+00</div><div> Residual norms for fieldsplit_u_ solve.</div><div> 0 KSP Residual norm 9.999955993437e-01 </div><div> 1 KSP Residual norm 4.019774691831e-06 </div><div> Residual norms for fieldsplit_wp_ solve.</div><div> 0 KSP Residual norm 0.000000000000e+00 </div><div> 1 KSP preconditioned resid norm 5.003913641475e-01 true resid norm 4.692996324114e+01 ||r(i)||/||b|| 5.205677185522e-06</div><div> Residual norms for fieldsplit_u_ solve.</div><div> 0 KSP Residual norm 1.000012180204e+00 </div><div> 1 KSP Residual norm 1.017367950422e-05 </div><div> Residual norms for fieldsplit_wp_ solve.</div><div> 0 KSP Residual norm 0.000000000000e+00 </div><div> 2 KSP preconditioned resid norm 2.330910333756e-07 true resid norm 3.474855463983e+01 ||r(i)||/||b|| 3.854461960453e-06</div><div> Residual norms for fieldsplit_u_ solve.</div><div> 0 KSP Residual norm 1.000004200085e+00 </div><div> 1 KSP Residual norm 6.231613102458e-06 </div><div> Residual norms for fieldsplit_wp_ solve.</div><div> 0 KSP Residual norm 0.000000000000e+00 </div><div> 3 KSP preconditioned resid norm 8.671259838389e-11 true resid norm 3.545103468011e+01 ||r(i)||/||b|| 3.932384125024e-06</div><div>Linear solve converged due to CONVERGED_ATOL iterations 3</div><div>KSP Object: 4 MPI processes</div><div> type: gmres</div><div> GMRES: restart=1000, using Modified Gram-Schmidt Orthogonalization</div><div> GMRES: happy breakdown tolerance 1e-30</div><div> maximum iterations=1000, initial guess is zero</div><div> tolerances: relative=1e-20, absolute=1e-09, divergence=10000</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div>PC Object: 4 MPI processes</div><div> type: fieldsplit</div><div> FieldSplit with MULTIPLICATIVE composition: total splits = 2</div><div> Solver info for each split is in the following KSP objects:</div><div> Split number 0 Defined by IS</div><div> KSP Object: (fieldsplit_u_) 4 MPI processes</div><div> type: richardson</div><div> Richardson: damping factor=1</div><div> maximum iterations=1, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div> PC Object: (fieldsplit_u_) 4 MPI processes</div><div> type: lu</div><div> LU: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 0, needed 0</div><div> Factored matrix follows:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=938910, cols=938910</div><div> package used to perform factorization: pastix</div><div> total: nonzeros=0, allocated nonzeros=0</div><div> Error : 3.36878e-14 </div><div> total number of mallocs used during MatSetValues calls =0</div><div> PaStiX run parameters:</div><div> Matrix type : Unsymmetric </div><div> Level of printing (0,1,2): 0 </div><div> Number of refinements iterations : 3 </div><div> Error : 3.36878e-14 </div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_u_) 4 MPI processes</div><div> type: mpiaij</div><div> rows=938910, cols=938910, bs=3</div><div> Error : 3.36878e-14 </div><div> Error : 3.36878e-14 </div><div> total: nonzeros=8.60906e+07, allocated nonzeros=8.60906e+07</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 78749 nodes, limit used is 5</div><div> Split number 1 Defined by IS</div><div> KSP Object: (fieldsplit_wp_) 4 MPI processes</div><div> type: richardson</div><div> Richardson: damping factor=1</div><div> maximum iterations=1, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div> PC Object: (fieldsplit_wp_) 4 MPI processes</div><div> type: lu</div><div> LU: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: natural</div><div> factor fill ratio given 0, needed 0</div><div> Factored matrix follows:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=34141, cols=34141</div><div> package used to perform factorization: pastix</div><div> Error : -nan </div><div> Error : -nan </div><div> Error : -nan </div><div>total: nonzeros=0, allocated nonzeros=0</div><div> total number of mallocs used during MatSetValues calls =0</div><div> PaStiX run parameters:</div><div> Matrix type : Symmetric </div><div> Level of printing (0,1,2): 0 </div><div> Number of refinements iterations : 0 </div><div> Error : -nan </div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_wp_) 4 MPI processes</div><div> type: mpiaij</div><div> rows=34141, cols=34141</div><div> total: nonzeros=485655, allocated nonzeros=485655</div><div> total number of mallocs used during MatSetValues calls =0</div><div> not using I-node (on process 0) routines</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 4 MPI processes</div><div> type: mpiaij</div><div> rows=973051, cols=973051</div><div> total: nonzeros=9.90037e+07, allocated nonzeros=9.90037e+07</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 78749 nodes, limit used is 5</div><div><br></div><div><br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Giang</div></div></div>
<br><div class="gmail_quote">On Sun, Apr 23, 2017 at 10:19 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
> On Apr 23, 2017, at 2:42 PM, Hoang Giang Bui <<a href="mailto:hgbk2008@gmail.com">hgbk2008@gmail.com</a>> wrote:<br>
><br>
> Dear Matt/Barry<br>
><br>
> With your options, it results in<br>
><br>
> 0 KSP preconditioned resid norm 1.106709687386e+31 true resid norm 9.015150491938e+06 ||r(i)||/||b|| 1.000000000000e+00<br>
> Residual norms for fieldsplit_u_ solve.<br>
> 0 KSP Residual norm 2.407308987203e+36<br>
> 1 KSP Residual norm 5.797185652683e+72<br>
<br>
</span>It looks like Matt is right, hypre is seemly producing useless garbage.<br>
<br>
First how do things run on one process. If you have similar problems then debug on one process (debugging any kind of problem is always far easy on one process).<br>
<br>
First run with -fieldsplit_u_type lu (instead of using hypre) to see if that works or also produces something bad.<br>
<br>
What is the operator and the boundary conditions for u? It could be singular.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
<br>
<br>
<br>
<br>
> Residual norms for fieldsplit_wp_ solve.<br>
> 0 KSP Residual norm 0.000000000000e+00<br>
> ...<br>
> 999 KSP preconditioned resid norm 2.920157329174e+12 true resid norm 9.015683504616e+06 ||r(i)||/||b|| 1.000059124102e+00<br>
> Residual norms for fieldsplit_u_ solve.<br>
> 0 KSP Residual norm 1.533726746719e+36<br>
> 1 KSP Residual norm 3.692757392261e+72<br>
> Residual norms for fieldsplit_wp_ solve.<br>
> 0 KSP Residual norm 0.000000000000e+00<br>
><br>
> Do you suggest that the pastix solver for the "wp" block encounters small pivot? In addition, seem like the "u" block is also singular.<br>
><br>
> Giang<br>
><br>
> On Sun, Apr 23, 2017 at 7:39 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> Huge preconditioned norms but normal unpreconditioned norms almost always come from a very small pivot in an LU or ILU factorization.<br>
><br>
> The first thing to do is monitor the two sub solves. Run with the additional options -fieldsplit_u_ksp_type richardson -fieldsplit_u_ksp_monitor -fieldsplit_u_ksp_max_it 1 -fieldsplit_wp_ksp_type richardson -fieldsplit_wp_ksp_monitor -fieldsplit_wp_ksp_max_it 1<br>
><br>
> > On Apr 23, 2017, at 12:22 PM, Hoang Giang Bui <<a href="mailto:hgbk2008@gmail.com">hgbk2008@gmail.com</a>> wrote:<br>
> ><br>
> > Hello<br>
> ><br>
> > I encountered a strange convergence behavior that I have trouble to understand<br>
> ><br>
> > KSPSetFromOptions completed<br>
> > 0 KSP preconditioned resid norm 1.106709687386e+31 true resid norm 9.015150491938e+06 ||r(i)||/||b|| 1.000000000000e+00<br>
> > 1 KSP preconditioned resid norm 2.933141742664e+29 true resid norm 9.015152282123e+06 ||r(i)||/||b|| 1.000000198575e+00<br>
> > 2 KSP preconditioned resid norm 9.686409637174e+16 true resid norm 9.015354521944e+06 ||r(i)||/||b|| 1.000022631902e+00<br>
> > 3 KSP preconditioned resid norm 4.219243615809e+15 true resid norm 9.017157702420e+06 ||r(i)||/||b|| 1.000222648583e+00<br>
> > .....<br>
> > 999 KSP preconditioned resid norm 3.043754298076e+12 true resid norm 9.015425041089e+06 ||r(i)||/||b|| 1.000030454195e+00<br>
> > 1000 KSP preconditioned resid norm 3.043000287819e+12 true resid norm 9.015424313455e+06 ||r(i)||/||b|| 1.000030373483e+00<br>
> > Linear solve did not converge due to DIVERGED_ITS iterations 1000<br>
> > KSP Object: 4 MPI processes<br>
> > type: gmres<br>
> > GMRES: restart=1000, using Modified Gram-Schmidt Orthogonalization<br>
> > GMRES: happy breakdown tolerance 1e-30<br>
> > maximum iterations=1000, initial guess is zero<br>
> > tolerances: relative=1e-20, absolute=1e-09, divergence=10000<br>
> > left preconditioning<br>
> > using PRECONDITIONED norm type for convergence test<br>
> > PC Object: 4 MPI processes<br>
> > type: fieldsplit<br>
> > FieldSplit with MULTIPLICATIVE composition: total splits = 2<br>
> > Solver info for each split is in the following KSP objects:<br>
> > Split number 0 Defined by IS<br>
> > KSP Object: (fieldsplit_u_) 4 MPI processes<br>
> > type: preonly<br>
> > maximum iterations=10000, initial guess is zero<br>
> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > left preconditioning<br>
> > using NONE norm type for convergence test<br>
> > PC Object: (fieldsplit_u_) 4 MPI processes<br>
> > type: hypre<br>
> > HYPRE BoomerAMG preconditioning<br>
> > HYPRE BoomerAMG: Cycle type V<br>
> > HYPRE BoomerAMG: Maximum number of levels 25<br>
> > HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1<br>
> > HYPRE BoomerAMG: Convergence tolerance PER hypre call 0<br>
> > HYPRE BoomerAMG: Threshold for strong coupling 0.6<br>
> > HYPRE BoomerAMG: Interpolation truncation factor 0<br>
> > HYPRE BoomerAMG: Interpolation: max elements per row 0<br>
> > HYPRE BoomerAMG: Number of levels of aggressive coarsening 0<br>
> > HYPRE BoomerAMG: Number of paths for aggressive coarsening 1<br>
> > HYPRE BoomerAMG: Maximum row sums 0.9<br>
> > HYPRE BoomerAMG: Sweeps down 1<br>
> > HYPRE BoomerAMG: Sweeps up 1<br>
> > HYPRE BoomerAMG: Sweeps on coarse 1<br>
> > HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi<br>
> > HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi<br>
> > HYPRE BoomerAMG: Relax on coarse Gaussian-elimination<br>
> > HYPRE BoomerAMG: Relax weight (all) 1<br>
> > HYPRE BoomerAMG: Outer relax weight (all) 1<br>
> > HYPRE BoomerAMG: Using CF-relaxation<br>
> > HYPRE BoomerAMG: Measure type local<br>
> > HYPRE BoomerAMG: Coarsen type PMIS<br>
> > HYPRE BoomerAMG: Interpolation type classical<br>
> > linear system matrix = precond matrix:<br>
> > Mat Object: (fieldsplit_u_) 4 MPI processes<br>
> > type: mpiaij<br>
> > rows=938910, cols=938910, bs=3<br>
> > total: nonzeros=8.60906e+07, allocated nonzeros=8.60906e+07<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > using I-node (on process 0) routines: found 78749 nodes, limit used is 5<br>
> > Split number 1 Defined by IS<br>
> > KSP Object: (fieldsplit_wp_) 4 MPI processes<br>
> > type: preonly<br>
> > maximum iterations=10000, initial guess is zero<br>
> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > left preconditioning<br>
> > using NONE norm type for convergence test<br>
> > PC Object: (fieldsplit_wp_) 4 MPI processes<br>
> > type: lu<br>
> > LU: out-of-place factorization<br>
> > tolerance for zero pivot 2.22045e-14<br>
> > matrix ordering: natural<br>
> > factor fill ratio given 0, needed 0<br>
> > Factored matrix follows:<br>
> > Mat Object: 4 MPI processes<br>
> > type: mpiaij<br>
> > rows=34141, cols=34141<br>
> > package used to perform factorization: pastix<br>
> > Error : -nan<br>
> > Error : -nan<br>
> > total: nonzeros=0, allocated nonzeros=0<br>
> > Error : -nan<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > PaStiX run parameters:<br>
> > Matrix type : Symmetric<br>
> > Level of printing (0,1,2): 0<br>
> > Number of refinements iterations : 0<br>
> > Error : -nan<br>
> > linear system matrix = precond matrix:<br>
> > Mat Object: (fieldsplit_wp_) 4 MPI processes<br>
> > type: mpiaij<br>
> > rows=34141, cols=34141<br>
> > total: nonzeros=485655, allocated nonzeros=485655<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > not using I-node (on process 0) routines<br>
> > linear system matrix = precond matrix:<br>
> > Mat Object: 4 MPI processes<br>
> > type: mpiaij<br>
> > rows=973051, cols=973051<br>
> > total: nonzeros=9.90037e+07, allocated nonzeros=9.90037e+07<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > using I-node (on process 0) routines: found 78749 nodes, limit used is 5<br>
> ><br>
> > The pattern of convergence gives a hint that this system is somehow bad/singular. But I don't know why the preconditioned error goes up too high. Anyone has an idea?<br>
> ><br>
> > Best regards<br>
> > Giang Bui<br>
> ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div></div></div></div>