<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jul 13, 2016 at 10:34 AM, Hoang Giang Bui <span dir="ltr"><<a href="mailto:hgbk2008@gmail.com" target="_blank">hgbk2008@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Barry<div><br></div><div>This is a good comment. Since material behaviour depends very much on the trajectory of the solution. I suspect that the error may concatenate during time stepping.</div><div><br></div><div>I have re-run the simulation as you suggested and post the log file here: <a href="https://www.dropbox.com/s/d6l8ixme37uh47a/log13Jul16?dl=0" target="_blank">https://www.dropbox.com/s/d6l8ixme37uh47a/log13Jul16?dl=0</a></div><div><br></div><div>However, I did not get what <span style="font-size:12.8px">-ksp_monitor_true_solution used for? I see that I have the same log that I had before.</span></div></div></blockquote><div><br></div><div>That option is showing the last two numbers in these lines</div><div><br></div><div><div> 0 KSP preconditioned resid norm 1.150038785083e+00 true resid norm 8.673040929526e+07 ||r(i)||/||b|| 1.000000000000e+00</div><div><br></div><div>Notice that there are 7 orders of magnitude between the apparent residual (using the preconditioner), and the actual residual, Ax - b.</div><div>You are using Hypre, and this generally means the Hypre coarse grid operator is crap. Please</div><div><br></div><div> a) Try ML or GAMG and look at the output again</div><div><br></div><div> b) Try MUMPS, although you have 200 nonzeros/row so that fill-in might be extreme.</div><div><br></div><div>The consequence is that you solve to what you think is machine precision (1e-13), but all you really get is (1e-4), so I can understand</div><div>why the trajectory is completely different.</div><div><br></div><div> Matt</div><div><br></div><div> 1 KSP preconditioned resid norm 5.202876635759e-01 true resid norm 2.037005052213e+08 ||r(i)||/||b|| 2.348663022307e+00</div><div> 2 KSP preconditioned resid norm 3.386127782775e-01 true resid norm 1.762196838305e+08 ||r(i)||/||b|| 2.031809664712e+00</div><div> 3 KSP preconditioned resid norm 2.334102526025e-01 true resid norm 1.027451552306e+08 ||r(i)||/||b|| 1.184649721655e+00</div><div> 4 KSP preconditioned resid norm 1.791251896569e-01 true resid norm 7.709961160729e+07 ||r(i)||/||b|| 8.889570824556e-01</div><div> 5 KSP preconditioned resid norm 1.338763110903e-01 true resid norm 7.416954924746e+07 ||r(i)||/||b|| 8.551735181482e-01</div><div> 6 KSP preconditioned resid norm 8.064262880339e-02 true resid norm 5.164444100149e+07 ||r(i)||/||b|| 5.954594405945e-01</div><div> 7 KSP preconditioned resid norm 4.635705318709e-02 true resid norm 2.934800965373e+07 ||r(i)||/||b|| 3.383820034081e-01</div><div> 8 KSP preconditioned resid norm 2.772133866748e-02 true resid norm 1.528356929458e+07 ||r(i)||/||b|| 1.762192686368e-01</div><div> 9 KSP preconditioned resid norm 1.746753670007e-02 true resid norm 1.011788107951e+07 ||r(i)||/||b|| 1.166589799555e-01</div><div> 10 KSP preconditioned resid norm 1.090702407895e-02 true resid norm 5.487922954253e+06 ||r(i)||/||b|| 6.327564920823e-02</div><div> 11 KSP preconditioned resid norm 7.298748576067e-03 true resid norm 3.635843038640e+06 ||r(i)||/||b|| 4.192120235779e-02</div><div> 12 KSP preconditioned resid norm 5.263606789063e-03 true resid norm 2.556946903793e+06 ||r(i)||/||b|| 2.948155006496e-02</div><div> 13 KSP preconditioned resid norm 3.653208280595e-03 true resid norm 1.955721190606e+06 ||r(i)||/||b|| 2.254942881623e-02</div><div> 14 KSP preconditioned resid norm 2.344759624903e-03 true resid norm 1.161259621408e+06 ||r(i)||/||b|| 1.338930175522e-02</div><div> 15 KSP preconditioned resid norm 1.394564491254e-03 true resid norm 7.455856541894e+05 ||r(i)||/||b|| 8.596588673428e-03</div><div> 16 KSP preconditioned resid norm 9.523395328600e-04 true resid norm 4.383808867461e+05 ||r(i)||/||b|| 5.054523440028e-03</div><div> 17 KSP preconditioned resid norm 7.226014371144e-04 true resid norm 2.463564216053e+05 ||r(i)||/||b|| 2.840484941869e-03</div><div> 18 KSP preconditioned resid norm 5.312593384754e-04 true resid norm 2.332075376781e+05 ||r(i)||/||b|| 2.688878555665e-03</div><div> 19 KSP preconditioned resid norm 3.987403871945e-04 true resid norm 1.524236218549e+05 ||r(i)||/||b|| 1.757441514383e-03</div><div> 20 KSP preconditioned resid norm 3.024350484979e-04 true resid norm 1.113568566173e+05 ||r(i)||/||b|| 1.283942477870e-03</div><div> 21 KSP preconditioned resid norm 2.181724540430e-04 true resid norm 9.095158030900e+04 ||r(i)||/||b|| 1.048670022983e-03</div><div> 22 KSP preconditioned resid norm 1.497651066688e-04 true resid norm 7.045647741653e+04 ||r(i)||/||b|| 8.123618692570e-04</div><div> 23 KSP preconditioned resid norm 1.067332245914e-04 true resid norm 4.317487154207e+04 ||r(i)||/||b|| 4.978054628463e-04</div><div> 24 KSP preconditioned resid norm 8.206743871631e-05 true resid norm 3.328488127932e+04 ||r(i)||/||b|| 3.837740597534e-04</div><div> 25 KSP preconditioned resid norm 6.446633932980e-05 true resid norm 2.816657573261e+04 ||r(i)||/||b|| 3.247600923538e-04</div><div> 26 KSP preconditioned resid norm 5.068725017435e-05 true resid norm 2.427030232896e+04 ||r(i)||/||b|| 2.798361327495e-04</div><div> 27 KSP preconditioned resid norm 4.056292508453e-05 true resid norm 1.963628903861e+04 ||r(i)||/||b|| 2.264060460243e-04</div><div> 28 KSP preconditioned resid norm 3.278196251068e-05 true resid norm 1.710046122873e+04 ||r(i)||/||b|| 1.971679987179e-04</div><div> 29 KSP preconditioned resid norm 2.796514916728e-05 true resid norm 1.500292999274e+04 ||r(i)||/||b|| 1.729835027259e-04</div><div> 30 KSP preconditioned resid norm 2.469882695602e-05 true resid norm 1.317997814765e+04 ||r(i)||/||b|| 1.519649019847e-04</div><div> 31 KSP preconditioned resid norm 2.175528107880e-05 true resid norm 1.158572445412e+04 ||r(i)||/||b|| 1.335831866616e-04</div><div> 32 KSP preconditioned resid norm 1.912573933887e-05 true resid norm 1.001695718951e+04 ||r(i)||/||b|| 1.154953293880e-04</div><div> 33 KSP preconditioned resid norm 1.647102125210e-05 true resid norm 8.271485921360e+03 ||r(i)||/||b|| 9.537007825249e-05</div><div> 34 KSP preconditioned resid norm 1.337436641169e-05 true resid norm 6.611637805300e+03 ||r(i)||/||b|| 7.623206046211e-05</div><div> 35 KSP preconditioned resid norm 9.896966695703e-06 true resid norm 4.752788536204e+03 ||r(i)||/||b|| 5.479956309238e-05</div><div> 36 KSP preconditioned resid norm 6.766260764791e-06 true resid norm 3.239548441802e+03 ||r(i)||/||b|| 3.735193305468e-05</div><div> 37 KSP preconditioned resid norm 4.835158711776e-06 true resid norm 2.113941262442e+03 ||r(i)||/||b|| 2.437370329068e-05</div><div> 38 KSP preconditioned resid norm 3.598894380040e-06 true resid norm 1.653467554688e+03 ||r(i)||/||b|| 1.906445003688e-05</div><div> 39 KSP preconditioned resid norm 2.522642742745e-06 true resid norm 1.344572919946e+03 ||r(i)||/||b|| 1.550290066507e-05</div><div> 40 KSP preconditioned resid norm 1.750002168280e-06 true resid norm 1.015690774521e+03 ||r(i)||/||b|| 1.171089566825e-05</div><div> 41 KSP preconditioned resid norm 1.371380245282e-06 true resid norm 8.480814540622e+02 ||r(i)||/||b|| 9.778363332462e-06</div><div> 42 KSP preconditioned resid norm 1.174063380270e-06 true resid norm 7.575955225454e+02 ||r(i)||/||b|| 8.735062231359e-06</div><div> 43 KSP preconditioned resid norm 1.022078284946e-06 true resid norm 6.758159410670e+02 ||r(i)||/||b|| 7.792145183661e-06</div><div> 44 KSP preconditioned resid norm 8.861345665105e-07 true resid norm 5.913685641420e+02 ||r(i)||/||b|| 6.818468504268e-06</div><div> 45 KSP preconditioned resid norm 7.574040382433e-07 true resid norm 4.958820201473e+02 ||r(i)||/||b|| 5.717510434653e-06</div><div> 46 KSP preconditioned resid norm 6.331382122180e-07 true resid norm 3.988451175342e+02 ||r(i)||/||b|| 4.598676759110e-06</div><div> 47 KSP preconditioned resid norm 5.210644796074e-07 true resid norm 3.077459761874e+02 ||r(i)||/||b|| 3.548305360116e-06</div><div> 48 KSP preconditioned resid norm 4.285762531134e-07 true resid norm 2.383304155333e+02 ||r(i)||/||b|| 2.747945241696e-06</div><div> 49 KSP preconditioned resid norm 3.365753654637e-07 true resid norm 1.802176480688e+02 ||r(i)||/||b|| 2.077906117741e-06</div><div> 50 KSP preconditioned resid norm 2.556504175739e-07 true resid norm 1.322207275993e+02 ||r(i)||/||b|| 1.524502520785e-06</div><div> 51 KSP preconditioned resid norm 1.929395464892e-07 true resid norm 1.007938656038e+02 ||r(i)||/||b|| 1.162151388686e-06</div><div> 52 KSP preconditioned resid norm 1.518353128559e-07 true resid norm 7.979486270816e+01 ||r(i)||/||b|| 9.200332773308e-07</div><div> 53 KSP preconditioned resid norm 1.206065500213e-07 true resid norm 6.580266981926e+01 ||r(i)||/||b|| 7.587035545427e-07</div><div> 54 KSP preconditioned resid norm 9.426597887251e-08 true resid norm 5.333098459078e+01 ||r(i)||/||b|| 6.149052566928e-07</div><div> 55 KSP preconditioned resid norm 7.613592162567e-08 true resid norm 4.265349984159e+01 ||r(i)||/||b|| 4.917940568733e-07</div><div> 56 KSP preconditioned resid norm 6.268355987149e-08 true resid norm 3.467681120568e+01 ||r(i)||/||b|| 3.998229858184e-07</div><div> 57 KSP preconditioned resid norm 5.012883291890e-08 true resid norm 2.749870530323e+01 ||r(i)||/||b|| 3.170595587716e-07</div><div> 58 KSP preconditioned resid norm 3.875711489918e-08 true resid norm 2.037239239206e+01 ||r(i)||/||b|| 2.348933039472e-07</div><div> 59 KSP preconditioned resid norm 2.803879910778e-08 true resid norm 1.495957468476e+01 ||r(i)||/||b|| 1.724836168342e-07</div><div> 60 KSP preconditioned resid norm 1.925214804831e-08 true resid norm 1.036952152845e+01 ||r(i)||/||b|| 1.195603896339e-07</div><div> 61 KSP preconditioned resid norm 1.316807047769e-08 true resid norm 7.239457203086e+00 ||r(i)||/||b|| 8.347080639779e-08</div><div> 62 KSP preconditioned resid norm 9.095263534284e-09 true resid norm 5.546725364022e+00 ||r(i)||/||b|| 6.395363989508e-08</div><div> 63 KSP preconditioned resid norm 6.520024982652e-09 true resid norm 4.395022539849e+00 ||r(i)||/||b|| 5.067452783356e-08</div><div> 64 KSP preconditioned resid norm 5.077084953418e-09 true resid norm 3.613138054874e+00 ||r(i)||/||b|| 4.165941431885e-08</div><div> 65 KSP preconditioned resid norm 4.181478103167e-09 true resid norm 3.038027368880e+00 ||r(i)||/||b|| 3.502839884610e-08</div><div> 66 KSP preconditioned resid norm 3.474545560062e-09 true resid norm 2.484725611092e+00 ||r(i)||/||b|| 2.864883990842e-08</div><div> 67 KSP preconditioned resid norm 2.726294735157e-09 true resid norm 1.845741997810e+00 ||r(i)||/||b|| 2.128137077650e-08</div><div> 68 KSP preconditioned resid norm 2.081101207644e-09 true resid norm 1.271838867185e+00 ||r(i)||/||b|| 1.466427839462e-08</div><div> 69 KSP preconditioned resid norm 1.574053677511e-09 true resid norm 8.732579381622e-01 ||r(i)||/||b|| 1.006864772411e-08</div><div> 70 KSP preconditioned resid norm 1.202717674216e-09 true resid norm 5.849220507056e-01 ||r(i)||/||b|| 6.744140324696e-09</div><div> 71 KSP preconditioned resid norm 9.075713740333e-10 true resid norm 4.120181311262e-01 ||r(i)||/||b|| 4.750561359898e-09</div><div> 72 KSP preconditioned resid norm 6.365151508838e-10 true resid norm 3.065749731760e-01 ||r(i)||/||b|| 3.534803717256e-09</div><div> 73 KSP preconditioned resid norm 4.005974496315e-10 true resid norm 2.122086214944e-01 ||r(i)||/||b|| 2.446761444097e-09</div><div> 74 KSP preconditioned resid norm 2.374916890000e-10 true resid norm 1.567794082480e-01 ||r(i)||/||b|| 1.807663650177e-09</div><div> 75 KSP preconditioned resid norm 1.481096397633e-10 true resid norm 1.235242757193e-01 ||r(i)||/||b|| 1.424232592963e-09</div><div> 76 KSP preconditioned resid norm 1.085014154415e-10 true resid norm 1.047268461651e-01 ||r(i)||/||b|| 1.207498581132e-09</div><div> 77 KSP preconditioned resid norm 8.764582618532e-11 true resid norm 8.962364559579e-02 ||r(i)||/||b|| 1.033358960531e-09</div><div> 78 KSP preconditioned resid norm 7.109092680274e-11 true resid norm 7.176047852904e-02 ||r(i)||/||b|| 8.273969777399e-10</div><div> 79 KSP preconditioned resid norm 5.460763497752e-11 true resid norm 5.069849340150e-02 ||r(i)||/||b|| 5.845526824266e-10</div><div> 80 KSP preconditioned resid norm 3.799942459039e-11 true resid norm 3.044234442091e-02 ||r(i)||/||b|| 3.509996628435e-10</div><div> 81 KSP preconditioned resid norm 2.481109284531e-11 true resid norm 1.726059230919e-02 ||r(i)||/||b|| 1.990143070861e-10</div><div> 82 KSP preconditioned resid norm 1.569622532234e-11 true resid norm 1.070220060596e-02 ||r(i)||/||b|| 1.233961731867e-10</div><div> 83 KSP preconditioned resid norm 1.022582071414e-11 true resid norm 7.402265790954e-03 ||r(i)||/||b|| 8.534798637643e-11</div><div> 84 KSP preconditioned resid norm 7.284827374238e-12 true resid norm 5.658340974708e-03 ||r(i)||/||b|| 6.524056580253e-11</div><div> 85 KSP preconditioned resid norm 5.402886839508e-12 true resid norm 4.464802757767e-03 ||r(i)||/||b|| 5.147909244343e-11</div><div> 86 KSP preconditioned resid norm 3.933784995327e-12 true resid norm 3.350654653931e-03 ||r(i)||/||b|| 3.863298560628e-11</div><div> 87 KSP preconditioned resid norm 2.792049995877e-12 true resid norm 2.402140873006e-03 ||r(i)||/||b|| 2.769663942007e-11</div><div> 88 KSP preconditioned resid norm 2.058524741199e-12 true resid norm 1.747330249674e-03 ||r(i)||/||b|| 2.014668515774e-11</div><div> 89 KSP preconditioned resid norm 1.568241303093e-12 true resid norm 1.266336540932e-03 ||r(i)||/||b|| 1.460083667564e-11</div><div> 90 KSP preconditioned resid norm 1.164779378453e-12 true resid norm 8.484550691359e-04 ||r(i)||/||b|| 9.782671107287e-12</div><div> 91 KSP preconditioned resid norm 7.995560038101e-13 true resid norm 5.065061038629e-04 ||r(i)||/||b|| 5.840005921551e-12</div><div>Linear solve converged due to CONVERGED_RTOL iterations 91</div><div>KSP Object: 8 MPI processes</div><div> type: gmres</div><div> GMRES: restart=300, using Modified Gram-Schmidt Orthogonalization</div><div> GMRES: happy breakdown tolerance 1e-30</div><div> maximum iterations=300, initial guess is zero</div><div> tolerances: relative=1e-12, absolute=1e-20, divergence=10000</div><div> left preconditioning</div><div> using PRECONDITIONED norm type for convergence test</div><div>PC Object: 8 MPI processes</div><div> type: fieldsplit</div><div> FieldSplit with MULTIPLICATIVE composition: total splits = 2</div><div> Solver info for each split is in the following KSP objects:</div><div> Split number 0 Defined by IS</div><div> KSP Object: (fieldsplit_u_) 8 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (fieldsplit_u_) 8 MPI processes</div><div> type: hypre</div><div> HYPRE BoomerAMG preconditioning</div><div> HYPRE BoomerAMG: Cycle type V</div><div> HYPRE BoomerAMG: Maximum number of levels 25</div><div> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1</div><div> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0</div><div> HYPRE BoomerAMG: Threshold for strong coupling 0.6</div><div> HYPRE BoomerAMG: Interpolation truncation factor 0</div><div> HYPRE BoomerAMG: Interpolation: max elements per row 0</div><div> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0</div><div> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1</div><div> HYPRE BoomerAMG: Maximum row sums 0.9</div><div> HYPRE BoomerAMG: Sweeps down 1</div><div> HYPRE BoomerAMG: Sweeps up 1</div><div> HYPRE BoomerAMG: Sweeps on coarse 1</div><div> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi</div><div> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi</div><div> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination</div><div> HYPRE BoomerAMG: Relax weight (all) 1</div><div> HYPRE BoomerAMG: Outer relax weight (all) 1</div><div> HYPRE BoomerAMG: Using CF-relaxation</div><div> HYPRE BoomerAMG: Measure type local</div><div> HYPRE BoomerAMG: Coarsen type PMIS</div><div> HYPRE BoomerAMG: Interpolation type classical</div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_u_) 8 MPI processes</div><div> type: mpiaij</div><div> rows=438420, cols=438420, bs=3</div><div> total: nonzeros=7.95766e+07, allocated nonzeros=7.95766e+07</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 17349 nodes, limit used is 5</div><div> Split number 1 Defined by IS</div><div> KSP Object: (fieldsplit_wp_) 8 MPI processes</div><div> type: preonly</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (fieldsplit_wp_) 8 MPI processes</div><div> type: hypre</div><div> HYPRE BoomerAMG preconditioning</div><div> HYPRE BoomerAMG: Cycle type V</div><div> HYPRE BoomerAMG: Maximum number of levels 25</div><div> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1</div><div> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0</div><div> HYPRE BoomerAMG: Threshold for strong coupling 0.6</div><div> HYPRE BoomerAMG: Interpolation truncation factor 0</div><div> HYPRE BoomerAMG: Interpolation: max elements per row 0</div><div> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0</div><div> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1</div><div> HYPRE BoomerAMG: Maximum row sums 0.9</div><div> HYPRE BoomerAMG: Sweeps down 1</div><div> HYPRE BoomerAMG: Sweeps up 1</div><div> HYPRE BoomerAMG: Sweeps on coarse 1</div><div> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi</div><div> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi</div><div> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination</div><div> HYPRE BoomerAMG: Relax weight (all) 1</div><div> HYPRE BoomerAMG: Outer relax weight (all) 1</div><div> HYPRE BoomerAMG: Using CF-relaxation</div><div> HYPRE BoomerAMG: Measure type local</div><div> HYPRE BoomerAMG: Coarsen type PMIS</div><div> HYPRE BoomerAMG: Interpolation type classical</div><div> linear system matrix = precond matrix:</div><div> Mat Object: (fieldsplit_wp_) 8 MPI processes</div><div> type: mpiaij</div><div> rows=146140, cols=146140</div><div> total: nonzeros=596012, allocated nonzeros=596012</div><div> total number of mallocs used during MatSetValues calls =0</div><div> not using I-node (on process 0) routines</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 8 MPI processes</div><div> type: mpiaij</div><div> rows=584560, cols=584560, bs=4</div><div> total: nonzeros=9.29667e+07, allocated nonzeros=9.29667e+07</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 32431 nodes, limit used is 5</div><div>KSPSolve completed</div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr">Giang</div></div></div>
<br><div class="gmail_quote">On Wed, Jul 13, 2016 at 5:43 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
It is not uncommon for an iterative linear solver to work fine for some time steps but then start to perform poorly at a later timestep because the physics (mathematically the conditioning or eigenstructure of the Jacobian) changes over time; perhaps becomes singular. Another possibility is the trajectory of the solution is very sensitive to the solution of the nonlinear problem at each time step so that an iterative linear solver and a direct linear solver result in very difficult physical solutions after many time steps. In other words after many time-steps the computed solutions can be very different and if the computed solution for the iterative linear solver is eventually "non-physical" or ill-conditioned the nonlinear solver could break down.<br>
<br>
Please run with the iterative solver (that eventually breaks) with the option -ksp_monitor_true_solution -ksp_converged_reason and and send ALL the output (it will be very large, don't worry about it). Then we can see if the linear solver is breaking down. Note that by default PETSc linear solvers do not generate an error that stops the program if the linear solve fails, hence your NR code should call KSPGetConvergedReason() after EVERY linear solve and if the reason is negative your code needs to do something different since the linear solve failed and your code should not just keep on running NR.<br>
<span><font color="#888888"><br>
Barry<br>
</font></span><div><div><br>
<br>
> On Jul 12, 2016, at 9:52 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
><br>
> On Tue, Jul 12, 2016 at 8:44 AM, Hoang Giang Bui <<a href="mailto:hgbk2008@gmail.com" target="_blank">hgbk2008@gmail.com</a>> wrote:<br>
> Hi Matt<br>
><br>
> 1) In the log you sent, the linear solver converges due to the Relative Tolerance, 1.0e-9, not the breakdown tolerance 1e-30. Change rtol will affect the convergence.<br>
><br>
> Sorry i got it wrong in the previous email, the ksp_rtol 1.0e-12 DOES affect the convergence, and it took more iterations. But the simulation still failed at a definite time step.<br>
><br>
> 2) What do you mean -fieldsplit_wp_ksp_rtol 1.0e-8 does not work? ALWAYS send the view output.<br>
><br>
> In the log file I sent previously, the line<br>
><br>
> KSP Object: (fieldsplit_wp_) 8 MPI processes<br>
> type: preonly<br>
> maximum iterations=10000, initial guess is zero<br>
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> left preconditioning<br>
> using NONE norm type for convergence test<br>
><br>
> impressed me that the rtol for fieldsplit_wp is still 1.0e-5<br>
><br>
> KSP "preonly" does no iterations, so it does not read the tolerance. If you want to lower the tolerance,<br>
> choose a solver like GMRES<br>
><br>
> -fieldsplit_wp_ksp_type gmres -fieldsplit_wp_ksp_rtol 1e-8<br>
><br>
> 3) I can't tell you anything about Newton convergence if you do not send the output, -snes_monitor -snes_view<br>
><br>
> I did not yet use SNES, instead using my NR iterator so I have no view for SNES.<br>
><br>
> It is hard to debug an iteration which we did not code. It could be you have a bug. If not, then very small changes in<br>
> the iterates are making a difference, which means your Jacobians are close to singular. A problem reformulation would<br>
> probably help more than solver tweaking.<br>
><br>
> Thanks,<br>
><br>
> Matt<br>
><br>
> 4) If there is a difference between LU and an iterative solver with residual 1e-9, then your system is very ill-conditioned.<br>
> Yes it is ill-conditioned<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> Giang<br>
><br>
> On Tue, Jul 12, 2016 at 2:49 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Tue, Jul 12, 2016 at 7:42 AM, Hoang Giang Bui <<a href="mailto:hgbk2008@gmail.com" target="_blank">hgbk2008@gmail.com</a>> wrote:<br>
> Hello<br>
><br>
> I encountered different convergence behaviour of Newton Raphson when using different solver settings with PETSc<br>
><br>
> For the first solver configuration, I used direct solver<br>
> -ksp_type preonly<br>
> -pc_type lu<br>
> -pc_factor_mat_solver_package mumps<br>
> -mat_mumps_icntl_1 6<br>
> -mat_mumps_icntl_4 3<br>
> -mat_mumps_icntl_7 4<br>
> -mat_mumps_icntl_14 40<br>
> -mat_mumps_icntl_23 0<br>
><br>
> The simulation can run completely and the NR typically converged after 6/7 iterations. Of course, it's very slow. For the second solver configuration:<br>
> -ksp_type gmres<br>
> -ksp_max_it 300<br>
> -ksp_gmres_restart 300<br>
> -ksp_gmres_modifiedgramschmidt<br>
> -pc_view<br>
> -pc_fieldsplit_type multiplicative<br>
> -fieldsplit_u_pc_type hypre<br>
> -fieldsplit_u_pc_hypre_type boomeramg<br>
> -fieldsplit_u_pc_hypre_boomeramg_coarsen_type PMIS<br>
> -fieldsplit_u_pc_hypre_boomeramg_strong_threshold 0.6<br>
> -fieldsplit_u_pc_hypre_boomeramg_max_levels 25<br>
> -fieldsplit_wp_ksp_rtol 1.0e-8<br>
> -fieldsplit_wp_pc_type hypre<br>
> -fieldsplit_wp_pc_hypre_type boomeramg<br>
> -fieldsplit_wp_pc_hypre_boomeramg_coarsen_type PMIS<br>
> -fieldsplit_wp_pc_hypre_boomeramg_strong_threshold 0.6<br>
> -fieldsplit_wp_pc_hypre_boomeramg_max_levels 25<br>
><br>
> The solver runs much faster, but the NR does not converge in 30 iterations after some time steps. I thought setting the solver tolerance -ksp_rtol 1.0e-12 but it doesn't help much because GMRES already terminate with tolerance 1e-30 (see sample log file). Can we set the tolerance of the sub-ksp of the Fieldsplit? I tried -fieldsplit_wp_ksp_rtol 1.0e-8 but it doesn't work.<br>
><br>
> 1) In the log you sent, the linear solver converges due to the Relative Tolerance, 1.0e-9, not the breakdown tolerance 1e-30. Change rtol will affect the convergence.<br>
><br>
> 2) What do you mean -fieldsplit_wp_ksp_rtol 1.0e-8 does not work? ALWAYS send the view output.<br>
><br>
> 3) I can't tell you anything about Newton convergence if you do not send the output, -snes_monitor -snes_view<br>
><br>
> 4) If there is a difference between LU and an iterative solver with residual 1e-9, then your system is very ill-conditioned.<br>
><br>
> Thanks,<br>
><br>
> Matt<br>
><br>
> Sorry this problem is run with many time steps and is quite big so I cannot reproduce in a simple test case.<br>
><br>
> Giang<br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
<br>
</div></div></blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>