<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I am using periodic along x and Neumann along y. I remove the
nullspace via -ksp_constant_null_space.<br>
Matt's suggestion worked; I will also give a try to umfpack and
cholesky.<br>
<br>
Thanks,<br>
Michele<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 06:50 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote
cite="mid:CAMYG4G=fUzb9jsoDuiF3ZXERZX-sD3QFojRYmkupj+0x5_hMBQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Feb 11, 2016 at 12:30 AM,
Dave May <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span class="">On 11 February
2016 at 07:05, Michele Rosso <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:mrosso@uci.edu" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:mrosso@uci.edu">mrosso@uci.edu</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
I tried setting
-mat_superlu_dist_replacetinypivot true: it
does help to advance the run past the previous
"critical" point but eventually it stops
later with the same error. <br>
I forgot to mention my system is singular: I
remove the constant null space but I am not
sure if the coarse solver needs to be
explicity informed of this.</div>
</blockquote>
<div><br>
</div>
</span>
<div>Right - are you using pure Newmann boundary
conditions?<br>
</div>
<div><br>
To make the solution unique, are you <br>
</div>
<div>(a) imposing a single Dichletet boundary
condition on your field<br>
</div>
<div>by explicitly modifying the matrix<br>
</div>
<div>(b) imposing a a condition like<br>
</div>
<div> \int \phi dV = 0 <br>
</div>
<div>via something like -ksp_constant_null_space<br>
</div>
<div><br>
If you removed removed the null space by modifying
the matrix explicitly (a), the sparse direct
solver<br>
</div>
<div>should go through. If you use (b), then this
method cannot be used to help the direct solver.<br>
<br>
</div>
<div>If this is the intended target problem size
(16x16), gather the matrix and using petsc
Cholesky or Umfpack.<br>
</div>
<div>Cholesky is more stable than LU and can usually
deal with a single zero eigenvaue without
resorting to tricks. Umfpack will solve the
problem easily as it uses clever re-ordering. If
you have access to MKL-Pardiso, that will also
work great. <br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>An easy fix is just to use -pc_type svd on the coarse
grid.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Thanks,<br>
</div>
<div> Dave <br>
</div>
<div>
<div class="h5">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div><span><font color="#888888"><br>
<br>
Michele</font></span>
<div>
<div><br>
<br>
On Wed, 2016-02-10 at 22:15 -0600, Barry
Smith wrote:
<blockquote type="CITE">
<pre> You can try the option
-mat_superlu_dist_replacetinypivot true
if you are luck it get you past the zero pivot but still give an adequate preconditioner.
Barry
<font color="#737373">> On Feb 10, 2016, at 9:49 PM, Michele Rosso <<a moz-do-not-send="true" href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">> </font>
<font color="#737373">> Hong,</font>
<font color="#737373">> </font>
<font color="#737373">> here if the output of grep -info:</font>
<font color="#737373">> </font>
<font color="#737373">> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]</font>
<font color="#737373">> Replace tiny pivots FALSE</font>
<font color="#737373">> tolerance for zero pivot 2.22045e-14</font>
<font color="#737373">> </font>
<font color="#737373">> It seems it is not replacing small pivots: could this be the problem?</font>
<font color="#737373">> I will also try Barry's suggestion to diagnose the problem.</font>
<font color="#737373">> </font>
<font color="#737373">> Thanks,</font>
<font color="#737373">> Michele</font>
<font color="#737373">> </font>
<font color="#737373">> </font>
<font color="#737373">> On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:</font>
<font color="#737373">>> > On Feb 10, 2016, at 9:00 PM, Hong <<a moz-do-not-send="true" href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > Michele :</font>
<font color="#737373">>> > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot.</font>
<font color="#737373">>> > Run your code with '-info |grep pivot' to verify.</font>
<font color="#737373">>> </font>
<font color="#737373">>> </font>
<font color="#737373">>> Michele</font>
<font color="#737373">>> </font>
<font color="#737373">>> You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened.</font>
<font color="#737373">>> </font>
<font color="#737373">>> Barry</font>
<font color="#737373">>> </font>
<font color="#737373">>> </font>
<font color="#737373">>> > </font>
<font color="#737373">>> > Hong</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > Hi Matt,</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > the ksp_view output was an attachment to my previous email.</font>
<font color="#737373">>> > Here it is:</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > KSP Object: 1 MPI processes</font>
<font color="#737373">>> > type: cg</font>
<font color="#737373">>> > maximum iterations=10000</font>
<font color="#737373">>> > tolerances: relative=1e-08, absolute=1e-50, divergence=10000.</font>
<font color="#737373">>> > left preconditioning</font>
<font color="#737373">>> > using nonzero initial guess</font>
<font color="#737373">>> > using UNPRECONDITIONED norm type for convergence test</font>
<font color="#737373">>> > PC Object: 1 MPI processes</font>
<font color="#737373">>> > type: mg</font>
<font color="#737373">>> > MG: type is MULTIPLICATIVE, levels=4 cycles=v</font>
<font color="#737373">>> > Cycles per PCApply=1</font>
<font color="#737373">>> > Using Galerkin computed coarse grid matrices</font>
<font color="#737373">>> > Coarse grid solver -- level -------------------------------</font>
<font color="#737373">>> > KSP Object: (mg_coarse_) 1 MPI processes</font>
<font color="#737373">>> > type: preonly</font>
<font color="#737373">>> > maximum iterations=1, initial guess is zero</font>
<font color="#737373">>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</font>
<font color="#737373">>> > left preconditioning</font>
<font color="#737373">>> > using NONE norm type for convergence test</font>
<font color="#737373">>> > PC Object: (mg_coarse_) 1 MPI processes</font>
<font color="#737373">>> > type: lu</font>
<font color="#737373">>> > LU: out-of-place factorization</font>
<font color="#737373">>> > tolerance for zero pivot 2.22045e-14</font>
<font color="#737373">>> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]</font>
<font color="#737373">>> > matrix ordering: nd</font>
<font color="#737373">>> > factor fill ratio given 0., needed 0.</font>
<font color="#737373">>> > Factored matrix follows:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=16, cols=16</font>
<font color="#737373">>> > package used to perform factorization: superlu_dist</font>
<font color="#737373">>> > total: nonzeros=0, allocated nonzeros=0</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > SuperLU_DIST run parameters:</font>
<font color="#737373">>> > Process grid nprow 1 x npcol 1 </font>
<font color="#737373">>> > Equilibrate matrix TRUE </font>
<font color="#737373">>> > Matrix input mode 0 </font>
<font color="#737373">>> > Replace tiny pivots FALSE </font>
<font color="#737373">>> > Use iterative refinement FALSE </font>
<font color="#737373">>> > Processors in row 1 col partition 1 </font>
<font color="#737373">>> > Row permutation LargeDiag </font>
<font color="#737373">>> > Column permutation METIS_AT_PLUS_A</font>
<font color="#737373">>> > Parallel symbolic factorization FALSE </font>
<font color="#737373">>> > Repeated factorization SamePattern</font>
<font color="#737373">>> > linear system matrix = precond matrix:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=16, cols=16</font>
<font color="#737373">>> > total: nonzeros=72, allocated nonzeros=72</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > not using I-node routines</font>
<font color="#737373">>> > Down solver (pre-smoother) on level 1 -------------------------------</font>
<font color="#737373">>> > KSP Object: (mg_levels_1_) 1 MPI processes</font>
<font color="#737373">>> > type: richardson</font>
<font color="#737373">>> > Richardson: damping factor=1.</font>
<font color="#737373">>> > maximum iterations=2</font>
<font color="#737373">>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</font>
<font color="#737373">>> > left preconditioning</font>
<font color="#737373">>> > using nonzero initial guess</font>
<font color="#737373">>> > using NONE norm type for convergence test</font>
<font color="#737373">>> > PC Object: (mg_levels_1_) 1 MPI processes</font>
<font color="#737373">>> > type: sor</font>
<font color="#737373">>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</font>
<font color="#737373">>> > linear system matrix = precond matrix:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=64, cols=64</font>
<font color="#737373">>> > total: nonzeros=304, allocated nonzeros=304</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > not using I-node routines</font>
<font color="#737373">>> > Up solver (post-smoother) same as down solver (pre-smoother)</font>
<font color="#737373">>> > Down solver (pre-smoother) on level 2 -------------------------------</font>
<font color="#737373">>> > KSP Object: (mg_levels_2_) 1 MPI processes</font>
<font color="#737373">>> > type: richardson</font>
<font color="#737373">>> > Richardson: damping factor=1.</font>
<font color="#737373">>> > maximum iterations=2</font>
<font color="#737373">>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</font>
<font color="#737373">>> > left preconditioning</font>
<font color="#737373">>> > using nonzero initial guess</font>
<font color="#737373">>> > using NONE norm type for convergence test</font>
<font color="#737373">>> > PC Object: (mg_levels_2_) 1 MPI processes</font>
<font color="#737373">>> > type: sor</font>
<font color="#737373">>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</font>
<font color="#737373">>> > linear system matrix = precond matrix:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=256, cols=256</font>
<font color="#737373">>> > total: nonzeros=1248, allocated nonzeros=1248</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > not using I-node routines</font>
<font color="#737373">>> > Up solver (post-smoother) same as down solver (pre-smoother)</font>
<font color="#737373">>> > Down solver (pre-smoother) on level 3 -------------------------------</font>
<font color="#737373">>> > KSP Object: (mg_levels_3_) 1 MPI processes</font>
<font color="#737373">>> > type: richardson</font>
<font color="#737373">>> > Richardson: damping factor=1.</font>
<font color="#737373">>> > maximum iterations=2</font>
<font color="#737373">>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</font>
<font color="#737373">>> > left preconditioning</font>
<font color="#737373">>> > using nonzero initial guess</font>
<font color="#737373">>> > using NONE norm type for convergence test</font>
<font color="#737373">>> > PC Object: (mg_levels_3_) 1 MPI processes</font>
<font color="#737373">>> > type: sor</font>
<font color="#737373">>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</font>
<font color="#737373">>> > linear system matrix = precond matrix:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=1024, cols=1024</font>
<font color="#737373">>> > total: nonzeros=5056, allocated nonzeros=5056</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > has attached null space</font>
<font color="#737373">>> > not using I-node routines</font>
<font color="#737373">>> > Up solver (post-smoother) same as down solver (pre-smoother)</font>
<font color="#737373">>> > linear system matrix = precond matrix:</font>
<font color="#737373">>> > Mat Object: 1 MPI processes</font>
<font color="#737373">>> > type: seqaij</font>
<font color="#737373">>> > rows=1024, cols=1024</font>
<font color="#737373">>> > total: nonzeros=5056, allocated nonzeros=5056</font>
<font color="#737373">>> > total number of mallocs used during MatSetValues calls =0</font>
<font color="#737373">>> > has attached null space</font>
<font color="#737373">>> > not using I-node routines</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > </font>
<font color="#737373">>> > Michele</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > </font>
<font color="#737373">>> > </font>
<font color="#737373">>> > </font>
<font color="#737373">>> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote:</font>
<font color="#737373">>> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso <<a moz-do-not-send="true" href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>> >> Hi,</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> I encountered the following error while solving a symmetric positive defined system:</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0</font>
<font color="#737373">>> >> PCSETUP_FAILED due to SUBPC_ERROR </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ).</font>
<font color="#737373">>> >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation.</font>
<font color="#737373">>> >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . </font>
<font color="#737373">>> >> The options I am using are:</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> -ksp_type cg</font>
<font color="#737373">>> >> -ksp_norm_type unpreconditioned</font>
<font color="#737373">>> >> -ksp_rtol 1e-8</font>
<font color="#737373">>> >> -ksp_lag_norm</font>
<font color="#737373">>> >> -ksp_initial_guess_nonzero yes</font>
<font color="#737373">>> >> -pc_type mg</font>
<font color="#737373">>> >> -pc_mg_galerkin</font>
<font color="#737373">>> >> -pc_mg_levels 4</font>
<font color="#737373">>> >> -mg_levels_ksp_type richardson</font>
<font color="#737373">>> >> -mg_coarse_ksp_constant_null_space</font>
<font color="#737373">>> >> -mg_coarse_pc_type lu</font>
<font color="#737373">>> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist</font>
<font color="#737373">>> >> -options_left</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday).</font>
<font color="#737373">>> >> I would appreciate any suggestion on this matter.</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view?</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> Matt</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> Thanks,</font>
<font color="#737373">>> >> Michele</font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> </font>
<font color="#737373">>> >> --</font>
<font color="#737373">>> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.</font>
<font color="#737373">>> >> -- Norbert Wiener</font>
<font color="#737373">>> > </font>
<font color="#737373">>> > </font>
<font color="#737373">>> </font>
<font color="#737373">>> </font>
<font color="#737373">>> </font>
<font color="#737373">> </font>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">What most experimenters take for
granted before they begin their experiments is infinitely
more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>