<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 22, 2020, at 11:56 AM, Ashish Patel <<a href="mailto:ashish.patel@onscale.com" class="">ashish.patel@onscale.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Thanks Stefano, Barry and Hong for your feedback. <br class=""></div><div class="">Stefano, the option that you suggested is helpful to freeze the dirichlet values. However, it did not completely solve the convergence issue for nonlinear problems using linesearch since the direction vector which is calculated before any Solve uses the residual which is not yet updated for the new Dirichlet values (u(t) =b(t)). The update happens only after the solve at SNES iteration 0. To fix this I now update the residual for nodes adjacent to Dirichlet nodes without depending on any Solve. The Solve stage only updates the solution for dirichlet nodes. Calculating the residual this way resolves the issue.<br class=""></div><div class=""><br class=""></div><div class="">Barry, Hong yes you are right, the dirichlet boundary conditions are time dependent. We do solve the equations implicitly and use TSIFunction and TSIJacobian to set up the problem. The dirichlet nodes are part of computational nodes. The jacobian for dirchlet nodes has diagonal 1 and rows & colums 0. For setting the residual in TSIFunction the residual for dirichlet nodes are set to (solution_value-Dirichlet_value). <br class=""></div></div></div></blockquote><div><br class=""></div> Understood. The reason the line search cannot succeed well but everything is fine with no line search is that you are putting into the Dirichlet nodes (solution_value-Dirichlet_value) for the next time-step but with multistage integrators and line searches the nonlinear function being evaluated is not simply at the new time step. hence for all those other function evaluations the solution you put in (solution_value-Dirichlet_value) is meaningless. That is your "function" on the Dirichlet nodes is not a function because it doesn't depend on the current input. </div><div><br class=""></div><div> As I indicated before there are two approaches to putting the simulation on solid footing. Rewriting the boundary condition as an ODE if possible so TS manages the handling of the boundary just like the interior points or telling TS you are solving a DAE, this means when you provide the IFunction and IJacobian you take into account that there is no U_t on the left handside for the boundary nodes.</div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Regards</div><div class="">Ashish<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 21, 2020 at 6:23 PM Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<span class="">Can you elaborate a bit on how you "set the RHS as increment of (Dirichlet_value-solution_value)”?<br class="">
</span><span class=""><br class="">
</span><span class="">It seems that the boundary condition is imposed as algebraic equations (e.g. 0 = u(t)-b(t) ), and the boundary nodes are included in the computational domain. This approach should work theoretically and is used in some TS examples such
as ex17.c, you might want to check if the algebraic equations are implemented properly in your IFunction() and IJacobian(). An alternative approach you can try is to remove the boundary nodes from the unknown variables, see the example ex25.c. In this way
you solve ODEs instead of DAEs and do not need to worry about SNES modifying certain values.</span>
<div class=""><br class="">
</div>
<div class="">Hong (Mr.)<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 19, 2020, at 5:15 PM, Ashish Patel <<a href="mailto:ashish.patel@onscale.com" target="_blank" class="">ashish.patel@onscale.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">Dear PETSc users,</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><br class="">
</span></font></div>
<div class="">
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><span style="color:rgb(29,28,29);font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">We
use PETSc as part of a finite element method program and we are trying to properly implement Dirichlet boundary conditions for non-linear, transient problems. We find that when we use a line search method it also changes the non-zero solution value of Dirichlet
nodes as it steps through the line search iteration. I was wondering if there is a way to freeze some index sets of a vector from changing during the line search operation?
<br class="">
</span></span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><span style="color:rgb(29,28,29);font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class=""><br class="">
</span></span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><span style="color:rgb(29,28,29);font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">We
are using the TS framework to setup the problem and use 'MatZeroRowsColumns' to set the diagonal of the jacobian to 1 for the dirichlet nodes and set the RHS as increment of (Dirichlet_value-solution_value). This works when the line search method is turned
off by using '-snes_linesearch_type basic' however using the default 'bt' linesearch, the TS diverges with error shown below. In a separate implementation if we overwrote the dirichlet nodes of the solution vector in TS residual function with the Dirichlet
values then the 'bt' line search method converged to the right solution. However we would like to avoid modifying the internal PETSc vector in our implementation.</span></span></font></div>
</div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><br class="">
0 TS dt 1. time 0.<br class="">
0 SNES Function norm 2.378549386020e+03 <br class="">
Line search: gnorm after quadratic fit 4.369235425165e+03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.385369069060e+03 lambda=1.0000000000000002e-02<br class="">
Line search: Cubically determined step, current gnorm 2.373925008934e+03 lambda=3.8846250444606093e-03<br class="">
1 SNES Function norm 2.373925008934e+03 <br class="">
Line search: gnorm after quadratic fit 5.006914179995e+03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.420957096780e+03 lambda=1.0000000000000002e-02<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.376034946750e+03 lambda=1.6129422079664700e-03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.374313344729e+03 lambda=4.8465026740690043e-04<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999473242e+03 lambda=1.5857251532828948e-04<br class="">
Line search: Cubically determined step, current gnorm 2.373921668024e+03 lambda=5.8116507162753387e-05<br class="">
2 SNES Function norm 2.373921668024e+03 <br class="">
Line search: gnorm after quadratic fit 4.771035112853e+03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.410650718394e+03 lambda=1.0000000000000002e-02<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.375104094198e+03 lambda=1.8983783738011522e-03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.374049151562e+03 lambda=7.0688528086485479e-04<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373935090907e+03 lambda=2.9132722794896019e-04<br class="">
Line search: Cubically determined step, current gnorm 2.373921032081e+03 lambda=1.2527602265373028e-04<br class="">
3 SNES Function norm 2.373921032081e+03 <br class="">
Line search: gnorm after quadratic fit 5.117914832660e+03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.422635362094e+03 lambda=1.0000000000000002e-02<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.375923870970e+03 lambda=1.5769887508300913e-03<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.374287081592e+03 lambda=4.8018017100729705e-04<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373999131160e+03 lambda=1.5908966655977892e-04<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373930608274e+03 lambda=5.7603977147935371e-05<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373922022038e+03 lambda=2.3884787050507805e-05<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921396636e+03 lambda=1.0282405393471519e-05<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921173719e+03 lambda=4.3868704034012554e-06<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921091304e+03 lambda=1.8774991151727107e-06<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921057168e+03 lambda=8.0331628193813397e-07<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921042769e+03 lambda=3.4377811174560817e-07<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921036645e+03 lambda=1.4712421886880395e-07<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921034032e+03 lambda=6.2965363212312132e-08<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032916e+03 lambda=2.6947718250469261e-08<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032438e+03 lambda=1.1533043989179318e-08<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032233e+03 lambda=4.9359018284334258e-09<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032146e+03 lambda=2.1124641857409436e-09<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032109e+03 lambda=9.0409137304143975e-10<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032093e+03 lambda=3.8693264359674819e-10<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032086e+03 lambda=1.6559911204766632e-10<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032083e+03 lambda=7.0873187020534353e-11<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=3.0332192901757729e-11<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.2981600435512875e-11<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=5.5559212521628090e-12<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=2.3777380405336958e-12<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=1.0176091649134191e-12<br class="">
Line search: Cubic step no good, shrinking lambda, current gnorm 2.373921032081e+03 lambda=4.3555419789721080e-13<br class="">
Line search: unable to find good step length! After 27 tries <br class="">
Line search: fnorm=2.3739210320805191e+03, gnorm=2.3739210320805323e+03, ynorm=8.5698020038772756e+03, minlambda=9.9999999999999998e-13, lambda=4.3555419789721080e-13, initial slope=-5.6355010665542409e+06<br class="">
SNES Object: 1 MPI processes<br class="">
type: newtonls<br class="">
maximum iterations=50, maximum function evaluations=10000<br class="">
tolerances: relative=1e-08, absolute=1e-50, solution=1e-08<br class="">
total number of linear solver iterations=4<br class="">
total number of function evaluations=48<br class="">
norm schedule ALWAYS<br class="">
SNESLineSearch Object: 1 MPI processes<br class="">
type: bt<br class="">
interpolation: cubic<br class="">
alpha=1.000000e-04<br class="">
maxstep=1.000000e+08, minlambda=1.000000e-12<br class="">
tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08<br class="">
maximum iterations=40<br class="">
KSP Object: 1 MPI processes<br class="">
type: preonly<br class="">
maximum iterations=10000, initial guess is zero<br class="">
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class="">
left preconditioning<br class="">
using NONE norm type for convergence test<br class="">
PC Object: 1 MPI processes<br class="">
type: cholesky<br class="">
out-of-place factorization<br class="">
tolerance for zero pivot 2.22045e-14<br class="">
matrix ordering: natural<br class="">
factor fill ratio given 0., needed 0.<br class="">
Factored matrix follows:<br class="">
Mat Object: 1 MPI processes<br class="">
type: mumps<br class="">
rows=38154, cols=38154<br class="">
package used to perform factorization: mumps<br class="">
total: nonzeros=7080060, allocated nonzeros=7080060<br class="">
total number of mallocs used during MatSetValues calls=0<br class="">
MUMPS run parameters:<br class="">
SYM (matrix type): 2 <br class="">
PAR (host participation): 1 <br class="">
ICNTL(1) (output for error): 6 <br class="">
ICNTL(2) (output of diagnostic msg): 0 <br class="">
ICNTL(3) (output for global info): 0 <br class="">
ICNTL(4) (level of printing): 0 <br class="">
ICNTL(5) (input mat struct): 0 <br class="">
ICNTL(6) (matrix prescaling): 7 <br class="">
ICNTL(7) (sequential matrix ordering):7 <br class="">
ICNTL(8) (scaling strategy): 77 <br class="">
ICNTL(10) (max num of refinements): 0 <br class="">
ICNTL(11) (error analysis): 0 <br class="">
ICNTL(12) (efficiency control): 0 <br class="">
ICNTL(13) (efficiency control): 0 <br class="">
ICNTL(14) (percentage of estimated workspace increase): 20 <br class="">
ICNTL(18) (input mat struct): 0 <br class="">
ICNTL(19) (Schur complement info): 0 <br class="">
ICNTL(20) (rhs sparse pattern): 0 <br class="">
ICNTL(21) (solution struct): 0 <br class="">
ICNTL(22) (in-core/out-of-core facility): 0 <br class="">
ICNTL(23) (max size of memory can be allocated locally):0 <br class="">
ICNTL(24) (detection of null pivot rows): 0 <br class="">
ICNTL(25) (computation of a null space basis): 0 <br class="">
ICNTL(26) (Schur options for rhs or solution): 0 <br class="">
ICNTL(27) (experimental parameter): -32 <br class="">
ICNTL(28) (use parallel or sequential ordering): 1 <br class="">
ICNTL(29) (parallel ordering): 0 <br class="">
ICNTL(30) (user-specified set of entries in inv(A)): 0 <br class="">
ICNTL(31) (factors is discarded in the solve phase): 0 <br class="">
ICNTL(33) (compute determinant): 0 <br class="">
ICNTL(35) (activate BLR based factorization): 0 <br class="">
ICNTL(36) (choice of BLR factorization variant): 0 <br class="">
ICNTL(38) (estimated compression rate of LU factors): 333 <br class="">
CNTL(1) (relative pivoting threshold): 0.01 <br class="">
CNTL(2) (stopping criterion of refinement): 1.49012e-08 <br class="">
CNTL(3) (absolute pivoting threshold): 0. <br class="">
CNTL(4) (value of static pivoting): -1. <br class="">
CNTL(5) (fixation for null pivots): 0. <br class="">
CNTL(7) (dropping parameter for BLR): 0. <br class="">
RINFO(1) (local estimated flops for the elimination after analysis):
<br class="">
[0] 2.73979e+09 <br class="">
RINFO(2) (local estimated flops for the assembly after factorization):
<br class="">
[0] 1.08826e+07 <br class="">
RINFO(3) (local estimated flops for the elimination after factorization):
<br class="">
[0] 2.73979e+09 <br class="">
INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization):
<br class="">
[0] 94 <br class="">
INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization):
<br class="">
[0] 94 <br class="">
INFO(23) (num of pivots eliminated on this processor after factorization):
<br class="">
[0] 38154 <br class="">
RINFOG(1) (global estimated flops for the elimination after analysis): 2.73979e+09
<br class="">
RINFOG(2) (global estimated flops for the assembly after factorization): 1.08826e+07
<br class="">
RINFOG(3) (global estimated flops for the elimination after factorization): 2.73979e+09
<br class="">
(RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0)<br class="">
INFOG(3) (estimated real workspace for factors on all processors after analysis): 8377336
<br class="">
INFOG(4) (estimated integer workspace for factors on all processors after analysis): 447902
<br class="">
INFOG(5) (estimated maximum front size in the complete tree): 990
<br class="">
INFOG(6) (number of nodes in the complete tree): 2730 <br class="">
INFOG(7) (ordering option effectively use after analysis): 5 <br class="">
INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100
<br class="">
INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 8377336
<br class="">
INFOG(10) (total integer space store the matrix factors after factorization): 447902
<br class="">
INFOG(11) (order of largest frontal matrix after factorization): 990
<br class="">
INFOG(12) (number of off-diagonal pivots): 10 <br class="">
INFOG(13) (number of delayed pivots after factorization): 0 <br class="">
INFOG(14) (number of memory compress after factorization): 0 <br class="">
INFOG(15) (number of steps of iterative refinement after solution): 0
<br class="">
INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 94
<br class="">
INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 94
<br class="">
INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 94
<br class="">
INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 94
<br class="">
INFOG(20) (estimated number of entries in the factors): 7080060 <br class="">
INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 80
<br class="">
INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 80
<br class="">
INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
<br class="">
INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
<br class="">
INFOG(25) (after factorization: number of pivots modified by static pivoting): 0
<br class="">
INFOG(28) (after factorization: number of null pivots encountered): 0<br class="">
INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 7080060<br class="">
INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 92, 92<br class="">
INFOG(32) (after analysis: type of analysis done): 1<br class="">
INFOG(33) (value used for ICNTL(8)): 7<br class="">
INFOG(34) (exponent of the determinant if determinant is requested): 0<br class="">
INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 7080060<br class="">
INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0
<br class="">
INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0
<br class="">
INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0
<br class="">
INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0
<br class="">
linear system matrix = precond matrix:<br class="">
Mat Object: 1 MPI processes<br class="">
type: seqaij<br class="">
rows=38154, cols=38154<br class="">
total: nonzeros=973446, allocated nonzeros=973446<br class="">
total number of mallocs used during MatSetValues calls=0<br class="">
not using I-node routines<br class="">
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">
[0]PETSC ERROR: <br class="">
[0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery<br class="">
[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank" class="">
https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class="">
[0]PETSC ERROR: Petsc Release Version 3.12.3, Jan, 03, 2020 </span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class=""><br class="">
</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">Thanks,</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">Ashish</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">Scientific Computing Division</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">OnScale</span></font></div>
<div class=""><font size="2" class=""><span style="font-family:arial,sans-serif" class="">CA, USA<br class="">
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote></div>
</div></blockquote></div><br class=""></body></html>