From roland.richter at ntnu.no Tue Jan 5 06:56:51 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 5 Jan 2021 13:56:51 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size Message-ID: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Hei, I would like to scale a given matrix with a fixed scalar value, and therefore would like to use MatScale(). Nevertheless, I observed an interesting behavior depending on the size of the matrix, and currently I am not sure why. When running the attached code, I intend to divide all elements in the matrix by a constant factor of 10. If I have three or fewer rows and 1024 columns, I get the expected result. If I have four or more rows (with the same number of columns), suddenly my scaling factor seems to be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based matrix still behaves as expected. I currently do not understand that behavior, but do not see any problems with the code either. Are there any possible explanations for that behavior? Thank you very much, regards, Roland Richter -------------- next part -------------- A non-text attachment was scrubbed... Name: main_clean.cpp Type: text/x-c++src Size: 4155 bytes Desc: not available URL: From knepley at gmail.com Tue Jan 5 08:21:19 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 09:21:19 -0500 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: On Tue, Jan 5, 2021 at 7:57 AM Roland Richter wrote: > Hei, > > I would like to scale a given matrix with a fixed scalar value, and > therefore would like to use MatScale(). Nevertheless, I observed an > interesting behavior depending on the size of the matrix, and currently > I am not sure why. > > When running the attached code, I intend to divide all elements in the > matrix by a constant factor of 10. If I have three or fewer rows and > 1024 columns, I get the expected result. If I have four or more rows > (with the same number of columns), suddenly my scaling factor seems to > be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based matrix > still behaves as expected. > 1) It looks like you assume the storage in your armadillo matrix is row major. I would be surprised if this was true. 2) I think it is unlikely that there is a problem with MatScale, so I would guess either you have a memory overwrite or are misinterpreting your output. If you send something I can run, I will figure out which it is. Thanks, Matt > I currently do not understand that behavior, but do not see any problems > with the code either. Are there any possible explanations for that > behavior? > > Thank you very much, > > regards, > > Roland Richter > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 5 08:24:45 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 5 Jan 2021 15:24:45 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: Hei, the code I attached to the original mail should work out of the box, but requires armadillo and PETSc to compile/run. Armadillo stores the data in column-major order, and therefore I am transposing the matrices before and after transferring using .st(). Thank you for your help! Regards, Roland Am 05.01.21 um 15:21 schrieb Matthew Knepley: > On Tue, Jan 5, 2021 at 7:57 AM Roland Richter > wrote: > > Hei, > > I would like to scale a given matrix with a fixed scalar value, and > therefore would like to use MatScale(). Nevertheless, I observed an > interesting behavior depending on the size of the matrix, and > currently > I am not sure why. > > When running the attached code, I intend to divide all elements in the > matrix by a constant factor of 10. If I have three or fewer rows and > 1024 columns, I get the expected result. If I have four or more rows > (with the same number of columns), suddenly my scaling factor seems to > be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based > matrix > still behaves as expected. > > > 1) It looks like you assume the storage in your armadillo matrix is > row major. I would be surprised if this was true. > > 2) I think it is unlikely that there is a problem?with MatScale, so I > would guess either you have a memory overwrite > or are misinterpreting your output. If you send something I can run, I > will figure out which it is. > > ? Thanks, > > ? ? ?Matt > ? > > I currently do not understand that behavior, but do not see any > problems > with the code either. Are there any possible explanations for that > behavior? > > Thank you very much, > > regards, > > Roland Richter > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 5 09:24:16 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 5 Jan 2021 16:24:16 +0100 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices Message-ID: Hei, is there a function to square/cube etc. each element of a matrix or vector in PETSc? It is rather useful for things like intensity calculations (I = |E|^2), and should work also for distributed matrices. Thanks! Regards, Roland Richter From pierre.seize at onera.fr Tue Jan 5 09:30:04 2021 From: pierre.seize at onera.fr (Pierre Seize) Date: Tue, 5 Jan 2021 16:30:04 +0100 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: References: Message-ID: <3597a991-3fc9-f59d-b45a-3efc3c17bffa@onera.fr> Hi, For Vec there is https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecSqrtAbs.html and https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPointwiseMult.html, I think it's what you're looking for. I don't know if there is the same for Mat. Pierre On 05/01/21 16:24, Roland Richter wrote: > Hei, > > is there a function to square/cube etc. each element of a matrix or > vector in PETSc? It is rather useful for things like intensity > calculations (I = |E|^2), and should work also for distributed matrices. > > Thanks! > > Regards, > > Roland Richter > From roland.richter at ntnu.no Tue Jan 5 09:33:20 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 5 Jan 2021 16:33:20 +0100 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: <3597a991-3fc9-f59d-b45a-3efc3c17bffa@onera.fr> References: <3597a991-3fc9-f59d-b45a-3efc3c17bffa@onera.fr> Message-ID: <3ca96456-8532-8655-5bdd-8183b5e4b1b5@ntnu.no> Hei, one solution would then be to iterate over all local rows and use VecPointwiseMult (n-1)-times for x^n if I would like to apply that to a matrix? Thanks, regards, Roland Am 05.01.21 um 16:30 schrieb Pierre Seize: > Hi, > > For Vec there is > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecSqrtAbs.html > and > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPointwiseMult.html, > I think it's what you're looking for. I don't know if there is the > same for Mat. > > Pierre > > On 05/01/21 16:24, Roland Richter wrote: >> Hei, >> >> is there a function to square/cube etc. each element of a matrix or >> vector in PETSc? It is rather useful for things like intensity >> calculations (I = |E|^2), and should work also for distributed matrices. >> >> Thanks! >> >> Regards, >> >> Roland Richter >> > From knepley at gmail.com Tue Jan 5 09:34:14 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 10:34:14 -0500 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 10:24 AM Roland Richter wrote: > Hei, > > is there a function to square/cube etc. each element of a matrix or > vector in PETSc? It is rather useful for things like intensity > calculations (I = |E|^2), and should work also for distributed matrices. > We don't have that. We tend to shy away from functions in the interface, since it makes cross-language things hard. However, 1) I would be willing to add functions that you want, like square or cube, if you think it will make things easier for you. 2) We could consider adding Vec/MatPointwiseFunction(A, f) where you apply a function f pointwise to each entry. Thanks, Matt > Thanks! > > Regards, > > Roland Richter > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 5 09:41:31 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 5 Jan 2021 16:41:31 +0100 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: References: Message-ID: Hei, thanks for the offer! Both options would be a possibility. Option one would be more specific, but I could imagine that implementing it using SSE/AVX-functions should be easier for better throughput. Option two would definitely be more flexible, but might loose the advantage of being able to use SSE/AVX functions. Or do you rely on higher-level commands for such constructions? By the way, why would that make cross-language things harder? Thanks, regards, Roland Am 05.01.21 um 16:34 schrieb Matthew Knepley: > On Tue, Jan 5, 2021 at 10:24 AM Roland Richter > wrote: > > Hei, > > is there a function to square/cube etc. each element of a matrix or > vector in PETSc? It is rather useful for things like intensity > calculations (I = |E|^2), and should work also for distributed > matrices. > > > We don't have that. We tend to shy away from functions in the > interface, since it makes cross-language > things hard. However, > > 1) I would be willing to add functions that you want, like square or > cube, if you think it will make things easier?for you. > > 2) We could consider adding Vec/MatPointwiseFunction(A, f) where you > apply a function f pointwise to each entry. > > ? Thanks, > > ? ? ?Matt > ? > > Thanks! > > Regards, > > Roland Richter > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 5 09:42:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 10:42:27 -0500 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 10:41 AM Roland Richter wrote: > Hei, > > thanks for the offer! Both options would be a possibility. Option one > would be more specific, but I could imagine that implementing it using > SSE/AVX-functions should be easier for better throughput. Option two would > definitely be more flexible, but might loose the advantage of being able to > use SSE/AVX functions. Or do you rely on higher-level commands for such > constructions? > > By the way, why would that make cross-language things harder? > > While most foreign function interfaces can handle basic types, there is no standard for function types. Thanks, Matt > Thanks, > > regards, > > Roland > Am 05.01.21 um 16:34 schrieb Matthew Knepley: > > On Tue, Jan 5, 2021 at 10:24 AM Roland Richter > wrote: > >> Hei, >> >> is there a function to square/cube etc. each element of a matrix or >> vector in PETSc? It is rather useful for things like intensity >> calculations (I = |E|^2), and should work also for distributed matrices. >> > > We don't have that. We tend to shy away from functions in the interface, > since it makes cross-language > things hard. However, > > 1) I would be willing to add functions that you want, like square or cube, > if you think it will make things easier for you. > > 2) We could consider adding Vec/MatPointwiseFunction(A, f) where you apply > a function f pointwise to each entry. > > Thanks, > > Matt > > >> Thanks! >> >> Regards, >> >> Roland Richter >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Tue Jan 5 14:13:56 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 5 Jan 2021 20:13:56 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard Message-ID: Dear Petsc Users I wrote a code in 2D solving PF equations for solidification of alloys. I have two variables in the equations 1-non-dimensional concentration (U) 2- order parameter (p). Everything is fine with this code. When I change the code to 3D, the code has convergence problems which I cannot solve it. I know the term which is causing problem is a term in the equation for U: (dp/dt)*(Pxx+Pyy+Pzz) where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, respectively. When I eliminate this term, the results of 2D and 3D are close to each other. But adding that term the time step should be reduced a lot to obtain convergence, otherwise the timestep does not converge. I have no idea how I should solve this problem. I solve the equations considering (I chose based on suggestions of Moose for solving Cahn-hillard) -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type ilu -sub_ksp_type preonly Would you please help me solve this issue? Best, sepideh -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 5 14:53:49 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 15:53:49 -0500 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi wrote: > Dear Petsc Users > I wrote a code in 2D solving PF equations for solidification of alloys. I > have two variables in the equations 1-non-dimensional concentration (U) 2- > order parameter (p). Everything is fine with this code. > When I change the code to 3D, the code has convergence problems which I > cannot solve it. > I know the term which is causing problem is a term in the equation for U: > (dp/dt)*(Pxx+Pyy+Pzz) > where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, > respectively. > > When I eliminate this term, the results of 2D and 3D are close to each > other. But adding that term the time step should be reduced a lot to obtain > convergence, otherwise the timestep does not converge. I have no idea how I > should solve this problem. > I solve the equations considering (I chose based on suggestions of Moose > for solving Cahn-hillard) > -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 > -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type > ilu -sub_ksp_type preonly > > Would you please help me solve this issue? > In order to see the solver you have we need the output of -ts_view and then to see what is happening during the run we need -ts_monitor -snes_monitor -snes_linesearch_monitor -ksp_monitor_true_residual Thanks, Matt > Best, > sepideh > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Tue Jan 5 15:17:48 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 5 Jan 2021 21:17:48 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: , Message-ID: Dear Matt, following is the output for the first 130 timesteps. It continues the run with the same small timestep 0 TS dt 0.0001 time 0. copy! Write output at step= 0! 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 1 SNES Function norm 5.870480277455e+01 0 SNES Function norm 5.870480198099e+01 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 1 SNES Function norm 5.870165343582e+01 1 TS dt 0.0002 time 0.0001 copy! 0 SNES Function norm 5.869324856421e+01 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 1 SNES Function norm 5.869298823911e+01 2 TS dt 0.0004 time 0.0003 copy! 0 SNES Function norm 5.867574766362e+01 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 1 SNES Function norm 5.867531973582e+01 3 TS dt 0.0008 time 0.0007 copy! 0 SNES Function norm 5.864011322220e+01 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 1 SNES Function norm 5.863990526788e+01 4 TS dt 0.0016 time 0.0015 copy! 0 SNES Function norm 5.856903590475e+01 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 1 SNES Function norm 5.856893794226e+01 5 TS dt 0.0032 time 0.0031 copy! 0 SNES Function norm 5.842679564768e+01 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 1 SNES Function norm 5.842675271515e+01 6 TS dt 0.0064 time 0.0063 copy! 0 SNES Function norm 5.814352339598e+01 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 1 SNES Function norm 5.814350805660e+01 7 TS dt 0.0128 time 0.0127 copy! 0 SNES Function norm 5.760173270961e+01 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 1 SNES Function norm 5.760173134258e+01 8 TS dt 0.0256 time 0.0255 copy! 0 SNES Function norm 5.683398372141e+01 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 1 SNES Function norm 5.683398372141e+01 0 SNES Function norm 5.747024606058e+01 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 1 SNES Function norm 5.747023898114e+01 9 TS dt 0.00758395 time 0.0319 copy! 0 SNES Function norm 5.725811890845e+01 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 1 SNES Function norm 5.725811208900e+01 10 TS dt 0.0151679 time 0.039484 copy! 0 SNES Function norm 5.684339053584e+01 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 1 SNES Function norm 5.684338688259e+01 11 TS dt 0.0303358 time 0.0546519 copy! 0 SNES Function norm 5.605303343054e+01 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 1 SNES Function norm 5.605303164316e+01 12 TS dt 0.0606716 time 0.0849877 copy! 0 SNES Function norm 5.463652619423e+01 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 1 SNES Function norm 5.463652534643e+01 13 TS dt 0.121343 time 0.145659 copy! 0 SNES Function norm 5.251966690761e+01 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 1 SNES Function norm 5.251966654744e+01 14 TS dt 0.242686 time 0.267003 copy! 0 SNES Function norm 5.150151326391e+01 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 1 SNES Function norm 5.150151326391e+01 0 SNES Function norm 5.184908859688e+01 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 1 SNES Function norm 5.184908859688e+01 0 SNES Function norm 5.232682115621e+01 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 1 SNES Function norm 5.232682115621e+01 0 SNES Function norm 5.246983753576e+01 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 1 SNES Function norm 5.246983737815e+01 15 TS dt 0.00758395 time 0.270794 copy! 0 SNES Function norm 5.237345394852e+01 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 1 SNES Function norm 5.237345394852e+01 0 SNES Function norm 5.244535357001e+01 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 1 SNES Function norm 5.244535357001e+01 0 SNES Function norm 5.246369218939e+01 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 1 SNES Function norm 5.246369181668e+01 16 TS dt 0.000947994 time 0.271268 copy! 0 SNES Function norm 5.245144714661e+01 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 1 SNES Function norm 5.245144714661e+01 0 SNES Function norm 5.246062448755e+01 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 1 SNES Function norm 5.246062426384e+01 17 TS dt 0.000473997 time 0.271505 copy! 0 SNES Function norm 5.245449911597e+01 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 1 SNES Function norm 5.245449911597e+01 0 SNES Function norm 5.245909110130e+01 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 1 SNES Function norm 5.245909110130e+01 0 SNES Function norm 5.246024056684e+01 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 1 SNES Function norm 5.246024055663e+01 18 TS dt 5.92496e-05 time 0.271535 copy! 0 SNES Function norm 5.245947411014e+01 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 1 SNES Function norm 5.245947411014e+01 0 SNES Function norm 5.246004891756e+01 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 1 SNES Function norm 5.246004890902e+01 19 TS dt 2.96248e-05 time 0.27155 copy! 0 SNES Function norm 5.245966566276e+01 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 1 SNES Function norm 5.245966566276e+01 0 SNES Function norm 5.245995306375e+01 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 1 SNES Function norm 5.245977670896e+01 20 TS dt 1.48124e-05 time 0.271557 copy! 0 SNES Function norm 5.245977453999e+01 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 1 SNES Function norm 5.245977453999e+01 0 SNES Function norm 5.245973715661e+01 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 1 SNES Function norm 5.245972630220e+01 21 TS dt 7.4062e-06 time 0.271561 copy! 0 SNES Function norm 5.245962991861e+01 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 1 SNES Function norm 5.245962991861e+01 0 SNES Function norm 5.245970348930e+01 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 1 SNES Function norm 5.245969886119e+01 22 TS dt 3.7031e-06 time 0.271563 copy! 0 SNES Function norm 5.245964973088e+01 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 1 SNES Function norm 5.245964973088e+01 0 SNES Function norm 5.245968740629e+01 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 1 SNES Function norm 5.245968066039e+01 23 TS dt 1.85155e-06 time 0.271564 copy! 0 SNES Function norm 5.245966664362e+01 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 1 SNES Function norm 5.245966664362e+01 0 SNES Function norm 5.245967459813e+01 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 1 SNES Function norm 5.245967167916e+01 24 TS dt 9.25775e-07 time 0.271564 copy! 0 SNES Function norm 5.245972527444e+01 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 1 SNES Function norm 5.245972527444e+01 0 SNES Function norm 5.245966286541e+01 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 1 SNES Function norm 5.245964735626e+01 25 TS dt 4.62888e-07 time 0.271565 copy! 0 SNES Function norm 5.245965045260e+01 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 1 SNES Function norm 5.245965045260e+01 0 SNES Function norm 5.245964737315e+01 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 1 SNES Function norm 5.245953907407e+01 26 TS dt 2.31444e-07 time 0.271565 copy! 0 SNES Function norm 5.245952669603e+01 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 1 SNES Function norm 5.245952669603e+01 0 SNES Function norm 5.245955619349e+01 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 1 SNES Function norm 5.245916156502e+01 27 TS dt 1.15722e-07 time 0.271565 copy! 0 SNES Function norm 5.245916087517e+01 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 1 SNES Function norm 5.245916087517e+01 0 SNES Function norm 5.245922267220e+01 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 1 SNES Function norm 5.245868047885e+01 28 TS dt 5.7861e-08 time 0.271565 copy! 0 SNES Function norm 5.245971666219e+01 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 1 SNES Function norm 5.245971666219e+01 0 SNES Function norm 5.245866772896e+01 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 1 SNES Function norm 5.245862997528e+01 29 TS dt 2.89305e-08 time 0.271565 copy! 0 SNES Function norm 5.245864268828e+01 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 1 SNES Function norm 5.245864268828e+01 0 SNES Function norm 5.245863432005e+01 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 1 SNES Function norm 5.245668231675e+01 30 TS dt 1.44652e-08 time 0.271565 copy! 0 SNES Function norm 5.245690451230e+01 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 1 SNES Function norm 5.245690451230e+01 0 SNES Function norm 5.245696584491e+01 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 1 SNES Function norm 5.245647736964e+01 31 TS dt 7.23262e-09 time 0.271565 copy! 0 SNES Function norm 5.245669639739e+01 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 1 SNES Function norm 5.245669639739e+01 0 SNES Function norm 5.245652990872e+01 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 1 SNES Function norm 5.245350296758e+01 32 TS dt 3.61631e-09 time 0.271565 copy! 0 SNES Function norm 5.246640623267e+01 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 1 SNES Function norm 5.246640623267e+01 0 SNES Function norm 5.245276947886e+01 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 1 SNES Function norm 5.245276947886e+01 0 SNES Function norm 5.245277054267e+01 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 1 SNES Function norm 5.240788106409e+01 33 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 5.240539051979e+01 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 1 SNES Function norm 5.240539051979e+01 0 SNES Function norm 5.240989970820e+01 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 1 SNES Function norm 5.240989970820e+01 0 SNES Function norm 5.241000025070e+01 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 1 SNES Function norm 5.228672678486e+01 34 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 5.229325617508e+01 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 1 SNES Function norm 5.229325617508e+01 0 SNES Function norm 5.228920001144e+01 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 1 SNES Function norm 5.228920001144e+01 0 SNES Function norm 5.228865204069e+01 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 1 SNES Function norm 5.175201563991e+01 35 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.168638826028e+01 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 1 SNES Function norm 5.168638826028e+01 0 SNES Function norm 5.178402875676e+01 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 1 SNES Function norm 5.178402875676e+01 0 SNES Function norm 5.178806568852e+01 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 1 SNES Function norm 4.925818844220e+01 36 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 5.270383537204e+01 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 1 SNES Function norm 5.270383537204e+01 0 SNES Function norm 4.874520711594e+01 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 1 SNES Function norm 4.874520711594e+01 0 SNES Function norm 4.879679718326e+01 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 1 SNES Function norm 4.872964148564e+01 37 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.866691749862e+01 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 1 SNES Function norm 4.866382095049e+01 38 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.893392986820e+01 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 1 SNES Function norm 4.893392986820e+01 0 SNES Function norm 4.865470718806e+01 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 1 SNES Function norm 4.843333577010e+01 39 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.842612728466e+01 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 1 SNES Function norm 4.842612728466e+01 0 SNES Function norm 4.840969985006e+01 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 1 SNES Function norm 3.917148384566e+01 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 2 SNES Function norm 3.373009701057e-01 40 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 4.835053852293e+01 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 1 SNES Function norm 4.835053852293e+01 0 SNES Function norm 1.287943764600e+01 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 1 SNES Function norm 1.128956057544e+01 41 TS dt 2.75903e-14 time 0.271565 copy! 0 SNES Function norm 1.669600342271e+01 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 1 SNES Function norm 1.472401994522e+01 42 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 5.552587859898e+01 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 1 SNES Function norm 2.907234222906e+01 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 2 SNES Function norm 1.772805631039e-01 43 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 7.725490953363e+00 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 1 SNES Function norm 5.415206018520e+00 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 2 SNES Function norm 2.952018920094e-02 44 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.042546652708e+00 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 1 SNES Function norm 7.553554533053e-01 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 2 SNES Function norm 6.213938456039e-02 45 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.726791649984e+00 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 1 SNES Function norm 1.726791649984e+00 0 SNES Function norm 4.569198217932e-01 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 1 SNES Function norm 5.060166636556e-02 46 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.648499891442e-01 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 1 SNES Function norm 4.303888962028e-01 47 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.205705397633e+00 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 1 SNES Function norm 1.205705397633e+00 0 SNES Function norm 3.168320704593e-01 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 1 SNES Function norm 1.059925414019e-01 48 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.008300299382e+00 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 1 SNES Function norm 5.842489315290e-01 49 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.508490594861e+00 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 1 SNES Function norm 4.745080825960e-01 50 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.014687630993e+00 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 1 SNES Function norm 5.592988640286e-01 51 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.841734166744e-01 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 1 SNES Function norm 9.841734166744e-01 0 SNES Function norm 5.290699913962e-01 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 1 SNES Function norm 4.072990107379e-01 52 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.293320470683e-01 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 1 SNES Function norm 3.293320470683e-01 0 SNES Function norm 6.190385444978e-01 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 1 SNES Function norm 3.612736786105e-01 53 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.245269204940e+00 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 1 SNES Function norm 5.113533173009e-01 54 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.607180712938e+00 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 1 SNES Function norm 1.607180712938e+00 0 SNES Function norm 3.045128849808e-01 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 1 SNES Function norm 2.935346973548e-01 55 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 8.467066865826e-01 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 1 SNES Function norm 5.502623367952e-01 56 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.717470692129e+00 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 1 SNES Function norm 4.045528183667e-01 57 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.544462481147e+00 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 1 SNES Function norm 1.293824512618e+00 58 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.191314292345e+00 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 1 SNES Function norm 2.617175498914e+00 59 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.489780310948e+00 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 1 SNES Function norm 2.949802369100e+00 60 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 4.631512871139e+00 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 1 SNES Function norm 2.195569697365e+00 61 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.662194684551e+00 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 1 SNES Function norm 1.662194684551e+00 0 SNES Function norm 2.329147999080e+00 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 1 SNES Function norm 1.548533538885e-01 62 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.019371357242e+00 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 1 SNES Function norm 2.813130743975e-01 63 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.849189158431e+00 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 1 SNES Function norm 1.849189158431e+00 0 SNES Function norm 1.106437703298e-01 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 1 SNES Function norm 1.071406781366e-01 64 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.036597476193e-01 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 1 SNES Function norm 1.036597476193e-01 0 SNES Function norm 1.081080288554e-01 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 1 SNES Function norm 4.280304180299e-02 65 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 2.901310268658e-02 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 1 SNES Function norm 2.901310268658e-02 0 SNES Function norm 6.115306732380e-02 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 1 SNES Function norm 6.115306732380e-02 0 SNES Function norm 8.316482031597e-02 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 1 SNES Function norm 8.316482031597e-02 0 SNES Function norm 3.114039374287e-01 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 1 SNES Function norm 1.476629082037e-01 66 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.741199964540e-01 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 1 SNES Function norm 2.741199964540e-01 0 SNES Function norm 1.363976538331e+00 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 1 SNES Function norm 3.620782473843e-01 67 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 2.101847656775e+00 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 1 SNES Function norm 5.837297362402e-01 68 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.691478479373e+00 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 1 SNES Function norm 1.646871085300e+00 69 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 4.140741966573e+00 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 1 SNES Function norm 1.278992537889e+00 70 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 4.925810717003e+00 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 1 SNES Function norm 4.757993340113e-01 71 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.014381910385e+00 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 1 SNES Function norm 1.014381910385e+00 0 SNES Function norm 5.247873040702e-01 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 1 SNES Function norm 5.128575534549e-01 72 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.463622536847e+00 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 1 SNES Function norm 1.463622536847e+00 0 SNES Function norm 5.760430307939e-01 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 1 SNES Function norm 4.031598199606e-01 73 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.304273554580e+00 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 1 SNES Function norm 5.901656814576e-01 74 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.806967885919e+00 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 1 SNES Function norm 1.806967885919e+00 0 SNES Function norm 3.525254737467e-01 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 1 SNES Function norm 2.971700641320e-01 75 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 9.276360188622e-01 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 1 SNES Function norm 3.629560323025e-01 76 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.281038880923e+00 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 1 SNES Function norm 1.281038880923e+00 0 SNES Function norm 2.062116742478e-01 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 1 SNES Function norm 4.076142877095e-02 77 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.970990313724e-01 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 1 SNES Function norm 2.970990313724e-01 0 SNES Function norm 3.372906354998e-01 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 1 SNES Function norm 2.538611106909e-01 78 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 6.929161829162e-01 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 1 SNES Function norm 4.533321906044e-01 79 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.430558276957e+00 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 1 SNES Function norm 3.940601459612e-01 80 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 7.524069011779e-01 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 1 SNES Function norm 6.651399131369e-02 81 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.121363306761e-01 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 1 SNES Function norm 9.121363306761e-01 0 SNES Function norm 1.548591006657e-01 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 1 SNES Function norm 1.548591006657e-01 0 SNES Function norm 3.363022877948e-01 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 1 SNES Function norm 2.439869058762e-01 82 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.602729745290e+00 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 1 SNES Function norm 1.007188504688e+00 83 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.461320490774e+00 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 1 SNES Function norm 2.147586456566e+00 84 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 6.341628072414e+00 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 1 SNES Function norm 3.965504408873e-01 85 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.612683816763e-01 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 1 SNES Function norm 9.612683816763e-01 0 SNES Function norm 7.029352448723e-01 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 1 SNES Function norm 2.593382931820e-01 86 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.105068885806e-01 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 1 SNES Function norm 3.105068885806e-01 0 SNES Function norm 4.981482205604e-01 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 1 SNES Function norm 4.981482205604e-01 0 SNES Function norm 7.138905144630e-01 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 1 SNES Function norm 3.696022475452e-01 87 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.789063973037e-01 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 1 SNES Function norm 2.343211520992e-01 88 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.552109100087e+00 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 1 SNES Function norm 2.944262139143e+00 89 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 7.582944894373e+00 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 1 SNES Function norm 2.445215958037e+00 90 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.577254475804e+00 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 1 SNES Function norm 9.478103979365e-02 91 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 3.398926029549e+00 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 1 SNES Function norm 2.854550344474e+00 92 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.846027110554e+00 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 1 SNES Function norm 4.478173098971e+00 93 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 9.479827241861e+00 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 1 SNES Function norm 2.980698125901e+00 94 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.583823756970e+00 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 1 SNES Function norm 1.929142014309e+00 95 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.257265283168e-01 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 1 SNES Function norm 1.257265283168e-01 0 SNES Function norm 2.389423098829e+00 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 1 SNES Function norm 1.440136831177e-01 96 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.101422245445e+00 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 1 SNES Function norm 7.435685625660e-01 97 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 3.062056964383e+00 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 1 SNES Function norm 3.062056964383e+00 0 SNES Function norm 1.640638339300e-01 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 1 SNES Function norm 7.185385907970e-03 98 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.778588005116e-01 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 1 SNES Function norm 1.179095579605e-01 99 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 2.512082795192e-01 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 1 SNES Function norm 2.512082795192e-01 0 SNES Function norm 8.541223283936e-02 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 1 SNES Function norm 6.589891795632e-02 100 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.174757993770e-01 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 1 SNES Function norm 2.174757993770e-01 0 SNES Function norm 3.006903173936e-02 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 1 SNES Function norm 3.006903173936e-02 0 SNES Function norm 5.071683999025e-02 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 1 SNES Function norm 4.513917663662e-02 101 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.261469809763e-01 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 1 SNES Function norm 9.629885791766e-02 102 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.223153986924e-01 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 1 SNES Function norm 1.444530545281e-01 103 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 3.103810286198e-01 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 1 SNES Function norm 3.103810286198e-01 0 SNES Function norm 1.063191577145e-01 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 1 SNES Function norm 1.280948192266e-02 104 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 1.183517461737e-01 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 1 SNES Function norm 3.338348178983e-02 105 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.506755043591e-01 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 1 SNES Function norm 1.506755043591e-01 0 SNES Function norm 1.678694983365e-02 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 1 SNES Function norm 1.678694983365e-02 0 SNES Function norm 2.211256020366e-02 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 1 SNES Function norm 2.211256020366e-02 0 SNES Function norm 6.319453516491e-02 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 1 SNES Function norm 6.319453516491e-02 0 SNES Function norm 1.993150558437e+00 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 1 SNES Function norm 7.832112166701e-01 106 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.629560288909e+00 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 1 SNES Function norm 1.672457508921e+00 107 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.062643571582e+00 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 1 SNES Function norm 9.036991011875e-01 108 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.105674791223e+00 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 1 SNES Function norm 6.003561441737e-01 109 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.053877992744e+00 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 1 SNES Function norm 3.053877992744e+00 0 SNES Function norm 5.829087412676e-01 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 1 SNES Function norm 4.254244459909e-01 110 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.180846267144e+00 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 1 SNES Function norm 3.911236882643e-01 111 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.532064854858e+00 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 1 SNES Function norm 1.532064854858e+00 0 SNES Function norm 1.764940070080e-01 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 1 SNES Function norm 1.706450714666e-01 112 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.407680008950e-01 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 1 SNES Function norm 4.407680008950e-01 0 SNES Function norm 1.064954076440e+00 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 1 SNES Function norm 2.964963371165e-01 113 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.622925709624e-01 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 1 SNES Function norm 8.622925709624e-01 0 SNES Function norm 2.612065167743e+00 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 1 SNES Function norm 1.331790654620e+00 114 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.606851714788e+00 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 1 SNES Function norm 9.013753678258e-01 115 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 3.227741616537e+00 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 1 SNES Function norm 7.734015655943e-01 116 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.393623504956e+00 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 1 SNES Function norm 4.969182786938e-01 117 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.596405006351e+00 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 1 SNES Function norm 2.596405006351e+00 0 SNES Function norm 4.974558747187e-01 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 1 SNES Function norm 2.909443551880e-01 118 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.088860095507e+00 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 1 SNES Function norm 6.203518870415e-01 119 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.828717001803e+00 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 1 SNES Function norm 9.407254842319e-01 120 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 2.970950728036e+00 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 1 SNES Function norm 2.277472194573e+00 121 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 6.655252327413e+00 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 1 SNES Function norm 3.860967926913e+00 122 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.141502067665e+00 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 1 SNES Function norm 6.426598300871e+00 123 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.224562198048e+01 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 1 SNES Function norm 5.283049961509e+00 124 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 5.780124419709e+00 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 1 SNES Function norm 1.227257133866e+00 125 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 5.063237678461e+00 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 1 SNES Function norm 4.907594073483e+00 126 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 8.280289521084e+00 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 1 SNES Function norm 3.515932015486e+00 127 TS dt 1.1301e-10 time 0.271565 copy! 0 SNES Function norm 2.638351398877e+00 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 1 SNES Function norm 6.341356656368e-01 128 TS dt 2.26019e-10 time 0.271565 copy! 0 SNES Function norm 4.327811744723e+00 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 1 SNES Function norm 1.736633786129e-01 129 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 3.450425343262e+00 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 1 SNES Function norm 1.373376592274e+00 130 TS dt 9.04078e-10 time 0.271565 copy! 0 SNES Function norm 4.603810494450e+00 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 1 SNES Function norm 1.142888930566e+00 ________________________________ From: Matthew Knepley Sent: Tuesday, January 5, 2021 2:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi > wrote: Dear Petsc Users I wrote a code in 2D solving PF equations for solidification of alloys. I have two variables in the equations 1-non-dimensional concentration (U) 2- order parameter (p). Everything is fine with this code. When I change the code to 3D, the code has convergence problems which I cannot solve it. I know the term which is causing problem is a term in the equation for U: (dp/dt)*(Pxx+Pyy+Pzz) where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, respectively. When I eliminate this term, the results of 2D and 3D are close to each other. But adding that term the time step should be reduced a lot to obtain convergence, otherwise the timestep does not converge. I have no idea how I should solve this problem. I solve the equations considering (I chose based on suggestions of Moose for solving Cahn-hillard) -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type ilu -sub_ksp_type preonly Would you please help me solve this issue? In order to see the solver you have we need the output of -ts_view and then to see what is happening during the run we need -ts_monitor -snes_monitor -snes_linesearch_monitor -ksp_monitor_true_residual Thanks, Matt Best, sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 5 15:25:33 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 16:25:33 -0500 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi wrote: > Dear Matt, > following is the output for the first 130 timesteps. It continues the run > with the same small timestep > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm > 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 > 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm > 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 > 1 SNES Function norm 5.870480277455e+01 > 0 SNES Function norm 5.870480198099e+01 > 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm > 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm > 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 > You can see here that the linear solver is very bad, which might contribute to Newton stagnating. While you are starting out, I would recommend -pc_type lu Once you have the nonlinear system converging, then you can try different linear solvers to speed things up. Thanks, Matt > 1 SNES Function norm 5.870165343582e+01 > 1 TS dt 0.0002 time 0.0001 > copy! > 0 SNES Function norm 5.869324856421e+01 > 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm > 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm > 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 > 1 SNES Function norm 5.869298823911e+01 > 2 TS dt 0.0004 time 0.0003 > copy! > 0 SNES Function norm 5.867574766362e+01 > 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm > 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm > 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 > 1 SNES Function norm 5.867531973582e+01 > 3 TS dt 0.0008 time 0.0007 > copy! > 0 SNES Function norm 5.864011322220e+01 > 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm > 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm > 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 > 1 SNES Function norm 5.863990526788e+01 > 4 TS dt 0.0016 time 0.0015 > copy! > 0 SNES Function norm 5.856903590475e+01 > 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm > 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm > 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 > 1 SNES Function norm 5.856893794226e+01 > 5 TS dt 0.0032 time 0.0031 > copy! > 0 SNES Function norm 5.842679564768e+01 > 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm > 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm > 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 > 1 SNES Function norm 5.842675271515e+01 > 6 TS dt 0.0064 time 0.0063 > copy! > 0 SNES Function norm 5.814352339598e+01 > 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm > 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm > 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 > 1 SNES Function norm 5.814350805660e+01 > 7 TS dt 0.0128 time 0.0127 > copy! > 0 SNES Function norm 5.760173270961e+01 > 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm > 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm > 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 > 1 SNES Function norm 5.760173134258e+01 > 8 TS dt 0.0256 time 0.0255 > copy! > 0 SNES Function norm 5.683398372141e+01 > 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm > 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm > 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 > 1 SNES Function norm 5.683398372141e+01 > 0 SNES Function norm 5.747024606058e+01 > 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm > 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm > 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 > 1 SNES Function norm 5.747023898114e+01 > 9 TS dt 0.00758395 time 0.0319 > copy! > 0 SNES Function norm 5.725811890845e+01 > 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm > 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm > 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 > 1 SNES Function norm 5.725811208900e+01 > 10 TS dt 0.0151679 time 0.039484 > copy! > 0 SNES Function norm 5.684339053584e+01 > 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm > 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm > 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 > 1 SNES Function norm 5.684338688259e+01 > 11 TS dt 0.0303358 time 0.0546519 > copy! > 0 SNES Function norm 5.605303343054e+01 > 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm > 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm > 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 > 1 SNES Function norm 5.605303164316e+01 > 12 TS dt 0.0606716 time 0.0849877 > copy! > 0 SNES Function norm 5.463652619423e+01 > 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm > 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm > 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 > 1 SNES Function norm 5.463652534643e+01 > 13 TS dt 0.121343 time 0.145659 > copy! > 0 SNES Function norm 5.251966690761e+01 > 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm > 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm > 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 > 1 SNES Function norm 5.251966654744e+01 > 14 TS dt 0.242686 time 0.267003 > copy! > 0 SNES Function norm 5.150151326391e+01 > 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm > 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm > 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 > 1 SNES Function norm 5.150151326391e+01 > 0 SNES Function norm 5.184908859688e+01 > 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm > 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm > 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 > 1 SNES Function norm 5.184908859688e+01 > 0 SNES Function norm 5.232682115621e+01 > 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm > 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm > 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 > 1 SNES Function norm 5.232682115621e+01 > 0 SNES Function norm 5.246983753576e+01 > 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm > 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm > 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 > 1 SNES Function norm 5.246983737815e+01 > 15 TS dt 0.00758395 time 0.270794 > copy! > 0 SNES Function norm 5.237345394852e+01 > 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm > 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm > 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 > 1 SNES Function norm 5.237345394852e+01 > 0 SNES Function norm 5.244535357001e+01 > 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm > 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm > 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 > 1 SNES Function norm 5.244535357001e+01 > 0 SNES Function norm 5.246369218939e+01 > 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm > 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm > 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 > 1 SNES Function norm 5.246369181668e+01 > 16 TS dt 0.000947994 time 0.271268 > copy! > 0 SNES Function norm 5.245144714661e+01 > 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm > 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm > 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 > 1 SNES Function norm 5.245144714661e+01 > 0 SNES Function norm 5.246062448755e+01 > 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm > 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm > 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 > 1 SNES Function norm 5.246062426384e+01 > 17 TS dt 0.000473997 time 0.271505 > copy! > 0 SNES Function norm 5.245449911597e+01 > 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm > 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm > 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 > 1 SNES Function norm 5.245449911597e+01 > 0 SNES Function norm 5.245909110130e+01 > 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm > 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm > 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 > 1 SNES Function norm 5.245909110130e+01 > 0 SNES Function norm 5.246024056684e+01 > 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm > 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm > 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 > 1 SNES Function norm 5.246024055663e+01 > 18 TS dt 5.92496e-05 time 0.271535 > copy! > 0 SNES Function norm 5.245947411014e+01 > 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm > 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm > 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 > 1 SNES Function norm 5.245947411014e+01 > 0 SNES Function norm 5.246004891756e+01 > 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm > 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm > 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 > 1 SNES Function norm 5.246004890902e+01 > 19 TS dt 2.96248e-05 time 0.27155 > copy! > 0 SNES Function norm 5.245966566276e+01 > 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm > 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm > 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 > 1 SNES Function norm 5.245966566276e+01 > 0 SNES Function norm 5.245995306375e+01 > 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm > 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm > 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 > 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm > 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 > 1 SNES Function norm 5.245977670896e+01 > 20 TS dt 1.48124e-05 time 0.271557 > copy! > 0 SNES Function norm 5.245977453999e+01 > 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm > 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm > 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 > 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm > 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 > 1 SNES Function norm 5.245977453999e+01 > 0 SNES Function norm 5.245973715661e+01 > 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm > 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm > 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 > 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm > 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 > 1 SNES Function norm 5.245972630220e+01 > 21 TS dt 7.4062e-06 time 0.271561 > copy! > 0 SNES Function norm 5.245962991861e+01 > 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm > 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm > 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 > 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm > 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 > 1 SNES Function norm 5.245962991861e+01 > 0 SNES Function norm 5.245970348930e+01 > 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm > 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm > 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 > 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm > 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 > 1 SNES Function norm 5.245969886119e+01 > 22 TS dt 3.7031e-06 time 0.271563 > copy! > 0 SNES Function norm 5.245964973088e+01 > 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm > 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm > 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 > 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm > 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 > 1 SNES Function norm 5.245964973088e+01 > 0 SNES Function norm 5.245968740629e+01 > 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm > 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm > 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 > 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm > 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 > 1 SNES Function norm 5.245968066039e+01 > 23 TS dt 1.85155e-06 time 0.271564 > copy! > 0 SNES Function norm 5.245966664362e+01 > 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm > 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm > 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 > 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm > 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 > 1 SNES Function norm 5.245966664362e+01 > 0 SNES Function norm 5.245967459813e+01 > 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm > 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm > 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 > 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm > 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 > 1 SNES Function norm 5.245967167916e+01 > 24 TS dt 9.25775e-07 time 0.271564 > copy! > 0 SNES Function norm 5.245972527444e+01 > 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm > 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm > 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 > 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm > 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 > 1 SNES Function norm 5.245972527444e+01 > 0 SNES Function norm 5.245966286541e+01 > 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm > 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm > 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 > 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm > 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 > 1 SNES Function norm 5.245964735626e+01 > 25 TS dt 4.62888e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245965045260e+01 > 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm > 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm > 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 > 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm > 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 > 1 SNES Function norm 5.245965045260e+01 > 0 SNES Function norm 5.245964737315e+01 > 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm > 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm > 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 > 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm > 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 > 1 SNES Function norm 5.245953907407e+01 > 26 TS dt 2.31444e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245952669603e+01 > 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm > 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm > 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 > 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm > 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 > 1 SNES Function norm 5.245952669603e+01 > 0 SNES Function norm 5.245955619349e+01 > 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm > 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm > 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 > 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm > 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 > 1 SNES Function norm 5.245916156502e+01 > 27 TS dt 1.15722e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245916087517e+01 > 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm > 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm > 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 > 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm > 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 > 1 SNES Function norm 5.245916087517e+01 > 0 SNES Function norm 5.245922267220e+01 > 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm > 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm > 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 > 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm > 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 > 1 SNES Function norm 5.245868047885e+01 > 28 TS dt 5.7861e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245971666219e+01 > 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm > 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm > 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 > 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm > 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 > 1 SNES Function norm 5.245971666219e+01 > 0 SNES Function norm 5.245866772896e+01 > 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm > 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm > 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 > 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm > 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 > 1 SNES Function norm 5.245862997528e+01 > 29 TS dt 2.89305e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245864268828e+01 > 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm > 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm > 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 > 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm > 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 > 1 SNES Function norm 5.245864268828e+01 > 0 SNES Function norm 5.245863432005e+01 > 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm > 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm > 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 > 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm > 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 > 1 SNES Function norm 5.245668231675e+01 > 30 TS dt 1.44652e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245690451230e+01 > 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm > 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm > 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 > 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm > 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 > 1 SNES Function norm 5.245690451230e+01 > 0 SNES Function norm 5.245696584491e+01 > 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm > 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm > 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 > 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm > 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 > 1 SNES Function norm 5.245647736964e+01 > 31 TS dt 7.23262e-09 time 0.271565 > copy! > 0 SNES Function norm 5.245669639739e+01 > 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm > 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm > 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 > 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm > 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 > 1 SNES Function norm 5.245669639739e+01 > 0 SNES Function norm 5.245652990872e+01 > 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm > 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm > 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 > 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm > 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 > 1 SNES Function norm 5.245350296758e+01 > 32 TS dt 3.61631e-09 time 0.271565 > copy! > 0 SNES Function norm 5.246640623267e+01 > 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm > 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm > 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 > 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm > 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 > 1 SNES Function norm 5.246640623267e+01 > 0 SNES Function norm 5.245276947886e+01 > 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm > 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm > 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 > 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm > 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 > 1 SNES Function norm 5.245276947886e+01 > 0 SNES Function norm 5.245277054267e+01 > 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm > 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm > 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 > 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm > 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 > 1 SNES Function norm 5.240788106409e+01 > 33 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 5.240539051979e+01 > 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm > 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm > 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 > 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm > 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 > 1 SNES Function norm 5.240539051979e+01 > 0 SNES Function norm 5.240989970820e+01 > 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm > 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm > 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 > 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm > 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 > 1 SNES Function norm 5.240989970820e+01 > 0 SNES Function norm 5.241000025070e+01 > 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm > 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm > 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 > 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm > 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 > 1 SNES Function norm 5.228672678486e+01 > 34 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 5.229325617508e+01 > 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm > 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm > 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 > 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm > 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 > 1 SNES Function norm 5.229325617508e+01 > 0 SNES Function norm 5.228920001144e+01 > 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm > 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm > 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 > 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm > 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 > 1 SNES Function norm 5.228920001144e+01 > 0 SNES Function norm 5.228865204069e+01 > 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm > 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm > 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 > 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm > 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 > 1 SNES Function norm 5.175201563991e+01 > 35 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.168638826028e+01 > 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm > 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm > 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 > 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm > 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 > 1 SNES Function norm 5.168638826028e+01 > 0 SNES Function norm 5.178402875676e+01 > 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm > 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm > 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 > 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm > 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 > 1 SNES Function norm 5.178402875676e+01 > 0 SNES Function norm 5.178806568852e+01 > 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm > 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm > 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 > 1 SNES Function norm 4.925818844220e+01 > 36 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 5.270383537204e+01 > 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm > 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm > 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 > 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm > 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 > 1 SNES Function norm 5.270383537204e+01 > 0 SNES Function norm 4.874520711594e+01 > 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm > 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm > 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 > 1 SNES Function norm 4.874520711594e+01 > 0 SNES Function norm 4.879679718326e+01 > 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm > 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm > 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 > 1 SNES Function norm 4.872964148564e+01 > 37 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.866691749862e+01 > 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm > 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm > 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 > 1 SNES Function norm 4.866382095049e+01 > 38 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.893392986820e+01 > 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm > 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm > 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 > 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm > 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 > 1 SNES Function norm 4.893392986820e+01 > 0 SNES Function norm 4.865470718806e+01 > 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm > 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm > 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 > 1 SNES Function norm 4.843333577010e+01 > 39 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.842612728466e+01 > 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm > 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm > 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 > 1 SNES Function norm 4.842612728466e+01 > 0 SNES Function norm 4.840969985006e+01 > 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm > 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm > 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 > 1 SNES Function norm 3.917148384566e+01 > 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm > 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm > 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 > 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm > 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 > 2 SNES Function norm 3.373009701057e-01 > 40 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 4.835053852293e+01 > 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm > 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm > 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 > 1 SNES Function norm 4.835053852293e+01 > 0 SNES Function norm 1.287943764600e+01 > 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm > 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm > 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 > 1 SNES Function norm 1.128956057544e+01 > 41 TS dt 2.75903e-14 time 0.271565 > copy! > 0 SNES Function norm 1.669600342271e+01 > 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm > 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm > 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 > 1 SNES Function norm 1.472401994522e+01 > 42 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 5.552587859898e+01 > 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm > 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm > 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 > 1 SNES Function norm 2.907234222906e+01 > 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm > 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm > 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 > 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm > 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 > 2 SNES Function norm 1.772805631039e-01 > 43 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 7.725490953363e+00 > 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm > 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm > 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 > 1 SNES Function norm 5.415206018520e+00 > 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm > 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm > 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 > 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm > 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 > 2 SNES Function norm 2.952018920094e-02 > 44 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.042546652708e+00 > 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm > 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm > 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 > 1 SNES Function norm 7.553554533053e-01 > 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm > 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm > 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 > 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm > 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 > 2 SNES Function norm 6.213938456039e-02 > 45 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.726791649984e+00 > 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm > 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm > 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 > 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm > 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 > 1 SNES Function norm 1.726791649984e+00 > 0 SNES Function norm 4.569198217932e-01 > 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm > 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm > 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 > 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm > 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 > 1 SNES Function norm 5.060166636556e-02 > 46 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.648499891442e-01 > 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm > 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm > 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 > 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm > 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 > 1 SNES Function norm 4.303888962028e-01 > 47 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.205705397633e+00 > 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm > 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm > 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 > 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm > 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 > 1 SNES Function norm 1.205705397633e+00 > 0 SNES Function norm 3.168320704593e-01 > 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm > 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm > 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 > 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm > 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 > 1 SNES Function norm 1.059925414019e-01 > 48 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.008300299382e+00 > 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm > 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm > 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 > 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm > 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 > 1 SNES Function norm 5.842489315290e-01 > 49 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.508490594861e+00 > 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm > 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm > 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 > 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm > 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 > 1 SNES Function norm 4.745080825960e-01 > 50 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.014687630993e+00 > 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm > 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm > 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 > 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm > 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 > 1 SNES Function norm 5.592988640286e-01 > 51 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.841734166744e-01 > 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm > 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm > 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 > 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm > 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 > 1 SNES Function norm 9.841734166744e-01 > 0 SNES Function norm 5.290699913962e-01 > 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm > 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm > 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 > 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm > 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 > 1 SNES Function norm 4.072990107379e-01 > 52 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.293320470683e-01 > 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm > 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm > 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 > 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm > 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 > 1 SNES Function norm 3.293320470683e-01 > 0 SNES Function norm 6.190385444978e-01 > 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm > 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm > 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 > 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm > 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 > 1 SNES Function norm 3.612736786105e-01 > 53 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.245269204940e+00 > 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm > 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm > 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 > 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm > 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 > 1 SNES Function norm 5.113533173009e-01 > 54 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.607180712938e+00 > 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm > 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm > 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 > 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm > 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 > 1 SNES Function norm 1.607180712938e+00 > 0 SNES Function norm 3.045128849808e-01 > 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm > 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm > 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 > 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm > 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 > 1 SNES Function norm 2.935346973548e-01 > 55 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 8.467066865826e-01 > 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm > 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm > 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 > 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm > 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 > 1 SNES Function norm 5.502623367952e-01 > 56 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.717470692129e+00 > 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm > 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm > 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 > 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm > 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 > 1 SNES Function norm 4.045528183667e-01 > 57 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.544462481147e+00 > 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm > 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm > 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 > 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm > 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 > 1 SNES Function norm 1.293824512618e+00 > 58 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.191314292345e+00 > 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm > 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm > 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 > 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm > 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 > 1 SNES Function norm 2.617175498914e+00 > 59 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.489780310948e+00 > 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm > 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm > 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 > 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm > 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 > 1 SNES Function norm 2.949802369100e+00 > 60 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 4.631512871139e+00 > 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm > 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm > 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 > 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm > 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 > 1 SNES Function norm 2.195569697365e+00 > 61 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.662194684551e+00 > 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm > 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm > 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 > 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm > 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 > 1 SNES Function norm 1.662194684551e+00 > 0 SNES Function norm 2.329147999080e+00 > 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm > 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm > 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 > 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm > 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 > 1 SNES Function norm 1.548533538885e-01 > 62 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.019371357242e+00 > 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm > 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm > 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 > 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm > 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 > 1 SNES Function norm 2.813130743975e-01 > 63 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.849189158431e+00 > 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm > 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm > 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 > 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm > 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 > 1 SNES Function norm 1.849189158431e+00 > 0 SNES Function norm 1.106437703298e-01 > 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm > 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm > 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 > 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm > 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 > 1 SNES Function norm 1.071406781366e-01 > 64 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.036597476193e-01 > 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm > 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm > 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 > 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm > 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 > 1 SNES Function norm 1.036597476193e-01 > 0 SNES Function norm 1.081080288554e-01 > 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm > 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm > 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 > 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm > 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 > 1 SNES Function norm 4.280304180299e-02 > 65 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 2.901310268658e-02 > 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm > 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm > 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 > 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm > 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 > 1 SNES Function norm 2.901310268658e-02 > 0 SNES Function norm 6.115306732380e-02 > 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm > 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm > 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 > 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm > 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 > 1 SNES Function norm 6.115306732380e-02 > 0 SNES Function norm 8.316482031597e-02 > 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm > 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm > 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 > 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm > 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 > 1 SNES Function norm 8.316482031597e-02 > 0 SNES Function norm 3.114039374287e-01 > 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm > 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm > 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 > 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm > 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 > 1 SNES Function norm 1.476629082037e-01 > 66 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.741199964540e-01 > 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm > 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm > 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 > 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm > 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 > 1 SNES Function norm 2.741199964540e-01 > 0 SNES Function norm 1.363976538331e+00 > 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm > 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm > 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 > 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm > 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 > 1 SNES Function norm 3.620782473843e-01 > 67 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 2.101847656775e+00 > 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm > 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm > 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 > 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm > 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 > 1 SNES Function norm 5.837297362402e-01 > 68 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.691478479373e+00 > 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm > 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm > 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 > 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm > 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 > 1 SNES Function norm 1.646871085300e+00 > 69 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 4.140741966573e+00 > 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm > 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm > 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 > 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm > 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 > 1 SNES Function norm 1.278992537889e+00 > 70 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 4.925810717003e+00 > 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm > 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm > 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 > 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm > 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 > 1 SNES Function norm 4.757993340113e-01 > 71 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.014381910385e+00 > 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm > 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm > 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 > 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm > 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 > 1 SNES Function norm 1.014381910385e+00 > 0 SNES Function norm 5.247873040702e-01 > 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm > 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm > 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 > 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm > 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 > 1 SNES Function norm 5.128575534549e-01 > 72 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.463622536847e+00 > 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm > 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm > 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 > 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm > 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 > 1 SNES Function norm 1.463622536847e+00 > 0 SNES Function norm 5.760430307939e-01 > 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm > 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm > 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 > 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm > 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 > 1 SNES Function norm 4.031598199606e-01 > 73 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.304273554580e+00 > 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm > 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm > 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 > 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm > 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 > 1 SNES Function norm 5.901656814576e-01 > 74 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.806967885919e+00 > 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm > 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm > 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 > 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm > 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 > 1 SNES Function norm 1.806967885919e+00 > 0 SNES Function norm 3.525254737467e-01 > 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm > 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm > 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 > 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm > 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 > 1 SNES Function norm 2.971700641320e-01 > 75 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 9.276360188622e-01 > 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm > 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm > 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 > 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm > 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 > 1 SNES Function norm 3.629560323025e-01 > 76 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.281038880923e+00 > 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm > 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm > 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 > 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm > 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 > 1 SNES Function norm 1.281038880923e+00 > 0 SNES Function norm 2.062116742478e-01 > 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm > 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm > 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 > 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm > 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 > 1 SNES Function norm 4.076142877095e-02 > 77 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970990313724e-01 > 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm > 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm > 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 > 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm > 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 > 1 SNES Function norm 2.970990313724e-01 > 0 SNES Function norm 3.372906354998e-01 > 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm > 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm > 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 > 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm > 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 > 1 SNES Function norm 2.538611106909e-01 > 78 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 6.929161829162e-01 > 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm > 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm > 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 > 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm > 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 > 1 SNES Function norm 4.533321906044e-01 > 79 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.430558276957e+00 > 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm > 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm > 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 > 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm > 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 > 1 SNES Function norm 3.940601459612e-01 > 80 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 7.524069011779e-01 > 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm > 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm > 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 > 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm > 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 > 1 SNES Function norm 6.651399131369e-02 > 81 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.121363306761e-01 > 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm > 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm > 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 > 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm > 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 > 1 SNES Function norm 9.121363306761e-01 > 0 SNES Function norm 1.548591006657e-01 > 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm > 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm > 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 > 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm > 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 > 1 SNES Function norm 1.548591006657e-01 > 0 SNES Function norm 3.363022877948e-01 > 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm > 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm > 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 > 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm > 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 > 1 SNES Function norm 2.439869058762e-01 > 82 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.602729745290e+00 > 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm > 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm > 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 > 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm > 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 > 1 SNES Function norm 1.007188504688e+00 > 83 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.461320490774e+00 > 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm > 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm > 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 > 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm > 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 > 1 SNES Function norm 2.147586456566e+00 > 84 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 6.341628072414e+00 > 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm > 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm > 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 > 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm > 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 > 1 SNES Function norm 3.965504408873e-01 > 85 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.612683816763e-01 > 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm > 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm > 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 > 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm > 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 > 1 SNES Function norm 9.612683816763e-01 > 0 SNES Function norm 7.029352448723e-01 > 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm > 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm > 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 > 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm > 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 > 1 SNES Function norm 2.593382931820e-01 > 86 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.105068885806e-01 > 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm > 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm > 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 > 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm > 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 > 1 SNES Function norm 3.105068885806e-01 > 0 SNES Function norm 4.981482205604e-01 > 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm > 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm > 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 > 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm > 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 > 1 SNES Function norm 4.981482205604e-01 > 0 SNES Function norm 7.138905144630e-01 > 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm > 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm > 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 > 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm > 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 > 1 SNES Function norm 3.696022475452e-01 > 87 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.789063973037e-01 > 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm > 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm > 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 > 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm > 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 > 1 SNES Function norm 2.343211520992e-01 > 88 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.552109100087e+00 > 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm > 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm > 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 > 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm > 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 > 1 SNES Function norm 2.944262139143e+00 > 89 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 7.582944894373e+00 > 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm > 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm > 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 > 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm > 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 > 1 SNES Function norm 2.445215958037e+00 > 90 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.577254475804e+00 > 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm > 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm > 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 > 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm > 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 > 1 SNES Function norm 9.478103979365e-02 > 91 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 3.398926029549e+00 > 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm > 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm > 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 > 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm > 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 > 1 SNES Function norm 2.854550344474e+00 > 92 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.846027110554e+00 > 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm > 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm > 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 > 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm > 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 > 1 SNES Function norm 4.478173098971e+00 > 93 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 9.479827241861e+00 > 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm > 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm > 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 > 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm > 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 > 1 SNES Function norm 2.980698125901e+00 > 94 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.583823756970e+00 > 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm > 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm > 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 > 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm > 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 > 1 SNES Function norm 1.929142014309e+00 > 95 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.257265283168e-01 > 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm > 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm > 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 > 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm > 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 > 1 SNES Function norm 1.257265283168e-01 > 0 SNES Function norm 2.389423098829e+00 > 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm > 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm > 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 > 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm > 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 > 1 SNES Function norm 1.440136831177e-01 > 96 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.101422245445e+00 > 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm > 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm > 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 > 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm > 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 > 1 SNES Function norm 7.435685625660e-01 > 97 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 3.062056964383e+00 > 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm > 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm > 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 > 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm > 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 > 1 SNES Function norm 3.062056964383e+00 > 0 SNES Function norm 1.640638339300e-01 > 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm > 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm > 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 > 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm > 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 > 1 SNES Function norm 7.185385907970e-03 > 98 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.778588005116e-01 > 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm > 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm > 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 > 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm > 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 > 1 SNES Function norm 1.179095579605e-01 > 99 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 2.512082795192e-01 > 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm > 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm > 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 > 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm > 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 > 1 SNES Function norm 2.512082795192e-01 > 0 SNES Function norm 8.541223283936e-02 > 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm > 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm > 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 > 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm > 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 > 1 SNES Function norm 6.589891795632e-02 > 100 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.174757993770e-01 > 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm > 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm > 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 > 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm > 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 > 1 SNES Function norm 2.174757993770e-01 > 0 SNES Function norm 3.006903173936e-02 > 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm > 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm > 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 > 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm > 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 > 1 SNES Function norm 3.006903173936e-02 > 0 SNES Function norm 5.071683999025e-02 > 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm > 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm > 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 > 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm > 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 > 1 SNES Function norm 4.513917663662e-02 > 101 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.261469809763e-01 > 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm > 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm > 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 > 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm > 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 > 1 SNES Function norm 9.629885791766e-02 > 102 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.223153986924e-01 > 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm > 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm > 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 > 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm > 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 > 1 SNES Function norm 1.444530545281e-01 > 103 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 3.103810286198e-01 > 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm > 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm > 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 > 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm > 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 > 1 SNES Function norm 3.103810286198e-01 > 0 SNES Function norm 1.063191577145e-01 > 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm > 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm > 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 > 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm > 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 > 1 SNES Function norm 1.280948192266e-02 > 104 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 1.183517461737e-01 > 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm > 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm > 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 > 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm > 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 > 1 SNES Function norm 3.338348178983e-02 > 105 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.506755043591e-01 > 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm > 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm > 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 > 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm > 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 > 1 SNES Function norm 1.506755043591e-01 > 0 SNES Function norm 1.678694983365e-02 > 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm > 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm > 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 > 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm > 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 > 1 SNES Function norm 1.678694983365e-02 > 0 SNES Function norm 2.211256020366e-02 > 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm > 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm > 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 > 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm > 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 > 1 SNES Function norm 2.211256020366e-02 > 0 SNES Function norm 6.319453516491e-02 > 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm > 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm > 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 > 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm > 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 > 1 SNES Function norm 6.319453516491e-02 > 0 SNES Function norm 1.993150558437e+00 > 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm > 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm > 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 > 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm > 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 > 1 SNES Function norm 7.832112166701e-01 > 106 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.629560288909e+00 > 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm > 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm > 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 > 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm > 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 > 1 SNES Function norm 1.672457508921e+00 > 107 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.062643571582e+00 > 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm > 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm > 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 > 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm > 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 > 1 SNES Function norm 9.036991011875e-01 > 108 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.105674791223e+00 > 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm > 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm > 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 > 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm > 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 > 1 SNES Function norm 6.003561441737e-01 > 109 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.053877992744e+00 > 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm > 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm > 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 > 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm > 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 > 1 SNES Function norm 3.053877992744e+00 > 0 SNES Function norm 5.829087412676e-01 > 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm > 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm > 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 > 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm > 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 > 1 SNES Function norm 4.254244459909e-01 > 110 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.180846267144e+00 > 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm > 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm > 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 > 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm > 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 > 1 SNES Function norm 3.911236882643e-01 > 111 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.532064854858e+00 > 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm > 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm > 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 > 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm > 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 > 1 SNES Function norm 1.532064854858e+00 > 0 SNES Function norm 1.764940070080e-01 > 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm > 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm > 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 > 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm > 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 > 1 SNES Function norm 1.706450714666e-01 > 112 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.407680008950e-01 > 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm > 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm > 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 > 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm > 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 > 1 SNES Function norm 4.407680008950e-01 > 0 SNES Function norm 1.064954076440e+00 > 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm > 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm > 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 > 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm > 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 > 1 SNES Function norm 2.964963371165e-01 > 113 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.622925709624e-01 > 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm > 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm > 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 > 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm > 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 > 1 SNES Function norm 8.622925709624e-01 > 0 SNES Function norm 2.612065167743e+00 > 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm > 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm > 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 > 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm > 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 > 1 SNES Function norm 1.331790654620e+00 > 114 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.606851714788e+00 > 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm > 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm > 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 > 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm > 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 > 1 SNES Function norm 9.013753678258e-01 > 115 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 3.227741616537e+00 > 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm > 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm > 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 > 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm > 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 > 1 SNES Function norm 7.734015655943e-01 > 116 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.393623504956e+00 > 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm > 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm > 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 > 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm > 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 > 1 SNES Function norm 4.969182786938e-01 > 117 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.596405006351e+00 > 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm > 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm > 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 > 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm > 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 > 1 SNES Function norm 2.596405006351e+00 > 0 SNES Function norm 4.974558747187e-01 > 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm > 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm > 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 > 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm > 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 > 1 SNES Function norm 2.909443551880e-01 > 118 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.088860095507e+00 > 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm > 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm > 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 > 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm > 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 > 1 SNES Function norm 6.203518870415e-01 > 119 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.828717001803e+00 > 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm > 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm > 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 > 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm > 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 > 1 SNES Function norm 9.407254842319e-01 > 120 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970950728036e+00 > 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm > 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm > 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 > 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm > 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 > 1 SNES Function norm 2.277472194573e+00 > 121 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 6.655252327413e+00 > 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm > 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm > 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 > 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm > 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 > 1 SNES Function norm 3.860967926913e+00 > 122 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.141502067665e+00 > 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm > 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm > 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 > 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm > 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 > 1 SNES Function norm 6.426598300871e+00 > 123 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.224562198048e+01 > 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm > 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm > 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 > 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm > 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 > 1 SNES Function norm 5.283049961509e+00 > 124 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 5.780124419709e+00 > 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm > 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm > 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 > 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm > 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 > 1 SNES Function norm 1.227257133866e+00 > 125 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 5.063237678461e+00 > 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm > 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm > 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 > 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm > 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 > 1 SNES Function norm 4.907594073483e+00 > 126 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 8.280289521084e+00 > 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm > 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm > 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 > 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm > 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 > 1 SNES Function norm 3.515932015486e+00 > 127 TS dt 1.1301e-10 time 0.271565 > copy! > 0 SNES Function norm 2.638351398877e+00 > 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm > 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm > 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 > 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm > 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 > 1 SNES Function norm 6.341356656368e-01 > 128 TS dt 2.26019e-10 time 0.271565 > copy! > 0 SNES Function norm 4.327811744723e+00 > 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm > 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm > 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 > 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm > 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 > 1 SNES Function norm 1.736633786129e-01 > 129 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 3.450425343262e+00 > 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm > 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm > 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 > 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm > 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 > 1 SNES Function norm 1.373376592274e+00 > 130 TS dt 9.04078e-10 time 0.271565 > copy! > 0 SNES Function norm 4.603810494450e+00 > 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm > 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm > 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 > 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm > 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 > 1 SNES Function norm 1.142888930566e+00 > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 2:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi wrote: > > Dear Petsc Users > I wrote a code in 2D solving PF equations for solidification of alloys. I > have two variables in the equations 1-non-dimensional concentration (U) 2- > order parameter (p). Everything is fine with this code. > When I change the code to 3D, the code has convergence problems which I > cannot solve it. > I know the term which is causing problem is a term in the equation for U: > (dp/dt)*(Pxx+Pyy+Pzz) > where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, > respectively. > > When I eliminate this term, the results of 2D and 3D are close to each > other. But adding that term the time step should be reduced a lot to obtain > convergence, otherwise the timestep does not converge. I have no idea how I > should solve this problem. > I solve the equations considering (I chose based on suggestions of Moose > for solving Cahn-hillard) > -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 > -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type > ilu -sub_ksp_type preonly > > Would you please help me solve this issue? > > > In order to see the solver you have we need the output of > > -ts_view > > and then to see what is happening during the run we need > > -ts_monitor -snes_monitor -snes_linesearch_monitor > -ksp_monitor_true_residual > > Thanks, > > Matt > > > Best, > sepideh > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Tue Jan 5 15:48:07 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 5 Jan 2021 21:48:07 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: , Message-ID: I have tried different pcs (lu,jacobi,sor,mg,asm,ksp,pbjacobi). they still do not solve the problem. the an example for lu: -pc_type lu: 0 TS dt 0.0001 time 0. copy! Write output at step= 0! 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.761968940423e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.026000796457e-17 true resid norm 1.790914574372e-13 ||r(i)||/||b|| 3.050633943390e-15 1 SNES Function norm 5.808754845048e+01 0 KSP preconditioned resid norm 1.191540511981e+06 true resid norm 5.808754845048e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.114882955273e-10 true resid norm 1.203029448994e-06 ||r(i)||/||b|| 2.071062527315e-08 2 SNES Function norm 5.794558475659e+01 0 KSP preconditioned resid norm 1.191526760369e+06 true resid norm 5.794558475659e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.166698266769e-09 true resid norm 1.059436818305e-06 ||r(i)||/||b|| 1.828330532439e-08 3 SNES Function norm 5.772046032228e+01 0 KSP preconditioned resid norm 1.191498793276e+06 true resid norm 5.772046032228e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.135175980555e-09 true resid norm 1.230437568859e-06 ||r(i)||/||b|| 2.131718219136e-08 4 SNES Function norm 5.750822623807e+01 0 KSP preconditioned resid norm 1.191443279124e+06 true resid norm 5.750822623807e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.905670672745e-10 true resid norm 1.110684454395e-06 ||r(i)||/||b|| 1.931348829639e-08 5 SNES Function norm 5.750395740069e+01 0 KSP preconditioned resid norm 1.191433462635e+06 true resid norm 5.750395740069e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.777205269843e-10 true resid norm 1.430076525401e-06 ||r(i)||/||b|| 2.486918448822e-08 6 SNES Function norm 5.750395740069e+01 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 4.405989777785e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.185260514636e-18 true resid norm 2.397998208837e-14 ||r(i)||/||b|| 4.084736836000e-16 1 SNES Function norm 3.428067045216e+01 0 KSP preconditioned resid norm 1.895265798836e+06 true resid norm 3.428067045216e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.403775884750e-09 true resid norm 8.602063630559e-06 ||r(i)||/||b|| 2.509304373893e-07 2 SNES Function norm 3.428067045216e+01 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.101564184740e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.380439828314e-19 true resid norm 3.289971303359e-14 ||r(i)||/||b|| 5.604118853253e-16 1 SNES Function norm 3.330623063627e+01 0 KSP preconditioned resid norm 6.892935831006e-04 true resid norm 3.330623063627e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.234145873951e-19 true resid norm 1.823438085202e-14 ||r(i)||/||b|| 5.474765683081e-16 2 SNES Function norm 1.228168587373e+01 0 KSP preconditioned resid norm 2.442179927376e-04 true resid norm 1.228168587373e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.114351195683e-19 true resid norm 1.328341389902e-14 ||r(i)||/||b|| 1.081562745993e-15 3 SNES Function norm 8.377002138765e+00 0 KSP preconditioned resid norm 1.670901807768e-04 true resid norm 8.377002138765e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.056553283627e-19 true resid norm 1.539291322344e-14 ||r(i)||/||b|| 1.837520507750e-15 4 SNES Function norm 6.871259668846e-01 0 KSP preconditioned resid norm 7.849703293659e-06 true resid norm 6.871259668846e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.813692242300e-21 true resid norm 3.443983247611e-16 ||r(i)||/||b|| 5.012157033194e-16 5 SNES Function norm 4.598894985660e-01 0 SNES Function norm 4.609452800861e-01 0 KSP preconditioned resid norm 4.482807772478e+00 true resid norm 4.609452800861e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.779043025722e-15 true resid norm 9.363790606967e-11 ||r(i)||/||b|| 2.031432148566e-10 1 SNES Function norm 3.909615632676e-01 1 TS dt 1.25e-05 time 6.25e-06 copy! 0 SNES Function norm 1.083064424363e+00 0 KSP preconditioned resid norm 2.899372173019e+02 true resid norm 1.083064424363e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.843039617595e-13 true resid norm 2.031204561200e-09 ||r(i)||/||b|| 1.875423581006e-09 1 SNES Function norm 1.083064424363e+00 0 SNES Function norm 4.081254017393e-01 0 KSP preconditioned resid norm 1.134610736819e+01 true resid norm 4.081254017393e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.857284280292e-15 true resid norm 1.950178080787e-10 ||r(i)||/||b|| 4.778379567837e-10 1 SNES Function norm 3.852294258555e-01 2 TS dt 6.25e-06 time 9.375e-06 copy! 0 SNES Function norm 3.855090631551e-01 0 KSP preconditioned resid norm 5.807020984426e+01 true resid norm 3.855090631551e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.595611115758e-14 true resid norm 6.458636528724e-10 ||r(i)||/||b|| 1.675352707888e-09 1 SNES Function norm 3.855090631551e-01 0 SNES Function norm 3.894882915800e-01 0 KSP preconditioned resid norm 3.545691207399e+00 true resid norm 3.894882915800e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.936099793899e-15 true resid norm 1.237085076854e-10 ||r(i)||/||b|| 3.176180397711e-10 1 SNES Function norm 3.826734119825e-01 3 TS dt 3.125e-06 time 1.09375e-05 copy! 0 SNES Function norm 3.846635056055e-01 0 KSP preconditioned resid norm 1.270632337662e+01 true resid norm 3.846635056055e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.538849751267e-15 true resid norm 2.462963992263e-10 ||r(i)||/||b|| 6.402905283117e-10 1 SNES Function norm 3.846635056055e-01 0 SNES Function norm 3.838572028974e-01 0 KSP preconditioned resid norm 9.750671428246e-01 true resid norm 3.838572028974e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.289810958767e-15 true resid norm 5.932884665420e-11 ||r(i)||/||b|| 1.545596805436e-10 1 SNES Function norm 3.823876176903e-01 4 TS dt 1.5625e-06 time 1.17188e-05 copy! 0 SNES Function norm 3.842190105197e-01 0 KSP preconditioned resid norm 2.949895830433e+00 true resid norm 3.842190105197e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.544123722663e-15 true resid norm 1.184005133464e-10 ||r(i)||/||b|| 3.081589148499e-10 1 SNES Function norm 3.842190105197e-01 0 SNES Function norm 3.825899662960e-01 0 KSP preconditioned resid norm 2.548468293806e-01 true resid norm 3.825899662960e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.086200228021e-16 true resid norm 2.882610131465e-11 ||r(i)||/||b|| 7.534463486778e-11 1 SNES Function norm 3.825019685537e-01 5 TS dt 7.8125e-07 time 1.21094e-05 copy! 0 SNES Function norm 3.827182409606e-01 0 KSP preconditioned resid norm 7.091231452893e-01 true resid norm 3.827182409606e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.614097858359e-16 true resid norm 5.440482894192e-11 ||r(i)||/||b|| 1.421537390153e-10 1 SNES Function norm 3.827182409606e-01 0 SNES Function norm 3.825683488225e-01 0 KSP preconditioned resid norm 6.509635329694e-02 true resid norm 3.825683488225e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.255654660173e-17 true resid norm 1.410340803479e-11 ||r(i)||/||b|| 3.686506758386e-11 1 SNES Function norm 3.825462954733e-01 6 TS dt 3.90625e-07 time 1.23047e-05 copy! 0 SNES Function norm 3.826632731069e-01 0 KSP preconditioned resid norm 1.737397651071e-01 true resid norm 3.826632731069e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.095925875164e-16 true resid norm 2.780031944787e-11 ||r(i)||/||b|| 7.264956268774e-11 1 SNES Function norm 3.826632731069e-01 0 SNES Function norm 3.825751343810e-01 0 KSP preconditioned resid norm 1.644741829353e-02 true resid norm 3.825751343810e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.211213666062e-18 true resid norm 7.515828526583e-12 ||r(i)||/||b|| 1.964536597169e-11 1 SNES Function norm 3.825729851167e-01 7 TS dt 1.95313e-07 time 1.24023e-05 copy! 0 SNES Function norm 3.826356173729e-01 0 KSP preconditioned resid norm 4.299238585205e-02 true resid norm 3.826356173729e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.260847149806e-17 true resid norm 1.287166330766e-11 ||r(i)||/||b|| 3.363948028683e-11 1 SNES Function norm 3.826356173729e-01 0 SNES Function norm 3.825855754640e-01 0 KSP preconditioned resid norm 4.133474793122e-03 true resid norm 3.825855754640e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.077297138396e-18 true resid norm 3.792689943805e-12 ||r(i)||/||b|| 9.913311392374e-12 1 SNES Function norm 3.825855754640e-01 0 SNES Function norm 3.825755706234e-01 0 KSP preconditioned resid norm 3.705095771277e-04 true resid norm 3.825755706234e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.848267031641e-19 true resid norm 1.079462463835e-12 ||r(i)||/||b|| 2.821566630813e-12 1 SNES Function norm 3.825755706234e-01 0 SNES Function norm 3.825734391910e-01 0 KSP preconditioned resid norm 2.690993850525e-05 true resid norm 3.825734391910e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.585015168707e-21 true resid norm 3.091320932665e-13 ||r(i)||/||b|| 8.080333384362e-13 1 SNES Function norm 3.825734391910e-01 0 SNES Function norm 3.825729114845e-01 0 KSP preconditioned resid norm 1.757556306807e-06 true resid norm 3.825729114845e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.987700736384e-22 true resid norm 8.038113355816e-14 ||r(i)||/||b|| 2.101067042260e-13 1 SNES Function norm 3.825729114845e-01 0 SNES Function norm 3.825565841303e-01 0 KSP preconditioned resid norm 1.103248802240e-07 true resid norm 3.825565841303e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.504795613889e-22 true resid norm 1.879813159274e-14 ||r(i)||/||b|| 4.913817294630e-14 1 SNES Function norm 3.825565841303e-01 0 SNES Function norm 3.825547574266e-01 0 KSP preconditioned resid norm 6.887463339138e-09 true resid norm 3.825547574266e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.559974044704e-23 true resid norm 5.311723680631e-15 ||r(i)||/||b|| 1.388487158378e-14 1 SNES Function norm 3.825547574266e-01 0 SNES Function norm 3.825908198230e-01 0 KSP preconditioned resid norm 4.314157283909e-10 true resid norm 3.825908198230e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.248846716324e-24 true resid norm 1.634240020842e-15 ||r(i)||/||b|| 4.271508714187e-15 1 SNES Function norm 3.825908198230e-01 0 SNES Function norm 3.823046631992e-01 0 KSP preconditioned resid norm 1.477599397951e-11 true resid norm 3.823046631992e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.674742673343e-25 true resid norm 4.342823046268e-15 ||r(i)||/||b|| 1.135958690623e-14 1 SNES Function norm 3.823046631992e-01 0 SNES Function norm 4.050006682433e-01 0 KSP preconditioned resid norm 4.744618612191e-12 true resid norm 4.050006682433e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.185835350170e-26 true resid norm 2.742859197646e-15 ||r(i)||/||b|| 6.772480671558e-15 1 SNES Function norm 4.050006682433e-01 0 SNES Function norm 4.420555752951e-01 0 KSP preconditioned resid norm 2.570862477765e-13 true resid norm 4.420555752951e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.558628106533e-27 true resid norm 2.669513100096e-15 ||r(i)||/||b|| 6.038863096149e-15 1 SNES Function norm 3.053219162874e-01 8 TS dt 3.72529e-13 time 1.24023e-05 copy! 0 SNES Function norm 2.938107171164e-01 0 KSP preconditioned resid norm 1.152686085246e-13 true resid norm 2.938107171164e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.059363368490e-28 true resid norm 1.486663802399e-15 ||r(i)||/||b|| 5.059937285439e-15 1 SNES Function norm 1.879261738365e-01 9 TS dt 7.45058e-13 time 1.24023e-05 copy! 0 SNES Function norm 2.598740119900e-01 0 KSP preconditioned resid norm 4.386389243102e-12 true resid norm 2.598740119900e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.765692021861e-27 true resid norm 4.362311845294e-16 ||r(i)||/||b|| 1.678625658599e-15 1 SNES Function norm 2.598740119900e-01 0 SNES Function norm 2.359036347057e-01 0 KSP preconditioned resid norm 3.416321738725e-13 true resid norm 2.359036347057e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.983917914615e-27 true resid norm 2.031818641817e-15 ||r(i)||/||b|| 8.612917916048e-15 1 SNES Function norm 1.708032078859e-01 10 TS dt 3.72529e-13 time 1.24023e-05 copy! 0 SNES Function norm 5.285427862819e-01 0 KSP preconditioned resid norm 1.592064460142e-12 true resid norm 5.285427862819e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.163339775654e-26 true resid norm 3.880080925123e-15 ||r(i)||/||b|| 7.341091442033e-15 1 SNES Function norm 5.285427862819e-01 0 SNES Function norm 1.646935190988e-01 0 KSP preconditioned resid norm 5.824112147206e-14 true resid norm 1.646935190988e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.820310421468e-29 true resid norm 7.168182199430e-17 ||r(i)||/||b|| 4.352437326407e-16 1 SNES Function norm 1.621698143957e-01 ________________________________ From: Matthew Knepley Sent: Tuesday, January 5, 2021 3:25 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi > wrote: Dear Matt, following is the output for the first 130 timesteps. It continues the run with the same small timestep 0 TS dt 0.0001 time 0. copy! Write output at step= 0! 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 1 SNES Function norm 5.870480277455e+01 0 SNES Function norm 5.870480198099e+01 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 You can see here that the linear solver is very bad, which might contribute to Newton stagnating. While you are starting out, I would recommend -pc_type lu Once you have the nonlinear system converging, then you can try different linear solvers to speed things up. Thanks, Matt 1 SNES Function norm 5.870165343582e+01 1 TS dt 0.0002 time 0.0001 copy! 0 SNES Function norm 5.869324856421e+01 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 1 SNES Function norm 5.869298823911e+01 2 TS dt 0.0004 time 0.0003 copy! 0 SNES Function norm 5.867574766362e+01 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 1 SNES Function norm 5.867531973582e+01 3 TS dt 0.0008 time 0.0007 copy! 0 SNES Function norm 5.864011322220e+01 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 1 SNES Function norm 5.863990526788e+01 4 TS dt 0.0016 time 0.0015 copy! 0 SNES Function norm 5.856903590475e+01 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 1 SNES Function norm 5.856893794226e+01 5 TS dt 0.0032 time 0.0031 copy! 0 SNES Function norm 5.842679564768e+01 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 1 SNES Function norm 5.842675271515e+01 6 TS dt 0.0064 time 0.0063 copy! 0 SNES Function norm 5.814352339598e+01 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 1 SNES Function norm 5.814350805660e+01 7 TS dt 0.0128 time 0.0127 copy! 0 SNES Function norm 5.760173270961e+01 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 1 SNES Function norm 5.760173134258e+01 8 TS dt 0.0256 time 0.0255 copy! 0 SNES Function norm 5.683398372141e+01 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 1 SNES Function norm 5.683398372141e+01 0 SNES Function norm 5.747024606058e+01 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 1 SNES Function norm 5.747023898114e+01 9 TS dt 0.00758395 time 0.0319 copy! 0 SNES Function norm 5.725811890845e+01 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 1 SNES Function norm 5.725811208900e+01 10 TS dt 0.0151679 time 0.039484 copy! 0 SNES Function norm 5.684339053584e+01 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 1 SNES Function norm 5.684338688259e+01 11 TS dt 0.0303358 time 0.0546519 copy! 0 SNES Function norm 5.605303343054e+01 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 1 SNES Function norm 5.605303164316e+01 12 TS dt 0.0606716 time 0.0849877 copy! 0 SNES Function norm 5.463652619423e+01 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 1 SNES Function norm 5.463652534643e+01 13 TS dt 0.121343 time 0.145659 copy! 0 SNES Function norm 5.251966690761e+01 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 1 SNES Function norm 5.251966654744e+01 14 TS dt 0.242686 time 0.267003 copy! 0 SNES Function norm 5.150151326391e+01 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 1 SNES Function norm 5.150151326391e+01 0 SNES Function norm 5.184908859688e+01 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 1 SNES Function norm 5.184908859688e+01 0 SNES Function norm 5.232682115621e+01 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 1 SNES Function norm 5.232682115621e+01 0 SNES Function norm 5.246983753576e+01 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 1 SNES Function norm 5.246983737815e+01 15 TS dt 0.00758395 time 0.270794 copy! 0 SNES Function norm 5.237345394852e+01 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 1 SNES Function norm 5.237345394852e+01 0 SNES Function norm 5.244535357001e+01 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 1 SNES Function norm 5.244535357001e+01 0 SNES Function norm 5.246369218939e+01 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 1 SNES Function norm 5.246369181668e+01 16 TS dt 0.000947994 time 0.271268 copy! 0 SNES Function norm 5.245144714661e+01 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 1 SNES Function norm 5.245144714661e+01 0 SNES Function norm 5.246062448755e+01 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 1 SNES Function norm 5.246062426384e+01 17 TS dt 0.000473997 time 0.271505 copy! 0 SNES Function norm 5.245449911597e+01 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 1 SNES Function norm 5.245449911597e+01 0 SNES Function norm 5.245909110130e+01 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 1 SNES Function norm 5.245909110130e+01 0 SNES Function norm 5.246024056684e+01 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 1 SNES Function norm 5.246024055663e+01 18 TS dt 5.92496e-05 time 0.271535 copy! 0 SNES Function norm 5.245947411014e+01 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 1 SNES Function norm 5.245947411014e+01 0 SNES Function norm 5.246004891756e+01 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 1 SNES Function norm 5.246004890902e+01 19 TS dt 2.96248e-05 time 0.27155 copy! 0 SNES Function norm 5.245966566276e+01 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 1 SNES Function norm 5.245966566276e+01 0 SNES Function norm 5.245995306375e+01 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 1 SNES Function norm 5.245977670896e+01 20 TS dt 1.48124e-05 time 0.271557 copy! 0 SNES Function norm 5.245977453999e+01 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 1 SNES Function norm 5.245977453999e+01 0 SNES Function norm 5.245973715661e+01 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 1 SNES Function norm 5.245972630220e+01 21 TS dt 7.4062e-06 time 0.271561 copy! 0 SNES Function norm 5.245962991861e+01 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 1 SNES Function norm 5.245962991861e+01 0 SNES Function norm 5.245970348930e+01 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 1 SNES Function norm 5.245969886119e+01 22 TS dt 3.7031e-06 time 0.271563 copy! 0 SNES Function norm 5.245964973088e+01 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 1 SNES Function norm 5.245964973088e+01 0 SNES Function norm 5.245968740629e+01 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 1 SNES Function norm 5.245968066039e+01 23 TS dt 1.85155e-06 time 0.271564 copy! 0 SNES Function norm 5.245966664362e+01 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 1 SNES Function norm 5.245966664362e+01 0 SNES Function norm 5.245967459813e+01 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 1 SNES Function norm 5.245967167916e+01 24 TS dt 9.25775e-07 time 0.271564 copy! 0 SNES Function norm 5.245972527444e+01 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 1 SNES Function norm 5.245972527444e+01 0 SNES Function norm 5.245966286541e+01 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 1 SNES Function norm 5.245964735626e+01 25 TS dt 4.62888e-07 time 0.271565 copy! 0 SNES Function norm 5.245965045260e+01 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 1 SNES Function norm 5.245965045260e+01 0 SNES Function norm 5.245964737315e+01 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 1 SNES Function norm 5.245953907407e+01 26 TS dt 2.31444e-07 time 0.271565 copy! 0 SNES Function norm 5.245952669603e+01 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 1 SNES Function norm 5.245952669603e+01 0 SNES Function norm 5.245955619349e+01 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 1 SNES Function norm 5.245916156502e+01 27 TS dt 1.15722e-07 time 0.271565 copy! 0 SNES Function norm 5.245916087517e+01 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 1 SNES Function norm 5.245916087517e+01 0 SNES Function norm 5.245922267220e+01 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 1 SNES Function norm 5.245868047885e+01 28 TS dt 5.7861e-08 time 0.271565 copy! 0 SNES Function norm 5.245971666219e+01 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 1 SNES Function norm 5.245971666219e+01 0 SNES Function norm 5.245866772896e+01 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 1 SNES Function norm 5.245862997528e+01 29 TS dt 2.89305e-08 time 0.271565 copy! 0 SNES Function norm 5.245864268828e+01 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 1 SNES Function norm 5.245864268828e+01 0 SNES Function norm 5.245863432005e+01 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 1 SNES Function norm 5.245668231675e+01 30 TS dt 1.44652e-08 time 0.271565 copy! 0 SNES Function norm 5.245690451230e+01 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 1 SNES Function norm 5.245690451230e+01 0 SNES Function norm 5.245696584491e+01 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 1 SNES Function norm 5.245647736964e+01 31 TS dt 7.23262e-09 time 0.271565 copy! 0 SNES Function norm 5.245669639739e+01 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 1 SNES Function norm 5.245669639739e+01 0 SNES Function norm 5.245652990872e+01 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 1 SNES Function norm 5.245350296758e+01 32 TS dt 3.61631e-09 time 0.271565 copy! 0 SNES Function norm 5.246640623267e+01 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 1 SNES Function norm 5.246640623267e+01 0 SNES Function norm 5.245276947886e+01 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 1 SNES Function norm 5.245276947886e+01 0 SNES Function norm 5.245277054267e+01 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 1 SNES Function norm 5.240788106409e+01 33 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 5.240539051979e+01 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 1 SNES Function norm 5.240539051979e+01 0 SNES Function norm 5.240989970820e+01 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 1 SNES Function norm 5.240989970820e+01 0 SNES Function norm 5.241000025070e+01 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 1 SNES Function norm 5.228672678486e+01 34 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 5.229325617508e+01 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 1 SNES Function norm 5.229325617508e+01 0 SNES Function norm 5.228920001144e+01 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 1 SNES Function norm 5.228920001144e+01 0 SNES Function norm 5.228865204069e+01 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 1 SNES Function norm 5.175201563991e+01 35 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.168638826028e+01 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 1 SNES Function norm 5.168638826028e+01 0 SNES Function norm 5.178402875676e+01 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 1 SNES Function norm 5.178402875676e+01 0 SNES Function norm 5.178806568852e+01 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 1 SNES Function norm 4.925818844220e+01 36 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 5.270383537204e+01 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 1 SNES Function norm 5.270383537204e+01 0 SNES Function norm 4.874520711594e+01 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 1 SNES Function norm 4.874520711594e+01 0 SNES Function norm 4.879679718326e+01 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 1 SNES Function norm 4.872964148564e+01 37 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.866691749862e+01 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 1 SNES Function norm 4.866382095049e+01 38 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.893392986820e+01 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 1 SNES Function norm 4.893392986820e+01 0 SNES Function norm 4.865470718806e+01 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 1 SNES Function norm 4.843333577010e+01 39 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.842612728466e+01 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 1 SNES Function norm 4.842612728466e+01 0 SNES Function norm 4.840969985006e+01 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 1 SNES Function norm 3.917148384566e+01 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 2 SNES Function norm 3.373009701057e-01 40 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 4.835053852293e+01 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 1 SNES Function norm 4.835053852293e+01 0 SNES Function norm 1.287943764600e+01 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 1 SNES Function norm 1.128956057544e+01 41 TS dt 2.75903e-14 time 0.271565 copy! 0 SNES Function norm 1.669600342271e+01 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 1 SNES Function norm 1.472401994522e+01 42 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 5.552587859898e+01 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 1 SNES Function norm 2.907234222906e+01 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 2 SNES Function norm 1.772805631039e-01 43 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 7.725490953363e+00 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 1 SNES Function norm 5.415206018520e+00 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 2 SNES Function norm 2.952018920094e-02 44 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.042546652708e+00 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 1 SNES Function norm 7.553554533053e-01 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 2 SNES Function norm 6.213938456039e-02 45 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.726791649984e+00 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 1 SNES Function norm 1.726791649984e+00 0 SNES Function norm 4.569198217932e-01 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 1 SNES Function norm 5.060166636556e-02 46 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.648499891442e-01 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 1 SNES Function norm 4.303888962028e-01 47 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.205705397633e+00 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 1 SNES Function norm 1.205705397633e+00 0 SNES Function norm 3.168320704593e-01 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 1 SNES Function norm 1.059925414019e-01 48 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.008300299382e+00 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 1 SNES Function norm 5.842489315290e-01 49 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.508490594861e+00 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 1 SNES Function norm 4.745080825960e-01 50 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.014687630993e+00 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 1 SNES Function norm 5.592988640286e-01 51 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.841734166744e-01 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 1 SNES Function norm 9.841734166744e-01 0 SNES Function norm 5.290699913962e-01 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 1 SNES Function norm 4.072990107379e-01 52 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.293320470683e-01 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 1 SNES Function norm 3.293320470683e-01 0 SNES Function norm 6.190385444978e-01 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 1 SNES Function norm 3.612736786105e-01 53 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.245269204940e+00 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 1 SNES Function norm 5.113533173009e-01 54 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.607180712938e+00 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 1 SNES Function norm 1.607180712938e+00 0 SNES Function norm 3.045128849808e-01 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 1 SNES Function norm 2.935346973548e-01 55 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 8.467066865826e-01 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 1 SNES Function norm 5.502623367952e-01 56 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.717470692129e+00 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 1 SNES Function norm 4.045528183667e-01 57 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.544462481147e+00 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 1 SNES Function norm 1.293824512618e+00 58 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.191314292345e+00 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 1 SNES Function norm 2.617175498914e+00 59 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.489780310948e+00 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 1 SNES Function norm 2.949802369100e+00 60 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 4.631512871139e+00 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 1 SNES Function norm 2.195569697365e+00 61 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.662194684551e+00 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 1 SNES Function norm 1.662194684551e+00 0 SNES Function norm 2.329147999080e+00 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 1 SNES Function norm 1.548533538885e-01 62 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.019371357242e+00 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 1 SNES Function norm 2.813130743975e-01 63 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.849189158431e+00 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 1 SNES Function norm 1.849189158431e+00 0 SNES Function norm 1.106437703298e-01 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 1 SNES Function norm 1.071406781366e-01 64 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.036597476193e-01 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 1 SNES Function norm 1.036597476193e-01 0 SNES Function norm 1.081080288554e-01 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 1 SNES Function norm 4.280304180299e-02 65 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 2.901310268658e-02 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 1 SNES Function norm 2.901310268658e-02 0 SNES Function norm 6.115306732380e-02 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 1 SNES Function norm 6.115306732380e-02 0 SNES Function norm 8.316482031597e-02 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 1 SNES Function norm 8.316482031597e-02 0 SNES Function norm 3.114039374287e-01 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 1 SNES Function norm 1.476629082037e-01 66 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.741199964540e-01 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 1 SNES Function norm 2.741199964540e-01 0 SNES Function norm 1.363976538331e+00 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 1 SNES Function norm 3.620782473843e-01 67 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 2.101847656775e+00 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 1 SNES Function norm 5.837297362402e-01 68 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.691478479373e+00 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 1 SNES Function norm 1.646871085300e+00 69 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 4.140741966573e+00 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 1 SNES Function norm 1.278992537889e+00 70 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 4.925810717003e+00 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 1 SNES Function norm 4.757993340113e-01 71 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.014381910385e+00 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 1 SNES Function norm 1.014381910385e+00 0 SNES Function norm 5.247873040702e-01 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 1 SNES Function norm 5.128575534549e-01 72 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.463622536847e+00 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 1 SNES Function norm 1.463622536847e+00 0 SNES Function norm 5.760430307939e-01 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 1 SNES Function norm 4.031598199606e-01 73 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.304273554580e+00 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 1 SNES Function norm 5.901656814576e-01 74 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.806967885919e+00 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 1 SNES Function norm 1.806967885919e+00 0 SNES Function norm 3.525254737467e-01 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 1 SNES Function norm 2.971700641320e-01 75 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 9.276360188622e-01 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 1 SNES Function norm 3.629560323025e-01 76 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.281038880923e+00 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 1 SNES Function norm 1.281038880923e+00 0 SNES Function norm 2.062116742478e-01 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 1 SNES Function norm 4.076142877095e-02 77 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.970990313724e-01 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 1 SNES Function norm 2.970990313724e-01 0 SNES Function norm 3.372906354998e-01 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 1 SNES Function norm 2.538611106909e-01 78 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 6.929161829162e-01 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 1 SNES Function norm 4.533321906044e-01 79 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.430558276957e+00 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 1 SNES Function norm 3.940601459612e-01 80 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 7.524069011779e-01 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 1 SNES Function norm 6.651399131369e-02 81 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.121363306761e-01 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 1 SNES Function norm 9.121363306761e-01 0 SNES Function norm 1.548591006657e-01 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 1 SNES Function norm 1.548591006657e-01 0 SNES Function norm 3.363022877948e-01 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 1 SNES Function norm 2.439869058762e-01 82 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.602729745290e+00 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 1 SNES Function norm 1.007188504688e+00 83 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.461320490774e+00 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 1 SNES Function norm 2.147586456566e+00 84 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 6.341628072414e+00 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 1 SNES Function norm 3.965504408873e-01 85 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.612683816763e-01 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 1 SNES Function norm 9.612683816763e-01 0 SNES Function norm 7.029352448723e-01 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 1 SNES Function norm 2.593382931820e-01 86 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.105068885806e-01 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 1 SNES Function norm 3.105068885806e-01 0 SNES Function norm 4.981482205604e-01 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 1 SNES Function norm 4.981482205604e-01 0 SNES Function norm 7.138905144630e-01 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 1 SNES Function norm 3.696022475452e-01 87 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.789063973037e-01 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 1 SNES Function norm 2.343211520992e-01 88 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.552109100087e+00 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 1 SNES Function norm 2.944262139143e+00 89 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 7.582944894373e+00 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 1 SNES Function norm 2.445215958037e+00 90 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.577254475804e+00 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 1 SNES Function norm 9.478103979365e-02 91 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 3.398926029549e+00 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 1 SNES Function norm 2.854550344474e+00 92 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.846027110554e+00 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 1 SNES Function norm 4.478173098971e+00 93 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 9.479827241861e+00 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 1 SNES Function norm 2.980698125901e+00 94 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.583823756970e+00 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 1 SNES Function norm 1.929142014309e+00 95 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.257265283168e-01 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 1 SNES Function norm 1.257265283168e-01 0 SNES Function norm 2.389423098829e+00 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 1 SNES Function norm 1.440136831177e-01 96 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.101422245445e+00 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 1 SNES Function norm 7.435685625660e-01 97 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 3.062056964383e+00 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 1 SNES Function norm 3.062056964383e+00 0 SNES Function norm 1.640638339300e-01 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 1 SNES Function norm 7.185385907970e-03 98 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.778588005116e-01 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 1 SNES Function norm 1.179095579605e-01 99 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 2.512082795192e-01 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 1 SNES Function norm 2.512082795192e-01 0 SNES Function norm 8.541223283936e-02 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 1 SNES Function norm 6.589891795632e-02 100 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.174757993770e-01 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 1 SNES Function norm 2.174757993770e-01 0 SNES Function norm 3.006903173936e-02 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 1 SNES Function norm 3.006903173936e-02 0 SNES Function norm 5.071683999025e-02 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 1 SNES Function norm 4.513917663662e-02 101 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.261469809763e-01 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 1 SNES Function norm 9.629885791766e-02 102 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.223153986924e-01 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 1 SNES Function norm 1.444530545281e-01 103 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 3.103810286198e-01 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 1 SNES Function norm 3.103810286198e-01 0 SNES Function norm 1.063191577145e-01 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 1 SNES Function norm 1.280948192266e-02 104 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 1.183517461737e-01 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 1 SNES Function norm 3.338348178983e-02 105 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.506755043591e-01 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 1 SNES Function norm 1.506755043591e-01 0 SNES Function norm 1.678694983365e-02 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 1 SNES Function norm 1.678694983365e-02 0 SNES Function norm 2.211256020366e-02 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 1 SNES Function norm 2.211256020366e-02 0 SNES Function norm 6.319453516491e-02 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 1 SNES Function norm 6.319453516491e-02 0 SNES Function norm 1.993150558437e+00 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 1 SNES Function norm 7.832112166701e-01 106 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.629560288909e+00 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 1 SNES Function norm 1.672457508921e+00 107 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.062643571582e+00 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 1 SNES Function norm 9.036991011875e-01 108 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.105674791223e+00 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 1 SNES Function norm 6.003561441737e-01 109 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.053877992744e+00 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 1 SNES Function norm 3.053877992744e+00 0 SNES Function norm 5.829087412676e-01 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 1 SNES Function norm 4.254244459909e-01 110 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.180846267144e+00 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 1 SNES Function norm 3.911236882643e-01 111 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.532064854858e+00 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 1 SNES Function norm 1.532064854858e+00 0 SNES Function norm 1.764940070080e-01 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 1 SNES Function norm 1.706450714666e-01 112 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.407680008950e-01 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 1 SNES Function norm 4.407680008950e-01 0 SNES Function norm 1.064954076440e+00 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 1 SNES Function norm 2.964963371165e-01 113 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.622925709624e-01 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 1 SNES Function norm 8.622925709624e-01 0 SNES Function norm 2.612065167743e+00 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 1 SNES Function norm 1.331790654620e+00 114 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.606851714788e+00 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 1 SNES Function norm 9.013753678258e-01 115 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 3.227741616537e+00 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 1 SNES Function norm 7.734015655943e-01 116 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.393623504956e+00 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 1 SNES Function norm 4.969182786938e-01 117 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.596405006351e+00 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 1 SNES Function norm 2.596405006351e+00 0 SNES Function norm 4.974558747187e-01 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 1 SNES Function norm 2.909443551880e-01 118 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.088860095507e+00 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 1 SNES Function norm 6.203518870415e-01 119 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.828717001803e+00 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 1 SNES Function norm 9.407254842319e-01 120 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 2.970950728036e+00 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 1 SNES Function norm 2.277472194573e+00 121 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 6.655252327413e+00 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 1 SNES Function norm 3.860967926913e+00 122 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.141502067665e+00 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 1 SNES Function norm 6.426598300871e+00 123 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.224562198048e+01 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 1 SNES Function norm 5.283049961509e+00 124 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 5.780124419709e+00 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 1 SNES Function norm 1.227257133866e+00 125 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 5.063237678461e+00 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 1 SNES Function norm 4.907594073483e+00 126 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 8.280289521084e+00 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 1 SNES Function norm 3.515932015486e+00 127 TS dt 1.1301e-10 time 0.271565 copy! 0 SNES Function norm 2.638351398877e+00 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 1 SNES Function norm 6.341356656368e-01 128 TS dt 2.26019e-10 time 0.271565 copy! 0 SNES Function norm 4.327811744723e+00 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 1 SNES Function norm 1.736633786129e-01 129 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 3.450425343262e+00 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 1 SNES Function norm 1.373376592274e+00 130 TS dt 9.04078e-10 time 0.271565 copy! 0 SNES Function norm 4.603810494450e+00 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 1 SNES Function norm 1.142888930566e+00 ________________________________ From: Matthew Knepley > Sent: Tuesday, January 5, 2021 2:53 PM To: Sepideh Kavousi > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi > wrote: Dear Petsc Users I wrote a code in 2D solving PF equations for solidification of alloys. I have two variables in the equations 1-non-dimensional concentration (U) 2- order parameter (p). Everything is fine with this code. When I change the code to 3D, the code has convergence problems which I cannot solve it. I know the term which is causing problem is a term in the equation for U: (dp/dt)*(Pxx+Pyy+Pzz) where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, respectively. When I eliminate this term, the results of 2D and 3D are close to each other. But adding that term the time step should be reduced a lot to obtain convergence, otherwise the timestep does not converge. I have no idea how I should solve this problem. I solve the equations considering (I chose based on suggestions of Moose for solving Cahn-hillard) -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type ilu -sub_ksp_type preonly Would you please help me solve this issue? In order to see the solver you have we need the output of -ts_view and then to see what is happening during the run we need -ts_monitor -snes_monitor -snes_linesearch_monitor -ksp_monitor_true_residual Thanks, Matt Best, sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 5 15:53:50 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 16:53:50 -0500 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 4:48 PM Sepideh Kavousi wrote: > I have tried different pcs (lu,jacobi,sor,mg,asm,ksp,pbjacobi). they still > do not solve the problem. the an example for lu: > -pc_type lu: > > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.761968940423e-02 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.026000796457e-17 true resid norm > 1.790914574372e-13 ||r(i)||/||b|| 3.050633943390e-15 > 1 SNES Function norm 5.808754845048e+01 > So with a perfect linear solver, Newton is making no progress. That means that your initial guess is very far from the solution, but it could also mean that your Jacobian does not match your residual, since it is not generating descent directions. Have you checked that it matches the finite difference approximation? You can do this using -snes_test_jacobian ( https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESComputeJacobian.html ) Thanks, Matt > 0 KSP preconditioned resid norm 1.191540511981e+06 true resid norm > 5.808754845048e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.114882955273e-10 true resid norm > 1.203029448994e-06 ||r(i)||/||b|| 2.071062527315e-08 > 2 SNES Function norm 5.794558475659e+01 > 0 KSP preconditioned resid norm 1.191526760369e+06 true resid norm > 5.794558475659e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.166698266769e-09 true resid norm > 1.059436818305e-06 ||r(i)||/||b|| 1.828330532439e-08 > 3 SNES Function norm 5.772046032228e+01 > 0 KSP preconditioned resid norm 1.191498793276e+06 true resid norm > 5.772046032228e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.135175980555e-09 true resid norm > 1.230437568859e-06 ||r(i)||/||b|| 2.131718219136e-08 > 4 SNES Function norm 5.750822623807e+01 > 0 KSP preconditioned resid norm 1.191443279124e+06 true resid norm > 5.750822623807e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.905670672745e-10 true resid norm > 1.110684454395e-06 ||r(i)||/||b|| 1.931348829639e-08 > 5 SNES Function norm 5.750395740069e+01 > 0 KSP preconditioned resid norm 1.191433462635e+06 true resid norm > 5.750395740069e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.777205269843e-10 true resid norm > 1.430076525401e-06 ||r(i)||/||b|| 2.486918448822e-08 > 6 SNES Function norm 5.750395740069e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 4.405989777785e-03 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.185260514636e-18 true resid norm > 2.397998208837e-14 ||r(i)||/||b|| 4.084736836000e-16 > 1 SNES Function norm 3.428067045216e+01 > 0 KSP preconditioned resid norm 1.895265798836e+06 true resid norm > 3.428067045216e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.403775884750e-09 true resid norm > 8.602063630559e-06 ||r(i)||/||b|| 2.509304373893e-07 > 2 SNES Function norm 3.428067045216e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.101564184740e-03 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.380439828314e-19 true resid norm > 3.289971303359e-14 ||r(i)||/||b|| 5.604118853253e-16 > 1 SNES Function norm 3.330623063627e+01 > 0 KSP preconditioned resid norm 6.892935831006e-04 true resid norm > 3.330623063627e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.234145873951e-19 true resid norm > 1.823438085202e-14 ||r(i)||/||b|| 5.474765683081e-16 > 2 SNES Function norm 1.228168587373e+01 > 0 KSP preconditioned resid norm 2.442179927376e-04 true resid norm > 1.228168587373e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.114351195683e-19 true resid norm > 1.328341389902e-14 ||r(i)||/||b|| 1.081562745993e-15 > 3 SNES Function norm 8.377002138765e+00 > 0 KSP preconditioned resid norm 1.670901807768e-04 true resid norm > 8.377002138765e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.056553283627e-19 true resid norm > 1.539291322344e-14 ||r(i)||/||b|| 1.837520507750e-15 > 4 SNES Function norm 6.871259668846e-01 > 0 KSP preconditioned resid norm 7.849703293659e-06 true resid norm > 6.871259668846e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.813692242300e-21 true resid norm > 3.443983247611e-16 ||r(i)||/||b|| 5.012157033194e-16 > 5 SNES Function norm 4.598894985660e-01 > 0 SNES Function norm 4.609452800861e-01 > 0 KSP preconditioned resid norm 4.482807772478e+00 true resid norm > 4.609452800861e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.779043025722e-15 true resid norm > 9.363790606967e-11 ||r(i)||/||b|| 2.031432148566e-10 > 1 SNES Function norm 3.909615632676e-01 > 1 TS dt 1.25e-05 time 6.25e-06 > copy! > 0 SNES Function norm 1.083064424363e+00 > 0 KSP preconditioned resid norm 2.899372173019e+02 true resid norm > 1.083064424363e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.843039617595e-13 true resid norm > 2.031204561200e-09 ||r(i)||/||b|| 1.875423581006e-09 > 1 SNES Function norm 1.083064424363e+00 > 0 SNES Function norm 4.081254017393e-01 > 0 KSP preconditioned resid norm 1.134610736819e+01 true resid norm > 4.081254017393e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.857284280292e-15 true resid norm > 1.950178080787e-10 ||r(i)||/||b|| 4.778379567837e-10 > 1 SNES Function norm 3.852294258555e-01 > 2 TS dt 6.25e-06 time 9.375e-06 > copy! > 0 SNES Function norm 3.855090631551e-01 > 0 KSP preconditioned resid norm 5.807020984426e+01 true resid norm > 3.855090631551e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.595611115758e-14 true resid norm > 6.458636528724e-10 ||r(i)||/||b|| 1.675352707888e-09 > 1 SNES Function norm 3.855090631551e-01 > 0 SNES Function norm 3.894882915800e-01 > 0 KSP preconditioned resid norm 3.545691207399e+00 true resid norm > 3.894882915800e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.936099793899e-15 true resid norm > 1.237085076854e-10 ||r(i)||/||b|| 3.176180397711e-10 > 1 SNES Function norm 3.826734119825e-01 > 3 TS dt 3.125e-06 time 1.09375e-05 > copy! > 0 SNES Function norm 3.846635056055e-01 > 0 KSP preconditioned resid norm 1.270632337662e+01 true resid norm > 3.846635056055e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.538849751267e-15 true resid norm > 2.462963992263e-10 ||r(i)||/||b|| 6.402905283117e-10 > 1 SNES Function norm 3.846635056055e-01 > 0 SNES Function norm 3.838572028974e-01 > 0 KSP preconditioned resid norm 9.750671428246e-01 true resid norm > 3.838572028974e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.289810958767e-15 true resid norm > 5.932884665420e-11 ||r(i)||/||b|| 1.545596805436e-10 > 1 SNES Function norm 3.823876176903e-01 > 4 TS dt 1.5625e-06 time 1.17188e-05 > copy! > 0 SNES Function norm 3.842190105197e-01 > 0 KSP preconditioned resid norm 2.949895830433e+00 true resid norm > 3.842190105197e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.544123722663e-15 true resid norm > 1.184005133464e-10 ||r(i)||/||b|| 3.081589148499e-10 > 1 SNES Function norm 3.842190105197e-01 > 0 SNES Function norm 3.825899662960e-01 > 0 KSP preconditioned resid norm 2.548468293806e-01 true resid norm > 3.825899662960e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.086200228021e-16 true resid norm > 2.882610131465e-11 ||r(i)||/||b|| 7.534463486778e-11 > 1 SNES Function norm 3.825019685537e-01 > 5 TS dt 7.8125e-07 time 1.21094e-05 > copy! > 0 SNES Function norm 3.827182409606e-01 > 0 KSP preconditioned resid norm 7.091231452893e-01 true resid norm > 3.827182409606e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.614097858359e-16 true resid norm > 5.440482894192e-11 ||r(i)||/||b|| 1.421537390153e-10 > 1 SNES Function norm 3.827182409606e-01 > 0 SNES Function norm 3.825683488225e-01 > 0 KSP preconditioned resid norm 6.509635329694e-02 true resid norm > 3.825683488225e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.255654660173e-17 true resid norm > 1.410340803479e-11 ||r(i)||/||b|| 3.686506758386e-11 > 1 SNES Function norm 3.825462954733e-01 > 6 TS dt 3.90625e-07 time 1.23047e-05 > copy! > 0 SNES Function norm 3.826632731069e-01 > 0 KSP preconditioned resid norm 1.737397651071e-01 true resid norm > 3.826632731069e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.095925875164e-16 true resid norm > 2.780031944787e-11 ||r(i)||/||b|| 7.264956268774e-11 > 1 SNES Function norm 3.826632731069e-01 > 0 SNES Function norm 3.825751343810e-01 > 0 KSP preconditioned resid norm 1.644741829353e-02 true resid norm > 3.825751343810e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.211213666062e-18 true resid norm > 7.515828526583e-12 ||r(i)||/||b|| 1.964536597169e-11 > 1 SNES Function norm 3.825729851167e-01 > 7 TS dt 1.95313e-07 time 1.24023e-05 > copy! > 0 SNES Function norm 3.826356173729e-01 > 0 KSP preconditioned resid norm 4.299238585205e-02 true resid norm > 3.826356173729e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.260847149806e-17 true resid norm > 1.287166330766e-11 ||r(i)||/||b|| 3.363948028683e-11 > 1 SNES Function norm 3.826356173729e-01 > 0 SNES Function norm 3.825855754640e-01 > 0 KSP preconditioned resid norm 4.133474793122e-03 true resid norm > 3.825855754640e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.077297138396e-18 true resid norm > 3.792689943805e-12 ||r(i)||/||b|| 9.913311392374e-12 > 1 SNES Function norm 3.825855754640e-01 > 0 SNES Function norm 3.825755706234e-01 > 0 KSP preconditioned resid norm 3.705095771277e-04 true resid norm > 3.825755706234e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.848267031641e-19 true resid norm > 1.079462463835e-12 ||r(i)||/||b|| 2.821566630813e-12 > 1 SNES Function norm 3.825755706234e-01 > 0 SNES Function norm 3.825734391910e-01 > 0 KSP preconditioned resid norm 2.690993850525e-05 true resid norm > 3.825734391910e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.585015168707e-21 true resid norm > 3.091320932665e-13 ||r(i)||/||b|| 8.080333384362e-13 > 1 SNES Function norm 3.825734391910e-01 > 0 SNES Function norm 3.825729114845e-01 > 0 KSP preconditioned resid norm 1.757556306807e-06 true resid norm > 3.825729114845e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.987700736384e-22 true resid norm > 8.038113355816e-14 ||r(i)||/||b|| 2.101067042260e-13 > 1 SNES Function norm 3.825729114845e-01 > 0 SNES Function norm 3.825565841303e-01 > 0 KSP preconditioned resid norm 1.103248802240e-07 true resid norm > 3.825565841303e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.504795613889e-22 true resid norm > 1.879813159274e-14 ||r(i)||/||b|| 4.913817294630e-14 > 1 SNES Function norm 3.825565841303e-01 > 0 SNES Function norm 3.825547574266e-01 > 0 KSP preconditioned resid norm 6.887463339138e-09 true resid norm > 3.825547574266e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.559974044704e-23 true resid norm > 5.311723680631e-15 ||r(i)||/||b|| 1.388487158378e-14 > 1 SNES Function norm 3.825547574266e-01 > 0 SNES Function norm 3.825908198230e-01 > 0 KSP preconditioned resid norm 4.314157283909e-10 true resid norm > 3.825908198230e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.248846716324e-24 true resid norm > 1.634240020842e-15 ||r(i)||/||b|| 4.271508714187e-15 > 1 SNES Function norm 3.825908198230e-01 > 0 SNES Function norm 3.823046631992e-01 > 0 KSP preconditioned resid norm 1.477599397951e-11 true resid norm > 3.823046631992e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.674742673343e-25 true resid norm > 4.342823046268e-15 ||r(i)||/||b|| 1.135958690623e-14 > 1 SNES Function norm 3.823046631992e-01 > 0 SNES Function norm 4.050006682433e-01 > 0 KSP preconditioned resid norm 4.744618612191e-12 true resid norm > 4.050006682433e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.185835350170e-26 true resid norm > 2.742859197646e-15 ||r(i)||/||b|| 6.772480671558e-15 > 1 SNES Function norm 4.050006682433e-01 > 0 SNES Function norm 4.420555752951e-01 > 0 KSP preconditioned resid norm 2.570862477765e-13 true resid norm > 4.420555752951e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.558628106533e-27 true resid norm > 2.669513100096e-15 ||r(i)||/||b|| 6.038863096149e-15 > 1 SNES Function norm 3.053219162874e-01 > 8 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.938107171164e-01 > 0 KSP preconditioned resid norm 1.152686085246e-13 true resid norm > 2.938107171164e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.059363368490e-28 true resid norm > 1.486663802399e-15 ||r(i)||/||b|| 5.059937285439e-15 > 1 SNES Function norm 1.879261738365e-01 > 9 TS dt 7.45058e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.598740119900e-01 > 0 KSP preconditioned resid norm 4.386389243102e-12 true resid norm > 2.598740119900e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.765692021861e-27 true resid norm > 4.362311845294e-16 ||r(i)||/||b|| 1.678625658599e-15 > 1 SNES Function norm 2.598740119900e-01 > 0 SNES Function norm 2.359036347057e-01 > 0 KSP preconditioned resid norm 3.416321738725e-13 true resid norm > 2.359036347057e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.983917914615e-27 true resid norm > 2.031818641817e-15 ||r(i)||/||b|| 8.612917916048e-15 > 1 SNES Function norm 1.708032078859e-01 > 10 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 5.285427862819e-01 > 0 KSP preconditioned resid norm 1.592064460142e-12 true resid norm > 5.285427862819e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.163339775654e-26 true resid norm > 3.880080925123e-15 ||r(i)||/||b|| 7.341091442033e-15 > 1 SNES Function norm 5.285427862819e-01 > 0 SNES Function norm 1.646935190988e-01 > 0 KSP preconditioned resid norm 5.824112147206e-14 true resid norm > 1.646935190988e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.820310421468e-29 true resid norm > 7.168182199430e-17 ||r(i)||/||b|| 4.352437326407e-16 > 1 SNES Function norm 1.621698143957e-01 > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 3:25 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi wrote: > > Dear Matt, > following is the output for the first 130 timesteps. It continues the run > with the same small timestep > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm > 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 > 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm > 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 > 1 SNES Function norm 5.870480277455e+01 > 0 SNES Function norm 5.870480198099e+01 > 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm > 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm > 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 > > > You can see here that the linear solver is very bad, which might > contribute to Newton stagnating. While you are starting out, I would > recommend > > -pc_type lu > > Once you have the nonlinear system converging, then you can try different > linear solvers to speed things up. > > Thanks, > > Matt > > > 1 SNES Function norm 5.870165343582e+01 > 1 TS dt 0.0002 time 0.0001 > copy! > 0 SNES Function norm 5.869324856421e+01 > 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm > 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm > 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 > 1 SNES Function norm 5.869298823911e+01 > 2 TS dt 0.0004 time 0.0003 > copy! > 0 SNES Function norm 5.867574766362e+01 > 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm > 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm > 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 > 1 SNES Function norm 5.867531973582e+01 > 3 TS dt 0.0008 time 0.0007 > copy! > 0 SNES Function norm 5.864011322220e+01 > 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm > 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm > 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 > 1 SNES Function norm 5.863990526788e+01 > 4 TS dt 0.0016 time 0.0015 > copy! > 0 SNES Function norm 5.856903590475e+01 > 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm > 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm > 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 > 1 SNES Function norm 5.856893794226e+01 > 5 TS dt 0.0032 time 0.0031 > copy! > 0 SNES Function norm 5.842679564768e+01 > 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm > 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm > 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 > 1 SNES Function norm 5.842675271515e+01 > 6 TS dt 0.0064 time 0.0063 > copy! > 0 SNES Function norm 5.814352339598e+01 > 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm > 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm > 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 > 1 SNES Function norm 5.814350805660e+01 > 7 TS dt 0.0128 time 0.0127 > copy! > 0 SNES Function norm 5.760173270961e+01 > 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm > 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm > 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 > 1 SNES Function norm 5.760173134258e+01 > 8 TS dt 0.0256 time 0.0255 > copy! > 0 SNES Function norm 5.683398372141e+01 > 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm > 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm > 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 > 1 SNES Function norm 5.683398372141e+01 > 0 SNES Function norm 5.747024606058e+01 > 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm > 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm > 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 > 1 SNES Function norm 5.747023898114e+01 > 9 TS dt 0.00758395 time 0.0319 > copy! > 0 SNES Function norm 5.725811890845e+01 > 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm > 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm > 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 > 1 SNES Function norm 5.725811208900e+01 > 10 TS dt 0.0151679 time 0.039484 > copy! > 0 SNES Function norm 5.684339053584e+01 > 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm > 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm > 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 > 1 SNES Function norm 5.684338688259e+01 > 11 TS dt 0.0303358 time 0.0546519 > copy! > 0 SNES Function norm 5.605303343054e+01 > 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm > 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm > 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 > 1 SNES Function norm 5.605303164316e+01 > 12 TS dt 0.0606716 time 0.0849877 > copy! > 0 SNES Function norm 5.463652619423e+01 > 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm > 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm > 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 > 1 SNES Function norm 5.463652534643e+01 > 13 TS dt 0.121343 time 0.145659 > copy! > 0 SNES Function norm 5.251966690761e+01 > 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm > 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm > 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 > 1 SNES Function norm 5.251966654744e+01 > 14 TS dt 0.242686 time 0.267003 > copy! > 0 SNES Function norm 5.150151326391e+01 > 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm > 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm > 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 > 1 SNES Function norm 5.150151326391e+01 > 0 SNES Function norm 5.184908859688e+01 > 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm > 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm > 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 > 1 SNES Function norm 5.184908859688e+01 > 0 SNES Function norm 5.232682115621e+01 > 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm > 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm > 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 > 1 SNES Function norm 5.232682115621e+01 > 0 SNES Function norm 5.246983753576e+01 > 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm > 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm > 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 > 1 SNES Function norm 5.246983737815e+01 > 15 TS dt 0.00758395 time 0.270794 > copy! > 0 SNES Function norm 5.237345394852e+01 > 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm > 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm > 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 > 1 SNES Function norm 5.237345394852e+01 > 0 SNES Function norm 5.244535357001e+01 > 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm > 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm > 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 > 1 SNES Function norm 5.244535357001e+01 > 0 SNES Function norm 5.246369218939e+01 > 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm > 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm > 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 > 1 SNES Function norm 5.246369181668e+01 > 16 TS dt 0.000947994 time 0.271268 > copy! > 0 SNES Function norm 5.245144714661e+01 > 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm > 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm > 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 > 1 SNES Function norm 5.245144714661e+01 > 0 SNES Function norm 5.246062448755e+01 > 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm > 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm > 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 > 1 SNES Function norm 5.246062426384e+01 > 17 TS dt 0.000473997 time 0.271505 > copy! > 0 SNES Function norm 5.245449911597e+01 > 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm > 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm > 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 > 1 SNES Function norm 5.245449911597e+01 > 0 SNES Function norm 5.245909110130e+01 > 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm > 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm > 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 > 1 SNES Function norm 5.245909110130e+01 > 0 SNES Function norm 5.246024056684e+01 > 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm > 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm > 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 > 1 SNES Function norm 5.246024055663e+01 > 18 TS dt 5.92496e-05 time 0.271535 > copy! > 0 SNES Function norm 5.245947411014e+01 > 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm > 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm > 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 > 1 SNES Function norm 5.245947411014e+01 > 0 SNES Function norm 5.246004891756e+01 > 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm > 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm > 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 > 1 SNES Function norm 5.246004890902e+01 > 19 TS dt 2.96248e-05 time 0.27155 > copy! > 0 SNES Function norm 5.245966566276e+01 > 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm > 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm > 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 > 1 SNES Function norm 5.245966566276e+01 > 0 SNES Function norm 5.245995306375e+01 > 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm > 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm > 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 > 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm > 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 > 1 SNES Function norm 5.245977670896e+01 > 20 TS dt 1.48124e-05 time 0.271557 > copy! > 0 SNES Function norm 5.245977453999e+01 > 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm > 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm > 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 > 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm > 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 > 1 SNES Function norm 5.245977453999e+01 > 0 SNES Function norm 5.245973715661e+01 > 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm > 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm > 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 > 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm > 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 > 1 SNES Function norm 5.245972630220e+01 > 21 TS dt 7.4062e-06 time 0.271561 > copy! > 0 SNES Function norm 5.245962991861e+01 > 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm > 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm > 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 > 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm > 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 > 1 SNES Function norm 5.245962991861e+01 > 0 SNES Function norm 5.245970348930e+01 > 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm > 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm > 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 > 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm > 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 > 1 SNES Function norm 5.245969886119e+01 > 22 TS dt 3.7031e-06 time 0.271563 > copy! > 0 SNES Function norm 5.245964973088e+01 > 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm > 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm > 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 > 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm > 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 > 1 SNES Function norm 5.245964973088e+01 > 0 SNES Function norm 5.245968740629e+01 > 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm > 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm > 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 > 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm > 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 > 1 SNES Function norm 5.245968066039e+01 > 23 TS dt 1.85155e-06 time 0.271564 > copy! > 0 SNES Function norm 5.245966664362e+01 > 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm > 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm > 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 > 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm > 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 > 1 SNES Function norm 5.245966664362e+01 > 0 SNES Function norm 5.245967459813e+01 > 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm > 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm > 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 > 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm > 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 > 1 SNES Function norm 5.245967167916e+01 > 24 TS dt 9.25775e-07 time 0.271564 > copy! > 0 SNES Function norm 5.245972527444e+01 > 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm > 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm > 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 > 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm > 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 > 1 SNES Function norm 5.245972527444e+01 > 0 SNES Function norm 5.245966286541e+01 > 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm > 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm > 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 > 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm > 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 > 1 SNES Function norm 5.245964735626e+01 > 25 TS dt 4.62888e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245965045260e+01 > 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm > 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm > 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 > 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm > 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 > 1 SNES Function norm 5.245965045260e+01 > 0 SNES Function norm 5.245964737315e+01 > 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm > 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm > 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 > 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm > 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 > 1 SNES Function norm 5.245953907407e+01 > 26 TS dt 2.31444e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245952669603e+01 > 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm > 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm > 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 > 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm > 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 > 1 SNES Function norm 5.245952669603e+01 > 0 SNES Function norm 5.245955619349e+01 > 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm > 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm > 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 > 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm > 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 > 1 SNES Function norm 5.245916156502e+01 > 27 TS dt 1.15722e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245916087517e+01 > 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm > 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm > 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 > 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm > 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 > 1 SNES Function norm 5.245916087517e+01 > 0 SNES Function norm 5.245922267220e+01 > 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm > 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm > 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 > 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm > 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 > 1 SNES Function norm 5.245868047885e+01 > 28 TS dt 5.7861e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245971666219e+01 > 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm > 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm > 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 > 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm > 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 > 1 SNES Function norm 5.245971666219e+01 > 0 SNES Function norm 5.245866772896e+01 > 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm > 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm > 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 > 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm > 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 > 1 SNES Function norm 5.245862997528e+01 > 29 TS dt 2.89305e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245864268828e+01 > 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm > 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm > 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 > 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm > 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 > 1 SNES Function norm 5.245864268828e+01 > 0 SNES Function norm 5.245863432005e+01 > 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm > 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm > 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 > 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm > 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 > 1 SNES Function norm 5.245668231675e+01 > 30 TS dt 1.44652e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245690451230e+01 > 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm > 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm > 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 > 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm > 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 > 1 SNES Function norm 5.245690451230e+01 > 0 SNES Function norm 5.245696584491e+01 > 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm > 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm > 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 > 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm > 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 > 1 SNES Function norm 5.245647736964e+01 > 31 TS dt 7.23262e-09 time 0.271565 > copy! > 0 SNES Function norm 5.245669639739e+01 > 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm > 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm > 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 > 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm > 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 > 1 SNES Function norm 5.245669639739e+01 > 0 SNES Function norm 5.245652990872e+01 > 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm > 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm > 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 > 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm > 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 > 1 SNES Function norm 5.245350296758e+01 > 32 TS dt 3.61631e-09 time 0.271565 > copy! > 0 SNES Function norm 5.246640623267e+01 > 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm > 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm > 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 > 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm > 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 > 1 SNES Function norm 5.246640623267e+01 > 0 SNES Function norm 5.245276947886e+01 > 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm > 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm > 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 > 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm > 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 > 1 SNES Function norm 5.245276947886e+01 > 0 SNES Function norm 5.245277054267e+01 > 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm > 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm > 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 > 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm > 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 > 1 SNES Function norm 5.240788106409e+01 > 33 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 5.240539051979e+01 > 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm > 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm > 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 > 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm > 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 > 1 SNES Function norm 5.240539051979e+01 > 0 SNES Function norm 5.240989970820e+01 > 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm > 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm > 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 > 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm > 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 > 1 SNES Function norm 5.240989970820e+01 > 0 SNES Function norm 5.241000025070e+01 > 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm > 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm > 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 > 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm > 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 > 1 SNES Function norm 5.228672678486e+01 > 34 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 5.229325617508e+01 > 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm > 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm > 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 > 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm > 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 > 1 SNES Function norm 5.229325617508e+01 > 0 SNES Function norm 5.228920001144e+01 > 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm > 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm > 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 > 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm > 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 > 1 SNES Function norm 5.228920001144e+01 > 0 SNES Function norm 5.228865204069e+01 > 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm > 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm > 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 > 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm > 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 > 1 SNES Function norm 5.175201563991e+01 > 35 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.168638826028e+01 > 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm > 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm > 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 > 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm > 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 > 1 SNES Function norm 5.168638826028e+01 > 0 SNES Function norm 5.178402875676e+01 > 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm > 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm > 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 > 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm > 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 > 1 SNES Function norm 5.178402875676e+01 > 0 SNES Function norm 5.178806568852e+01 > 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm > 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm > 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 > 1 SNES Function norm 4.925818844220e+01 > 36 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 5.270383537204e+01 > 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm > 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm > 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 > 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm > 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 > 1 SNES Function norm 5.270383537204e+01 > 0 SNES Function norm 4.874520711594e+01 > 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm > 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm > 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 > 1 SNES Function norm 4.874520711594e+01 > 0 SNES Function norm 4.879679718326e+01 > 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm > 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm > 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 > 1 SNES Function norm 4.872964148564e+01 > 37 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.866691749862e+01 > 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm > 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm > 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 > 1 SNES Function norm 4.866382095049e+01 > 38 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.893392986820e+01 > 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm > 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm > 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 > 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm > 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 > 1 SNES Function norm 4.893392986820e+01 > 0 SNES Function norm 4.865470718806e+01 > 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm > 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm > 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 > 1 SNES Function norm 4.843333577010e+01 > 39 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.842612728466e+01 > 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm > 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm > 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 > 1 SNES Function norm 4.842612728466e+01 > 0 SNES Function norm 4.840969985006e+01 > 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm > 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm > 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 > 1 SNES Function norm 3.917148384566e+01 > 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm > 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm > 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 > 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm > 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 > 2 SNES Function norm 3.373009701057e-01 > 40 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 4.835053852293e+01 > 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm > 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm > 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 > 1 SNES Function norm 4.835053852293e+01 > 0 SNES Function norm 1.287943764600e+01 > 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm > 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm > 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 > 1 SNES Function norm 1.128956057544e+01 > 41 TS dt 2.75903e-14 time 0.271565 > copy! > 0 SNES Function norm 1.669600342271e+01 > 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm > 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm > 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 > 1 SNES Function norm 1.472401994522e+01 > 42 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 5.552587859898e+01 > 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm > 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm > 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 > 1 SNES Function norm 2.907234222906e+01 > 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm > 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm > 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 > 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm > 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 > 2 SNES Function norm 1.772805631039e-01 > 43 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 7.725490953363e+00 > 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm > 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm > 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 > 1 SNES Function norm 5.415206018520e+00 > 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm > 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm > 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 > 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm > 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 > 2 SNES Function norm 2.952018920094e-02 > 44 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.042546652708e+00 > 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm > 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm > 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 > 1 SNES Function norm 7.553554533053e-01 > 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm > 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm > 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 > 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm > 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 > 2 SNES Function norm 6.213938456039e-02 > 45 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.726791649984e+00 > 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm > 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm > 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 > 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm > 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 > 1 SNES Function norm 1.726791649984e+00 > 0 SNES Function norm 4.569198217932e-01 > 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm > 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm > 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 > 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm > 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 > 1 SNES Function norm 5.060166636556e-02 > 46 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.648499891442e-01 > 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm > 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm > 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 > 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm > 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 > 1 SNES Function norm 4.303888962028e-01 > 47 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.205705397633e+00 > 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm > 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm > 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 > 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm > 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 > 1 SNES Function norm 1.205705397633e+00 > 0 SNES Function norm 3.168320704593e-01 > 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm > 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm > 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 > 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm > 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 > 1 SNES Function norm 1.059925414019e-01 > 48 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.008300299382e+00 > 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm > 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm > 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 > 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm > 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 > 1 SNES Function norm 5.842489315290e-01 > 49 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.508490594861e+00 > 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm > 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm > 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 > 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm > 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 > 1 SNES Function norm 4.745080825960e-01 > 50 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.014687630993e+00 > 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm > 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm > 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 > 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm > 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 > 1 SNES Function norm 5.592988640286e-01 > 51 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.841734166744e-01 > 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm > 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm > 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 > 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm > 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 > 1 SNES Function norm 9.841734166744e-01 > 0 SNES Function norm 5.290699913962e-01 > 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm > 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm > 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 > 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm > 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 > 1 SNES Function norm 4.072990107379e-01 > 52 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.293320470683e-01 > 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm > 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm > 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 > 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm > 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 > 1 SNES Function norm 3.293320470683e-01 > 0 SNES Function norm 6.190385444978e-01 > 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm > 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm > 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 > 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm > 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 > 1 SNES Function norm 3.612736786105e-01 > 53 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.245269204940e+00 > 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm > 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm > 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 > 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm > 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 > 1 SNES Function norm 5.113533173009e-01 > 54 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.607180712938e+00 > 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm > 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm > 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 > 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm > 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 > 1 SNES Function norm 1.607180712938e+00 > 0 SNES Function norm 3.045128849808e-01 > 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm > 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm > 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 > 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm > 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 > 1 SNES Function norm 2.935346973548e-01 > 55 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 8.467066865826e-01 > 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm > 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm > 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 > 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm > 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 > 1 SNES Function norm 5.502623367952e-01 > 56 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.717470692129e+00 > 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm > 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm > 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 > 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm > 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 > 1 SNES Function norm 4.045528183667e-01 > 57 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.544462481147e+00 > 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm > 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm > 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 > 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm > 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 > 1 SNES Function norm 1.293824512618e+00 > 58 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.191314292345e+00 > 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm > 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm > 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 > 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm > 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 > 1 SNES Function norm 2.617175498914e+00 > 59 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.489780310948e+00 > 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm > 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm > 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 > 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm > 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 > 1 SNES Function norm 2.949802369100e+00 > 60 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 4.631512871139e+00 > 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm > 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm > 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 > 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm > 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 > 1 SNES Function norm 2.195569697365e+00 > 61 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.662194684551e+00 > 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm > 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm > 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 > 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm > 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 > 1 SNES Function norm 1.662194684551e+00 > 0 SNES Function norm 2.329147999080e+00 > 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm > 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm > 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 > 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm > 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 > 1 SNES Function norm 1.548533538885e-01 > 62 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.019371357242e+00 > 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm > 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm > 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 > 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm > 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 > 1 SNES Function norm 2.813130743975e-01 > 63 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.849189158431e+00 > 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm > 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm > 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 > 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm > 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 > 1 SNES Function norm 1.849189158431e+00 > 0 SNES Function norm 1.106437703298e-01 > 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm > 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm > 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 > 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm > 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 > 1 SNES Function norm 1.071406781366e-01 > 64 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.036597476193e-01 > 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm > 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm > 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 > 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm > 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 > 1 SNES Function norm 1.036597476193e-01 > 0 SNES Function norm 1.081080288554e-01 > 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm > 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm > 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 > 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm > 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 > 1 SNES Function norm 4.280304180299e-02 > 65 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 2.901310268658e-02 > 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm > 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm > 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 > 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm > 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 > 1 SNES Function norm 2.901310268658e-02 > 0 SNES Function norm 6.115306732380e-02 > 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm > 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm > 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 > 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm > 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 > 1 SNES Function norm 6.115306732380e-02 > 0 SNES Function norm 8.316482031597e-02 > 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm > 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm > 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 > 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm > 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 > 1 SNES Function norm 8.316482031597e-02 > 0 SNES Function norm 3.114039374287e-01 > 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm > 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm > 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 > 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm > 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 > 1 SNES Function norm 1.476629082037e-01 > 66 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.741199964540e-01 > 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm > 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm > 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 > 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm > 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 > 1 SNES Function norm 2.741199964540e-01 > 0 SNES Function norm 1.363976538331e+00 > 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm > 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm > 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 > 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm > 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 > 1 SNES Function norm 3.620782473843e-01 > 67 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 2.101847656775e+00 > 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm > 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm > 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 > 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm > 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 > 1 SNES Function norm 5.837297362402e-01 > 68 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.691478479373e+00 > 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm > 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm > 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 > 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm > 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 > 1 SNES Function norm 1.646871085300e+00 > 69 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 4.140741966573e+00 > 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm > 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm > 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 > 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm > 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 > 1 SNES Function norm 1.278992537889e+00 > 70 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 4.925810717003e+00 > 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm > 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm > 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 > 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm > 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 > 1 SNES Function norm 4.757993340113e-01 > 71 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.014381910385e+00 > 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm > 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm > 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 > 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm > 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 > 1 SNES Function norm 1.014381910385e+00 > 0 SNES Function norm 5.247873040702e-01 > 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm > 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm > 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 > 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm > 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 > 1 SNES Function norm 5.128575534549e-01 > 72 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.463622536847e+00 > 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm > 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm > 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 > 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm > 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 > 1 SNES Function norm 1.463622536847e+00 > 0 SNES Function norm 5.760430307939e-01 > 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm > 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm > 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 > 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm > 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 > 1 SNES Function norm 4.031598199606e-01 > 73 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.304273554580e+00 > 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm > 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm > 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 > 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm > 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 > 1 SNES Function norm 5.901656814576e-01 > 74 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.806967885919e+00 > 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm > 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm > 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 > 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm > 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 > 1 SNES Function norm 1.806967885919e+00 > 0 SNES Function norm 3.525254737467e-01 > 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm > 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm > 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 > 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm > 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 > 1 SNES Function norm 2.971700641320e-01 > 75 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 9.276360188622e-01 > 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm > 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm > 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 > 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm > 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 > 1 SNES Function norm 3.629560323025e-01 > 76 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.281038880923e+00 > 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm > 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm > 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 > 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm > 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 > 1 SNES Function norm 1.281038880923e+00 > 0 SNES Function norm 2.062116742478e-01 > 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm > 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm > 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 > 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm > 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 > 1 SNES Function norm 4.076142877095e-02 > 77 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970990313724e-01 > 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm > 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm > 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 > 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm > 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 > 1 SNES Function norm 2.970990313724e-01 > 0 SNES Function norm 3.372906354998e-01 > 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm > 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm > 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 > 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm > 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 > 1 SNES Function norm 2.538611106909e-01 > 78 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 6.929161829162e-01 > 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm > 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm > 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 > 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm > 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 > 1 SNES Function norm 4.533321906044e-01 > 79 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.430558276957e+00 > 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm > 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm > 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 > 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm > 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 > 1 SNES Function norm 3.940601459612e-01 > 80 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 7.524069011779e-01 > 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm > 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm > 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 > 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm > 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 > 1 SNES Function norm 6.651399131369e-02 > 81 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.121363306761e-01 > 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm > 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm > 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 > 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm > 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 > 1 SNES Function norm 9.121363306761e-01 > 0 SNES Function norm 1.548591006657e-01 > 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm > 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm > 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 > 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm > 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 > 1 SNES Function norm 1.548591006657e-01 > 0 SNES Function norm 3.363022877948e-01 > 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm > 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm > 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 > 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm > 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 > 1 SNES Function norm 2.439869058762e-01 > 82 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.602729745290e+00 > 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm > 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm > 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 > 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm > 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 > 1 SNES Function norm 1.007188504688e+00 > 83 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.461320490774e+00 > 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm > 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm > 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 > 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm > 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 > 1 SNES Function norm 2.147586456566e+00 > 84 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 6.341628072414e+00 > 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm > 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm > 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 > 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm > 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 > 1 SNES Function norm 3.965504408873e-01 > 85 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.612683816763e-01 > 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm > 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm > 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 > 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm > 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 > 1 SNES Function norm 9.612683816763e-01 > 0 SNES Function norm 7.029352448723e-01 > 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm > 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm > 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 > 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm > 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 > 1 SNES Function norm 2.593382931820e-01 > 86 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.105068885806e-01 > 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm > 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm > 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 > 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm > 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 > 1 SNES Function norm 3.105068885806e-01 > 0 SNES Function norm 4.981482205604e-01 > 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm > 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm > 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 > 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm > 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 > 1 SNES Function norm 4.981482205604e-01 > 0 SNES Function norm 7.138905144630e-01 > 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm > 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm > 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 > 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm > 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 > 1 SNES Function norm 3.696022475452e-01 > 87 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.789063973037e-01 > 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm > 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm > 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 > 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm > 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 > 1 SNES Function norm 2.343211520992e-01 > 88 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.552109100087e+00 > 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm > 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm > 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 > 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm > 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 > 1 SNES Function norm 2.944262139143e+00 > 89 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 7.582944894373e+00 > 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm > 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm > 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 > 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm > 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 > 1 SNES Function norm 2.445215958037e+00 > 90 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.577254475804e+00 > 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm > 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm > 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 > 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm > 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 > 1 SNES Function norm 9.478103979365e-02 > 91 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 3.398926029549e+00 > 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm > 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm > 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 > 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm > 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 > 1 SNES Function norm 2.854550344474e+00 > 92 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.846027110554e+00 > 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm > 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm > 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 > 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm > 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 > 1 SNES Function norm 4.478173098971e+00 > 93 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 9.479827241861e+00 > 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm > 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm > 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 > 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm > 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 > 1 SNES Function norm 2.980698125901e+00 > 94 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.583823756970e+00 > 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm > 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm > 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 > 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm > 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 > 1 SNES Function norm 1.929142014309e+00 > 95 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.257265283168e-01 > 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm > 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm > 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 > 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm > 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 > 1 SNES Function norm 1.257265283168e-01 > 0 SNES Function norm 2.389423098829e+00 > 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm > 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm > 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 > 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm > 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 > 1 SNES Function norm 1.440136831177e-01 > 96 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.101422245445e+00 > 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm > 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm > 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 > 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm > 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 > 1 SNES Function norm 7.435685625660e-01 > 97 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 3.062056964383e+00 > 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm > 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm > 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 > 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm > 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 > 1 SNES Function norm 3.062056964383e+00 > 0 SNES Function norm 1.640638339300e-01 > 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm > 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm > 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 > 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm > 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 > 1 SNES Function norm 7.185385907970e-03 > 98 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.778588005116e-01 > 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm > 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm > 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 > 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm > 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 > 1 SNES Function norm 1.179095579605e-01 > 99 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 2.512082795192e-01 > 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm > 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm > 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 > 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm > 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 > 1 SNES Function norm 2.512082795192e-01 > 0 SNES Function norm 8.541223283936e-02 > 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm > 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm > 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 > 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm > 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 > 1 SNES Function norm 6.589891795632e-02 > 100 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.174757993770e-01 > 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm > 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm > 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 > 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm > 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 > 1 SNES Function norm 2.174757993770e-01 > 0 SNES Function norm 3.006903173936e-02 > 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm > 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm > 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 > 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm > 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 > 1 SNES Function norm 3.006903173936e-02 > 0 SNES Function norm 5.071683999025e-02 > 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm > 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm > 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 > 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm > 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 > 1 SNES Function norm 4.513917663662e-02 > 101 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.261469809763e-01 > 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm > 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm > 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 > 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm > 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 > 1 SNES Function norm 9.629885791766e-02 > 102 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.223153986924e-01 > 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm > 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm > 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 > 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm > 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 > 1 SNES Function norm 1.444530545281e-01 > 103 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 3.103810286198e-01 > 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm > 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm > 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 > 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm > 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 > 1 SNES Function norm 3.103810286198e-01 > 0 SNES Function norm 1.063191577145e-01 > 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm > 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm > 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 > 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm > 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 > 1 SNES Function norm 1.280948192266e-02 > 104 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 1.183517461737e-01 > 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm > 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm > 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 > 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm > 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 > 1 SNES Function norm 3.338348178983e-02 > 105 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.506755043591e-01 > 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm > 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm > 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 > 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm > 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 > 1 SNES Function norm 1.506755043591e-01 > 0 SNES Function norm 1.678694983365e-02 > 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm > 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm > 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 > 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm > 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 > 1 SNES Function norm 1.678694983365e-02 > 0 SNES Function norm 2.211256020366e-02 > 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm > 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm > 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 > 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm > 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 > 1 SNES Function norm 2.211256020366e-02 > 0 SNES Function norm 6.319453516491e-02 > 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm > 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm > 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 > 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm > 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 > 1 SNES Function norm 6.319453516491e-02 > 0 SNES Function norm 1.993150558437e+00 > 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm > 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm > 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 > 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm > 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 > 1 SNES Function norm 7.832112166701e-01 > 106 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.629560288909e+00 > 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm > 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm > 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 > 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm > 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 > 1 SNES Function norm 1.672457508921e+00 > 107 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.062643571582e+00 > 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm > 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm > 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 > 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm > 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 > 1 SNES Function norm 9.036991011875e-01 > 108 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.105674791223e+00 > 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm > 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm > 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 > 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm > 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 > 1 SNES Function norm 6.003561441737e-01 > 109 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.053877992744e+00 > 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm > 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm > 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 > 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm > 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 > 1 SNES Function norm 3.053877992744e+00 > 0 SNES Function norm 5.829087412676e-01 > 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm > 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm > 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 > 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm > 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 > 1 SNES Function norm 4.254244459909e-01 > 110 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.180846267144e+00 > 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm > 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm > 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 > 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm > 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 > 1 SNES Function norm 3.911236882643e-01 > 111 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.532064854858e+00 > 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm > 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm > 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 > 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm > 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 > 1 SNES Function norm 1.532064854858e+00 > 0 SNES Function norm 1.764940070080e-01 > 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm > 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm > 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 > 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm > 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 > 1 SNES Function norm 1.706450714666e-01 > 112 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.407680008950e-01 > 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm > 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm > 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 > 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm > 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 > 1 SNES Function norm 4.407680008950e-01 > 0 SNES Function norm 1.064954076440e+00 > 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm > 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm > 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 > 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm > 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 > 1 SNES Function norm 2.964963371165e-01 > 113 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.622925709624e-01 > 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm > 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm > 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 > 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm > 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 > 1 SNES Function norm 8.622925709624e-01 > 0 SNES Function norm 2.612065167743e+00 > 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm > 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm > 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 > 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm > 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 > 1 SNES Function norm 1.331790654620e+00 > 114 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.606851714788e+00 > 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm > 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm > 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 > 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm > 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 > 1 SNES Function norm 9.013753678258e-01 > 115 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 3.227741616537e+00 > 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm > 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm > 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 > 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm > 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 > 1 SNES Function norm 7.734015655943e-01 > 116 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.393623504956e+00 > 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm > 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm > 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 > 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm > 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 > 1 SNES Function norm 4.969182786938e-01 > 117 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.596405006351e+00 > 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm > 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm > 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 > 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm > 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 > 1 SNES Function norm 2.596405006351e+00 > 0 SNES Function norm 4.974558747187e-01 > 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm > 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm > 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 > 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm > 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 > 1 SNES Function norm 2.909443551880e-01 > 118 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.088860095507e+00 > 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm > 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm > 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 > 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm > 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 > 1 SNES Function norm 6.203518870415e-01 > 119 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.828717001803e+00 > 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm > 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm > 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 > 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm > 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 > 1 SNES Function norm 9.407254842319e-01 > 120 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970950728036e+00 > 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm > 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm > 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 > 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm > 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 > 1 SNES Function norm 2.277472194573e+00 > 121 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 6.655252327413e+00 > 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm > 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm > 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 > 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm > 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 > 1 SNES Function norm 3.860967926913e+00 > 122 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.141502067665e+00 > 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm > 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm > 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 > 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm > 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 > 1 SNES Function norm 6.426598300871e+00 > 123 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.224562198048e+01 > 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm > 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm > 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 > 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm > 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 > 1 SNES Function norm 5.283049961509e+00 > 124 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 5.780124419709e+00 > 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm > 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm > 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 > 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm > 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 > 1 SNES Function norm 1.227257133866e+00 > 125 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 5.063237678461e+00 > 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm > 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm > 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 > 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm > 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 > 1 SNES Function norm 4.907594073483e+00 > 126 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 8.280289521084e+00 > 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm > 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm > 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 > 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm > 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 > 1 SNES Function norm 3.515932015486e+00 > 127 TS dt 1.1301e-10 time 0.271565 > copy! > 0 SNES Function norm 2.638351398877e+00 > 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm > 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm > 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 > 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm > 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 > 1 SNES Function norm 6.341356656368e-01 > 128 TS dt 2.26019e-10 time 0.271565 > copy! > 0 SNES Function norm 4.327811744723e+00 > 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm > 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm > 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 > 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm > 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 > 1 SNES Function norm 1.736633786129e-01 > 129 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 3.450425343262e+00 > 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm > 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm > 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 > 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm > 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 > 1 SNES Function norm 1.373376592274e+00 > 130 TS dt 9.04078e-10 time 0.271565 > copy! > 0 SNES Function norm 4.603810494450e+00 > 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm > 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm > 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 > 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm > 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 > 1 SNES Function norm 1.142888930566e+00 > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 2:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi wrote: > > Dear Petsc Users > I wrote a code in 2D solving PF equations for solidification of alloys. I > have two variables in the equations 1-non-dimensional concentration (U) 2- > order parameter (p). Everything is fine with this code. > When I change the code to 3D, the code has convergence problems which I > cannot solve it. > I know the term which is causing problem is a term in the equation for U: > (dp/dt)*(Pxx+Pyy+Pzz) > where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, > respectively. > > When I eliminate this term, the results of 2D and 3D are close to each > other. But adding that term the time step should be reduced a lot to obtain > convergence, otherwise the timestep does not converge. I have no idea how I > should solve this problem. > I solve the equations considering (I chose based on suggestions of Moose > for solving Cahn-hillard) > -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 > -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type > ilu -sub_ksp_type preonly > > Would you please help me solve this issue? > > > In order to see the solver you have we need the output of > > -ts_view > > and then to see what is happening during the run we need > > -ts_monitor -snes_monitor -snes_linesearch_monitor > -ksp_monitor_true_residual > > Thanks, > > Matt > > > Best, > sepideh > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jan 5 18:44:49 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 5 Jan 2021 18:44:49 -0600 Subject: [petsc-users] Elementwise pow/exponential-function on PETSc-vectors and matrices In-Reply-To: References: Message-ID: Use VecGetArray() and then loop over the array entries performing whatever element operations you like. For dense matrices use MatDenseGetArray() and loop over the entries. For sequential AIJ matrices uses MatSeqAIJGetArray() For MPIAIJ use MatMPIAIJGetSeqAIJ() then use MatSeqAIJGetArray on each of the two matrices. Barry > On Jan 5, 2021, at 9:24 AM, Roland Richter wrote: > > Hei, > > is there a function to square/cube etc. each element of a matrix or > vector in PETSc? It is rather useful for things like intensity > calculations (I = |E|^2), and should work also for distributed matrices. > > Thanks! > > Regards, > > Roland Richter > From skavou1 at lsu.edu Tue Jan 5 19:47:47 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Wed, 6 Jan 2021 01:47:47 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: , Message-ID: The output for -snes_test_jacobian is given as : 0 TS dt 0.0001 time 0. copy! Write output at step= 0! ---------- Testing Jacobian ------------- Run with -snes_test_jacobian_view and optionally -snes_test_jacobian to show difference of hand-coded and finite difference Jacobian entries greater than . Testing hand-coded Jacobian, if (for double precision runs) ||J - Jfd||_F/||J||_F is O(1.e-8), the hand-coded Jacobian is probably correct. ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 6.673211216524e-03 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.067366268006e-18 true resid norm 1.921588081784e-14 ||r(i)||/||b|| 3.774355453546e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 1.668754123581e-03 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.604939763954e-18 true resid norm 4.960207918618e-14 ||r(i)||/||b|| 9.742768487084e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 4.172167491706e-04 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.588913006621e-20 true resid norm 1.948840699675e-14 ||r(i)||/||b|| 3.827884650536e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 1.043059511038e-04 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.104783140057e-19 true resid norm 5.333375230602e-14 ||r(i)||/||b|| 1.047573831159e-15 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 2.607659801680e-05 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.703990624142e-20 true resid norm 6.287819675470e-14 ||r(i)||/||b|| 1.235044425390e-15 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 6.519156394293e-06 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.557215852400e-21 true resid norm 1.840177917795e-14 ||r(i)||/||b|| 3.614450789618e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 1.629789529205e-06 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.173065981676e-22 true resid norm 1.530485325589e-14 ||r(i)||/||b|| 3.006157089528e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 4.074474092157e-07 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.739825180849e-22 true resid norm 2.686974828851e-14 ||r(i)||/||b|| 5.277717006551e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 1.018618539861e-07 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.045098765057e-22 true resid norm 5.010251971826e-14 ||r(i)||/||b|| 9.841064290920e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 2.546546360165e-08 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.287065233667e-24 true resid norm 1.149255127646e-14 ||r(i)||/||b|| 2.257350261311e-16 ---------- Testing Jacobian ------------- ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. 0 KSP preconditioned resid norm 6.366365906984e-09 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.164978605997e-24 true resid norm 2.417410255090e-14 ||r(i)||/||b|| 4.748242178566e-16 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.0, Sep 29, 2020 [0]PETSC ERROR: /scratch/07065/tg863649/alloy-3D-cubic-9/one.out on a skylake named c456-001.stampede2.tacc.utexas.edu by tg863649 Tue Jan 5 18:35:45 2021 [0]PETSC ERROR: Configure options --with-x=0 -with-pic --with-make-np=8 --download-petsc4py=1 --with-python-exec=/opt/apps/intel18/python3/3.7.0/bin/python3 --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" --with-packages-build-dir=/tmp/petsc-3.14/skylake --with-mpi=1 --with-mpi-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64 --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-plapack=1 --download-plapack --with-spai=1 --download-spai --with-sundials=1 --download-sundials --with-elemental=1 --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-fftw=1 --with-fftw-dir=/opt/apps/intel18/impi18_0/fftw3/3.3.8 --with-hdf5=1 --with-hdf5-dir=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64 --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 --with-debugging=no --LIBS= --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" [0]PETSC ERROR: #1 TSStep() line 3773 in /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c [0]PETSC ERROR: #2 TSSolve() line 4154 in /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c [0]PETSC ERROR: #3 main() line 324 in /scratch/07065/tg863649/alloy-3D-cubic-9/one.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_gmres_restart 1001 [0]PETSC ERROR: -ksp_monitor_true_residual [0]PETSC ERROR: -ksp_type gmres [0]PETSC ERROR: -pc_type lu [0]PETSC ERROR: -snes_fd_color [0]PETSC ERROR: -snes_linesearch_monitor [0]PETSC ERROR: -snes_linesearch_type l2 [0]PETSC ERROR: -snes_test_jacobian [0]PETSC ERROR: -snes_type newtontr [0]PETSC ERROR: -sub_ksp_type preonly [0]PETSC ERROR: -sub_pc_type ilu [0]PETSC ERROR: -ts_bdf_adapt [0]PETSC ERROR: -ts_max_snes_failures -1 [0]PETSC ERROR: -ts_monitor [0]PETSC ERROR: -ts_type bdf [0]PETSC ERROR: -ts_view [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov--------- ________________________________ From: Matthew Knepley Sent: Tuesday, January 5, 2021 3:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 4:48 PM Sepideh Kavousi > wrote: I have tried different pcs (lu,jacobi,sor,mg,asm,ksp,pbjacobi). they still do not solve the problem. the an example for lu: -pc_type lu: 0 TS dt 0.0001 time 0. copy! Write output at step= 0! 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.761968940423e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.026000796457e-17 true resid norm 1.790914574372e-13 ||r(i)||/||b|| 3.050633943390e-15 1 SNES Function norm 5.808754845048e+01 So with a perfect linear solver, Newton is making no progress. That means that your initial guess is very far from the solution, but it could also mean that your Jacobian does not match your residual, since it is not generating descent directions. Have you checked that it matches the finite difference approximation? You can do this using -snes_test_jacobian (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESComputeJacobian.html) Thanks, Matt 0 KSP preconditioned resid norm 1.191540511981e+06 true resid norm 5.808754845048e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.114882955273e-10 true resid norm 1.203029448994e-06 ||r(i)||/||b|| 2.071062527315e-08 2 SNES Function norm 5.794558475659e+01 0 KSP preconditioned resid norm 1.191526760369e+06 true resid norm 5.794558475659e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.166698266769e-09 true resid norm 1.059436818305e-06 ||r(i)||/||b|| 1.828330532439e-08 3 SNES Function norm 5.772046032228e+01 0 KSP preconditioned resid norm 1.191498793276e+06 true resid norm 5.772046032228e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.135175980555e-09 true resid norm 1.230437568859e-06 ||r(i)||/||b|| 2.131718219136e-08 4 SNES Function norm 5.750822623807e+01 0 KSP preconditioned resid norm 1.191443279124e+06 true resid norm 5.750822623807e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.905670672745e-10 true resid norm 1.110684454395e-06 ||r(i)||/||b|| 1.931348829639e-08 5 SNES Function norm 5.750395740069e+01 0 KSP preconditioned resid norm 1.191433462635e+06 true resid norm 5.750395740069e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.777205269843e-10 true resid norm 1.430076525401e-06 ||r(i)||/||b|| 2.486918448822e-08 6 SNES Function norm 5.750395740069e+01 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 4.405989777785e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.185260514636e-18 true resid norm 2.397998208837e-14 ||r(i)||/||b|| 4.084736836000e-16 1 SNES Function norm 3.428067045216e+01 0 KSP preconditioned resid norm 1.895265798836e+06 true resid norm 3.428067045216e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.403775884750e-09 true resid norm 8.602063630559e-06 ||r(i)||/||b|| 2.509304373893e-07 2 SNES Function norm 3.428067045216e+01 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.101564184740e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.380439828314e-19 true resid norm 3.289971303359e-14 ||r(i)||/||b|| 5.604118853253e-16 1 SNES Function norm 3.330623063627e+01 0 KSP preconditioned resid norm 6.892935831006e-04 true resid norm 3.330623063627e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.234145873951e-19 true resid norm 1.823438085202e-14 ||r(i)||/||b|| 5.474765683081e-16 2 SNES Function norm 1.228168587373e+01 0 KSP preconditioned resid norm 2.442179927376e-04 true resid norm 1.228168587373e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.114351195683e-19 true resid norm 1.328341389902e-14 ||r(i)||/||b|| 1.081562745993e-15 3 SNES Function norm 8.377002138765e+00 0 KSP preconditioned resid norm 1.670901807768e-04 true resid norm 8.377002138765e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.056553283627e-19 true resid norm 1.539291322344e-14 ||r(i)||/||b|| 1.837520507750e-15 4 SNES Function norm 6.871259668846e-01 0 KSP preconditioned resid norm 7.849703293659e-06 true resid norm 6.871259668846e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.813692242300e-21 true resid norm 3.443983247611e-16 ||r(i)||/||b|| 5.012157033194e-16 5 SNES Function norm 4.598894985660e-01 0 SNES Function norm 4.609452800861e-01 0 KSP preconditioned resid norm 4.482807772478e+00 true resid norm 4.609452800861e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.779043025722e-15 true resid norm 9.363790606967e-11 ||r(i)||/||b|| 2.031432148566e-10 1 SNES Function norm 3.909615632676e-01 1 TS dt 1.25e-05 time 6.25e-06 copy! 0 SNES Function norm 1.083064424363e+00 0 KSP preconditioned resid norm 2.899372173019e+02 true resid norm 1.083064424363e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.843039617595e-13 true resid norm 2.031204561200e-09 ||r(i)||/||b|| 1.875423581006e-09 1 SNES Function norm 1.083064424363e+00 0 SNES Function norm 4.081254017393e-01 0 KSP preconditioned resid norm 1.134610736819e+01 true resid norm 4.081254017393e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.857284280292e-15 true resid norm 1.950178080787e-10 ||r(i)||/||b|| 4.778379567837e-10 1 SNES Function norm 3.852294258555e-01 2 TS dt 6.25e-06 time 9.375e-06 copy! 0 SNES Function norm 3.855090631551e-01 0 KSP preconditioned resid norm 5.807020984426e+01 true resid norm 3.855090631551e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.595611115758e-14 true resid norm 6.458636528724e-10 ||r(i)||/||b|| 1.675352707888e-09 1 SNES Function norm 3.855090631551e-01 0 SNES Function norm 3.894882915800e-01 0 KSP preconditioned resid norm 3.545691207399e+00 true resid norm 3.894882915800e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.936099793899e-15 true resid norm 1.237085076854e-10 ||r(i)||/||b|| 3.176180397711e-10 1 SNES Function norm 3.826734119825e-01 3 TS dt 3.125e-06 time 1.09375e-05 copy! 0 SNES Function norm 3.846635056055e-01 0 KSP preconditioned resid norm 1.270632337662e+01 true resid norm 3.846635056055e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.538849751267e-15 true resid norm 2.462963992263e-10 ||r(i)||/||b|| 6.402905283117e-10 1 SNES Function norm 3.846635056055e-01 0 SNES Function norm 3.838572028974e-01 0 KSP preconditioned resid norm 9.750671428246e-01 true resid norm 3.838572028974e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.289810958767e-15 true resid norm 5.932884665420e-11 ||r(i)||/||b|| 1.545596805436e-10 1 SNES Function norm 3.823876176903e-01 4 TS dt 1.5625e-06 time 1.17188e-05 copy! 0 SNES Function norm 3.842190105197e-01 0 KSP preconditioned resid norm 2.949895830433e+00 true resid norm 3.842190105197e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.544123722663e-15 true resid norm 1.184005133464e-10 ||r(i)||/||b|| 3.081589148499e-10 1 SNES Function norm 3.842190105197e-01 0 SNES Function norm 3.825899662960e-01 0 KSP preconditioned resid norm 2.548468293806e-01 true resid norm 3.825899662960e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.086200228021e-16 true resid norm 2.882610131465e-11 ||r(i)||/||b|| 7.534463486778e-11 1 SNES Function norm 3.825019685537e-01 5 TS dt 7.8125e-07 time 1.21094e-05 copy! 0 SNES Function norm 3.827182409606e-01 0 KSP preconditioned resid norm 7.091231452893e-01 true resid norm 3.827182409606e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.614097858359e-16 true resid norm 5.440482894192e-11 ||r(i)||/||b|| 1.421537390153e-10 1 SNES Function norm 3.827182409606e-01 0 SNES Function norm 3.825683488225e-01 0 KSP preconditioned resid norm 6.509635329694e-02 true resid norm 3.825683488225e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.255654660173e-17 true resid norm 1.410340803479e-11 ||r(i)||/||b|| 3.686506758386e-11 1 SNES Function norm 3.825462954733e-01 6 TS dt 3.90625e-07 time 1.23047e-05 copy! 0 SNES Function norm 3.826632731069e-01 0 KSP preconditioned resid norm 1.737397651071e-01 true resid norm 3.826632731069e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.095925875164e-16 true resid norm 2.780031944787e-11 ||r(i)||/||b|| 7.264956268774e-11 1 SNES Function norm 3.826632731069e-01 0 SNES Function norm 3.825751343810e-01 0 KSP preconditioned resid norm 1.644741829353e-02 true resid norm 3.825751343810e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.211213666062e-18 true resid norm 7.515828526583e-12 ||r(i)||/||b|| 1.964536597169e-11 1 SNES Function norm 3.825729851167e-01 7 TS dt 1.95313e-07 time 1.24023e-05 copy! 0 SNES Function norm 3.826356173729e-01 0 KSP preconditioned resid norm 4.299238585205e-02 true resid norm 3.826356173729e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.260847149806e-17 true resid norm 1.287166330766e-11 ||r(i)||/||b|| 3.363948028683e-11 1 SNES Function norm 3.826356173729e-01 0 SNES Function norm 3.825855754640e-01 0 KSP preconditioned resid norm 4.133474793122e-03 true resid norm 3.825855754640e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.077297138396e-18 true resid norm 3.792689943805e-12 ||r(i)||/||b|| 9.913311392374e-12 1 SNES Function norm 3.825855754640e-01 0 SNES Function norm 3.825755706234e-01 0 KSP preconditioned resid norm 3.705095771277e-04 true resid norm 3.825755706234e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.848267031641e-19 true resid norm 1.079462463835e-12 ||r(i)||/||b|| 2.821566630813e-12 1 SNES Function norm 3.825755706234e-01 0 SNES Function norm 3.825734391910e-01 0 KSP preconditioned resid norm 2.690993850525e-05 true resid norm 3.825734391910e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.585015168707e-21 true resid norm 3.091320932665e-13 ||r(i)||/||b|| 8.080333384362e-13 1 SNES Function norm 3.825734391910e-01 0 SNES Function norm 3.825729114845e-01 0 KSP preconditioned resid norm 1.757556306807e-06 true resid norm 3.825729114845e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.987700736384e-22 true resid norm 8.038113355816e-14 ||r(i)||/||b|| 2.101067042260e-13 1 SNES Function norm 3.825729114845e-01 0 SNES Function norm 3.825565841303e-01 0 KSP preconditioned resid norm 1.103248802240e-07 true resid norm 3.825565841303e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.504795613889e-22 true resid norm 1.879813159274e-14 ||r(i)||/||b|| 4.913817294630e-14 1 SNES Function norm 3.825565841303e-01 0 SNES Function norm 3.825547574266e-01 0 KSP preconditioned resid norm 6.887463339138e-09 true resid norm 3.825547574266e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.559974044704e-23 true resid norm 5.311723680631e-15 ||r(i)||/||b|| 1.388487158378e-14 1 SNES Function norm 3.825547574266e-01 0 SNES Function norm 3.825908198230e-01 0 KSP preconditioned resid norm 4.314157283909e-10 true resid norm 3.825908198230e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.248846716324e-24 true resid norm 1.634240020842e-15 ||r(i)||/||b|| 4.271508714187e-15 1 SNES Function norm 3.825908198230e-01 0 SNES Function norm 3.823046631992e-01 0 KSP preconditioned resid norm 1.477599397951e-11 true resid norm 3.823046631992e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.674742673343e-25 true resid norm 4.342823046268e-15 ||r(i)||/||b|| 1.135958690623e-14 1 SNES Function norm 3.823046631992e-01 0 SNES Function norm 4.050006682433e-01 0 KSP preconditioned resid norm 4.744618612191e-12 true resid norm 4.050006682433e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.185835350170e-26 true resid norm 2.742859197646e-15 ||r(i)||/||b|| 6.772480671558e-15 1 SNES Function norm 4.050006682433e-01 0 SNES Function norm 4.420555752951e-01 0 KSP preconditioned resid norm 2.570862477765e-13 true resid norm 4.420555752951e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.558628106533e-27 true resid norm 2.669513100096e-15 ||r(i)||/||b|| 6.038863096149e-15 1 SNES Function norm 3.053219162874e-01 8 TS dt 3.72529e-13 time 1.24023e-05 copy! 0 SNES Function norm 2.938107171164e-01 0 KSP preconditioned resid norm 1.152686085246e-13 true resid norm 2.938107171164e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.059363368490e-28 true resid norm 1.486663802399e-15 ||r(i)||/||b|| 5.059937285439e-15 1 SNES Function norm 1.879261738365e-01 9 TS dt 7.45058e-13 time 1.24023e-05 copy! 0 SNES Function norm 2.598740119900e-01 0 KSP preconditioned resid norm 4.386389243102e-12 true resid norm 2.598740119900e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.765692021861e-27 true resid norm 4.362311845294e-16 ||r(i)||/||b|| 1.678625658599e-15 1 SNES Function norm 2.598740119900e-01 0 SNES Function norm 2.359036347057e-01 0 KSP preconditioned resid norm 3.416321738725e-13 true resid norm 2.359036347057e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.983917914615e-27 true resid norm 2.031818641817e-15 ||r(i)||/||b|| 8.612917916048e-15 1 SNES Function norm 1.708032078859e-01 10 TS dt 3.72529e-13 time 1.24023e-05 copy! 0 SNES Function norm 5.285427862819e-01 0 KSP preconditioned resid norm 1.592064460142e-12 true resid norm 5.285427862819e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.163339775654e-26 true resid norm 3.880080925123e-15 ||r(i)||/||b|| 7.341091442033e-15 1 SNES Function norm 5.285427862819e-01 0 SNES Function norm 1.646935190988e-01 0 KSP preconditioned resid norm 5.824112147206e-14 true resid norm 1.646935190988e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.820310421468e-29 true resid norm 7.168182199430e-17 ||r(i)||/||b|| 4.352437326407e-16 1 SNES Function norm 1.621698143957e-01 ________________________________ From: Matthew Knepley > Sent: Tuesday, January 5, 2021 3:25 PM To: Sepideh Kavousi > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi > wrote: Dear Matt, following is the output for the first 130 timesteps. It continues the run with the same small timestep 0 TS dt 0.0001 time 0. copy! Write output at step= 0! 0 SNES Function norm 5.870630851181e+01 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 1 SNES Function norm 5.870480277455e+01 0 SNES Function norm 5.870480198099e+01 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 You can see here that the linear solver is very bad, which might contribute to Newton stagnating. While you are starting out, I would recommend -pc_type lu Once you have the nonlinear system converging, then you can try different linear solvers to speed things up. Thanks, Matt 1 SNES Function norm 5.870165343582e+01 1 TS dt 0.0002 time 0.0001 copy! 0 SNES Function norm 5.869324856421e+01 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 1 SNES Function norm 5.869298823911e+01 2 TS dt 0.0004 time 0.0003 copy! 0 SNES Function norm 5.867574766362e+01 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 1 SNES Function norm 5.867531973582e+01 3 TS dt 0.0008 time 0.0007 copy! 0 SNES Function norm 5.864011322220e+01 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 1 SNES Function norm 5.863990526788e+01 4 TS dt 0.0016 time 0.0015 copy! 0 SNES Function norm 5.856903590475e+01 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 1 SNES Function norm 5.856893794226e+01 5 TS dt 0.0032 time 0.0031 copy! 0 SNES Function norm 5.842679564768e+01 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 1 SNES Function norm 5.842675271515e+01 6 TS dt 0.0064 time 0.0063 copy! 0 SNES Function norm 5.814352339598e+01 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 1 SNES Function norm 5.814350805660e+01 7 TS dt 0.0128 time 0.0127 copy! 0 SNES Function norm 5.760173270961e+01 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 1 SNES Function norm 5.760173134258e+01 8 TS dt 0.0256 time 0.0255 copy! 0 SNES Function norm 5.683398372141e+01 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 1 SNES Function norm 5.683398372141e+01 0 SNES Function norm 5.747024606058e+01 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 1 SNES Function norm 5.747023898114e+01 9 TS dt 0.00758395 time 0.0319 copy! 0 SNES Function norm 5.725811890845e+01 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 1 SNES Function norm 5.725811208900e+01 10 TS dt 0.0151679 time 0.039484 copy! 0 SNES Function norm 5.684339053584e+01 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 1 SNES Function norm 5.684338688259e+01 11 TS dt 0.0303358 time 0.0546519 copy! 0 SNES Function norm 5.605303343054e+01 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 1 SNES Function norm 5.605303164316e+01 12 TS dt 0.0606716 time 0.0849877 copy! 0 SNES Function norm 5.463652619423e+01 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 1 SNES Function norm 5.463652534643e+01 13 TS dt 0.121343 time 0.145659 copy! 0 SNES Function norm 5.251966690761e+01 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 1 SNES Function norm 5.251966654744e+01 14 TS dt 0.242686 time 0.267003 copy! 0 SNES Function norm 5.150151326391e+01 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 1 SNES Function norm 5.150151326391e+01 0 SNES Function norm 5.184908859688e+01 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 1 SNES Function norm 5.184908859688e+01 0 SNES Function norm 5.232682115621e+01 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 1 SNES Function norm 5.232682115621e+01 0 SNES Function norm 5.246983753576e+01 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 1 SNES Function norm 5.246983737815e+01 15 TS dt 0.00758395 time 0.270794 copy! 0 SNES Function norm 5.237345394852e+01 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 1 SNES Function norm 5.237345394852e+01 0 SNES Function norm 5.244535357001e+01 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 1 SNES Function norm 5.244535357001e+01 0 SNES Function norm 5.246369218939e+01 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 1 SNES Function norm 5.246369181668e+01 16 TS dt 0.000947994 time 0.271268 copy! 0 SNES Function norm 5.245144714661e+01 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 1 SNES Function norm 5.245144714661e+01 0 SNES Function norm 5.246062448755e+01 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 1 SNES Function norm 5.246062426384e+01 17 TS dt 0.000473997 time 0.271505 copy! 0 SNES Function norm 5.245449911597e+01 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 1 SNES Function norm 5.245449911597e+01 0 SNES Function norm 5.245909110130e+01 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 1 SNES Function norm 5.245909110130e+01 0 SNES Function norm 5.246024056684e+01 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 1 SNES Function norm 5.246024055663e+01 18 TS dt 5.92496e-05 time 0.271535 copy! 0 SNES Function norm 5.245947411014e+01 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 1 SNES Function norm 5.245947411014e+01 0 SNES Function norm 5.246004891756e+01 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 1 SNES Function norm 5.246004890902e+01 19 TS dt 2.96248e-05 time 0.27155 copy! 0 SNES Function norm 5.245966566276e+01 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 1 SNES Function norm 5.245966566276e+01 0 SNES Function norm 5.245995306375e+01 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 1 SNES Function norm 5.245977670896e+01 20 TS dt 1.48124e-05 time 0.271557 copy! 0 SNES Function norm 5.245977453999e+01 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 1 SNES Function norm 5.245977453999e+01 0 SNES Function norm 5.245973715661e+01 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 1 SNES Function norm 5.245972630220e+01 21 TS dt 7.4062e-06 time 0.271561 copy! 0 SNES Function norm 5.245962991861e+01 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 1 SNES Function norm 5.245962991861e+01 0 SNES Function norm 5.245970348930e+01 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 1 SNES Function norm 5.245969886119e+01 22 TS dt 3.7031e-06 time 0.271563 copy! 0 SNES Function norm 5.245964973088e+01 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 1 SNES Function norm 5.245964973088e+01 0 SNES Function norm 5.245968740629e+01 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 1 SNES Function norm 5.245968066039e+01 23 TS dt 1.85155e-06 time 0.271564 copy! 0 SNES Function norm 5.245966664362e+01 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 1 SNES Function norm 5.245966664362e+01 0 SNES Function norm 5.245967459813e+01 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 1 SNES Function norm 5.245967167916e+01 24 TS dt 9.25775e-07 time 0.271564 copy! 0 SNES Function norm 5.245972527444e+01 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 1 SNES Function norm 5.245972527444e+01 0 SNES Function norm 5.245966286541e+01 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 1 SNES Function norm 5.245964735626e+01 25 TS dt 4.62888e-07 time 0.271565 copy! 0 SNES Function norm 5.245965045260e+01 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 1 SNES Function norm 5.245965045260e+01 0 SNES Function norm 5.245964737315e+01 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 1 SNES Function norm 5.245953907407e+01 26 TS dt 2.31444e-07 time 0.271565 copy! 0 SNES Function norm 5.245952669603e+01 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 1 SNES Function norm 5.245952669603e+01 0 SNES Function norm 5.245955619349e+01 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 1 SNES Function norm 5.245916156502e+01 27 TS dt 1.15722e-07 time 0.271565 copy! 0 SNES Function norm 5.245916087517e+01 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 1 SNES Function norm 5.245916087517e+01 0 SNES Function norm 5.245922267220e+01 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 1 SNES Function norm 5.245868047885e+01 28 TS dt 5.7861e-08 time 0.271565 copy! 0 SNES Function norm 5.245971666219e+01 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 1 SNES Function norm 5.245971666219e+01 0 SNES Function norm 5.245866772896e+01 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 1 SNES Function norm 5.245862997528e+01 29 TS dt 2.89305e-08 time 0.271565 copy! 0 SNES Function norm 5.245864268828e+01 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 1 SNES Function norm 5.245864268828e+01 0 SNES Function norm 5.245863432005e+01 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 1 SNES Function norm 5.245668231675e+01 30 TS dt 1.44652e-08 time 0.271565 copy! 0 SNES Function norm 5.245690451230e+01 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 1 SNES Function norm 5.245690451230e+01 0 SNES Function norm 5.245696584491e+01 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 1 SNES Function norm 5.245647736964e+01 31 TS dt 7.23262e-09 time 0.271565 copy! 0 SNES Function norm 5.245669639739e+01 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 1 SNES Function norm 5.245669639739e+01 0 SNES Function norm 5.245652990872e+01 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 1 SNES Function norm 5.245350296758e+01 32 TS dt 3.61631e-09 time 0.271565 copy! 0 SNES Function norm 5.246640623267e+01 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 1 SNES Function norm 5.246640623267e+01 0 SNES Function norm 5.245276947886e+01 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 1 SNES Function norm 5.245276947886e+01 0 SNES Function norm 5.245277054267e+01 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 1 SNES Function norm 5.240788106409e+01 33 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 5.240539051979e+01 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 1 SNES Function norm 5.240539051979e+01 0 SNES Function norm 5.240989970820e+01 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 1 SNES Function norm 5.240989970820e+01 0 SNES Function norm 5.241000025070e+01 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 1 SNES Function norm 5.228672678486e+01 34 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 5.229325617508e+01 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 1 SNES Function norm 5.229325617508e+01 0 SNES Function norm 5.228920001144e+01 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 1 SNES Function norm 5.228920001144e+01 0 SNES Function norm 5.228865204069e+01 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 1 SNES Function norm 5.175201563991e+01 35 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.168638826028e+01 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 1 SNES Function norm 5.168638826028e+01 0 SNES Function norm 5.178402875676e+01 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 1 SNES Function norm 5.178402875676e+01 0 SNES Function norm 5.178806568852e+01 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 1 SNES Function norm 4.925818844220e+01 36 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 5.270383537204e+01 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 1 SNES Function norm 5.270383537204e+01 0 SNES Function norm 4.874520711594e+01 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 1 SNES Function norm 4.874520711594e+01 0 SNES Function norm 4.879679718326e+01 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 1 SNES Function norm 4.872964148564e+01 37 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.866691749862e+01 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 1 SNES Function norm 4.866382095049e+01 38 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.893392986820e+01 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 1 SNES Function norm 4.893392986820e+01 0 SNES Function norm 4.865470718806e+01 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 1 SNES Function norm 4.843333577010e+01 39 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.842612728466e+01 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 1 SNES Function norm 4.842612728466e+01 0 SNES Function norm 4.840969985006e+01 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 1 SNES Function norm 3.917148384566e+01 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 2 SNES Function norm 3.373009701057e-01 40 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 4.835053852293e+01 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 1 SNES Function norm 4.835053852293e+01 0 SNES Function norm 1.287943764600e+01 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 1 SNES Function norm 1.128956057544e+01 41 TS dt 2.75903e-14 time 0.271565 copy! 0 SNES Function norm 1.669600342271e+01 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 1 SNES Function norm 1.472401994522e+01 42 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 5.552587859898e+01 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 1 SNES Function norm 2.907234222906e+01 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 2 SNES Function norm 1.772805631039e-01 43 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 7.725490953363e+00 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 1 SNES Function norm 5.415206018520e+00 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 2 SNES Function norm 2.952018920094e-02 44 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.042546652708e+00 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 1 SNES Function norm 7.553554533053e-01 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 2 SNES Function norm 6.213938456039e-02 45 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.726791649984e+00 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 1 SNES Function norm 1.726791649984e+00 0 SNES Function norm 4.569198217932e-01 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 1 SNES Function norm 5.060166636556e-02 46 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.648499891442e-01 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 1 SNES Function norm 4.303888962028e-01 47 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.205705397633e+00 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 1 SNES Function norm 1.205705397633e+00 0 SNES Function norm 3.168320704593e-01 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 1 SNES Function norm 1.059925414019e-01 48 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.008300299382e+00 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 1 SNES Function norm 5.842489315290e-01 49 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.508490594861e+00 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 1 SNES Function norm 4.745080825960e-01 50 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.014687630993e+00 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 1 SNES Function norm 5.592988640286e-01 51 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.841734166744e-01 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 1 SNES Function norm 9.841734166744e-01 0 SNES Function norm 5.290699913962e-01 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 1 SNES Function norm 4.072990107379e-01 52 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.293320470683e-01 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 1 SNES Function norm 3.293320470683e-01 0 SNES Function norm 6.190385444978e-01 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 1 SNES Function norm 3.612736786105e-01 53 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.245269204940e+00 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 1 SNES Function norm 5.113533173009e-01 54 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.607180712938e+00 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 1 SNES Function norm 1.607180712938e+00 0 SNES Function norm 3.045128849808e-01 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 1 SNES Function norm 2.935346973548e-01 55 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 8.467066865826e-01 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 1 SNES Function norm 5.502623367952e-01 56 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.717470692129e+00 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 1 SNES Function norm 4.045528183667e-01 57 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.544462481147e+00 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 1 SNES Function norm 1.293824512618e+00 58 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.191314292345e+00 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 1 SNES Function norm 2.617175498914e+00 59 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 5.489780310948e+00 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 1 SNES Function norm 2.949802369100e+00 60 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 4.631512871139e+00 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 1 SNES Function norm 2.195569697365e+00 61 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.662194684551e+00 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 1 SNES Function norm 1.662194684551e+00 0 SNES Function norm 2.329147999080e+00 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 1 SNES Function norm 1.548533538885e-01 62 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.019371357242e+00 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 1 SNES Function norm 2.813130743975e-01 63 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.849189158431e+00 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 1 SNES Function norm 1.849189158431e+00 0 SNES Function norm 1.106437703298e-01 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 1 SNES Function norm 1.071406781366e-01 64 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.036597476193e-01 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 1 SNES Function norm 1.036597476193e-01 0 SNES Function norm 1.081080288554e-01 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 1 SNES Function norm 4.280304180299e-02 65 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 2.901310268658e-02 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 1 SNES Function norm 2.901310268658e-02 0 SNES Function norm 6.115306732380e-02 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 1 SNES Function norm 6.115306732380e-02 0 SNES Function norm 8.316482031597e-02 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 1 SNES Function norm 8.316482031597e-02 0 SNES Function norm 3.114039374287e-01 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 1 SNES Function norm 1.476629082037e-01 66 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.741199964540e-01 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 1 SNES Function norm 2.741199964540e-01 0 SNES Function norm 1.363976538331e+00 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 1 SNES Function norm 3.620782473843e-01 67 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 2.101847656775e+00 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 1 SNES Function norm 5.837297362402e-01 68 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.691478479373e+00 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 1 SNES Function norm 1.646871085300e+00 69 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 4.140741966573e+00 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 1 SNES Function norm 1.278992537889e+00 70 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 4.925810717003e+00 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 1 SNES Function norm 4.757993340113e-01 71 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.014381910385e+00 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 1 SNES Function norm 1.014381910385e+00 0 SNES Function norm 5.247873040702e-01 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 1 SNES Function norm 5.128575534549e-01 72 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.463622536847e+00 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 1 SNES Function norm 1.463622536847e+00 0 SNES Function norm 5.760430307939e-01 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 1 SNES Function norm 4.031598199606e-01 73 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.304273554580e+00 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 1 SNES Function norm 5.901656814576e-01 74 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.806967885919e+00 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 1 SNES Function norm 1.806967885919e+00 0 SNES Function norm 3.525254737467e-01 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 1 SNES Function norm 2.971700641320e-01 75 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 9.276360188622e-01 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 1 SNES Function norm 3.629560323025e-01 76 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 1.281038880923e+00 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 1 SNES Function norm 1.281038880923e+00 0 SNES Function norm 2.062116742478e-01 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 1 SNES Function norm 4.076142877095e-02 77 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.970990313724e-01 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 1 SNES Function norm 2.970990313724e-01 0 SNES Function norm 3.372906354998e-01 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 1 SNES Function norm 2.538611106909e-01 78 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 6.929161829162e-01 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 1 SNES Function norm 4.533321906044e-01 79 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.430558276957e+00 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 1 SNES Function norm 3.940601459612e-01 80 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 7.524069011779e-01 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 1 SNES Function norm 6.651399131369e-02 81 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.121363306761e-01 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 1 SNES Function norm 9.121363306761e-01 0 SNES Function norm 1.548591006657e-01 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 1 SNES Function norm 1.548591006657e-01 0 SNES Function norm 3.363022877948e-01 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 1 SNES Function norm 2.439869058762e-01 82 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.602729745290e+00 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 1 SNES Function norm 1.007188504688e+00 83 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.461320490774e+00 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 1 SNES Function norm 2.147586456566e+00 84 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 6.341628072414e+00 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 1 SNES Function norm 3.965504408873e-01 85 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 9.612683816763e-01 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 1 SNES Function norm 9.612683816763e-01 0 SNES Function norm 7.029352448723e-01 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 1 SNES Function norm 2.593382931820e-01 86 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.105068885806e-01 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 1 SNES Function norm 3.105068885806e-01 0 SNES Function norm 4.981482205604e-01 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 1 SNES Function norm 4.981482205604e-01 0 SNES Function norm 7.138905144630e-01 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 1 SNES Function norm 3.696022475452e-01 87 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.789063973037e-01 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 1 SNES Function norm 2.343211520992e-01 88 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 3.552109100087e+00 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 1 SNES Function norm 2.944262139143e+00 89 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 7.582944894373e+00 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 1 SNES Function norm 2.445215958037e+00 90 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 3.577254475804e+00 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 1 SNES Function norm 9.478103979365e-02 91 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 3.398926029549e+00 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 1 SNES Function norm 2.854550344474e+00 92 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.846027110554e+00 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 1 SNES Function norm 4.478173098971e+00 93 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 9.479827241861e+00 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 1 SNES Function norm 2.980698125901e+00 94 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.583823756970e+00 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 1 SNES Function norm 1.929142014309e+00 95 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 1.257265283168e-01 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 1 SNES Function norm 1.257265283168e-01 0 SNES Function norm 2.389423098829e+00 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 1 SNES Function norm 1.440136831177e-01 96 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.101422245445e+00 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 1 SNES Function norm 7.435685625660e-01 97 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 3.062056964383e+00 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 1 SNES Function norm 3.062056964383e+00 0 SNES Function norm 1.640638339300e-01 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 1 SNES Function norm 7.185385907970e-03 98 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 1.778588005116e-01 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 1 SNES Function norm 1.179095579605e-01 99 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 2.512082795192e-01 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 1 SNES Function norm 2.512082795192e-01 0 SNES Function norm 8.541223283936e-02 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 1 SNES Function norm 6.589891795632e-02 100 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 2.174757993770e-01 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 1 SNES Function norm 2.174757993770e-01 0 SNES Function norm 3.006903173936e-02 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 1 SNES Function norm 3.006903173936e-02 0 SNES Function norm 5.071683999025e-02 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 1 SNES Function norm 4.513917663662e-02 101 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 1.261469809763e-01 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 1 SNES Function norm 9.629885791766e-02 102 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 3.223153986924e-01 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 1 SNES Function norm 1.444530545281e-01 103 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 3.103810286198e-01 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 1 SNES Function norm 3.103810286198e-01 0 SNES Function norm 1.063191577145e-01 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 1 SNES Function norm 1.280948192266e-02 104 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 1.183517461737e-01 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 1 SNES Function norm 3.338348178983e-02 105 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.506755043591e-01 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 1 SNES Function norm 1.506755043591e-01 0 SNES Function norm 1.678694983365e-02 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 1 SNES Function norm 1.678694983365e-02 0 SNES Function norm 2.211256020366e-02 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 1 SNES Function norm 2.211256020366e-02 0 SNES Function norm 6.319453516491e-02 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 1 SNES Function norm 6.319453516491e-02 0 SNES Function norm 1.993150558437e+00 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 1 SNES Function norm 7.832112166701e-01 106 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.629560288909e+00 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 1 SNES Function norm 1.672457508921e+00 107 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 4.062643571582e+00 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 1 SNES Function norm 9.036991011875e-01 108 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.105674791223e+00 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 1 SNES Function norm 6.003561441737e-01 109 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 3.053877992744e+00 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 1 SNES Function norm 3.053877992744e+00 0 SNES Function norm 5.829087412676e-01 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 1 SNES Function norm 4.254244459909e-01 110 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.180846267144e+00 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 1 SNES Function norm 3.911236882643e-01 111 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.532064854858e+00 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 1 SNES Function norm 1.532064854858e+00 0 SNES Function norm 1.764940070080e-01 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 1 SNES Function norm 1.706450714666e-01 112 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 4.407680008950e-01 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 1 SNES Function norm 4.407680008950e-01 0 SNES Function norm 1.064954076440e+00 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 1 SNES Function norm 2.964963371165e-01 113 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 8.622925709624e-01 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 1 SNES Function norm 8.622925709624e-01 0 SNES Function norm 2.612065167743e+00 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 1 SNES Function norm 1.331790654620e+00 114 TS dt 5.51805e-14 time 0.271565 copy! 0 SNES Function norm 3.606851714788e+00 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 1 SNES Function norm 9.013753678258e-01 115 TS dt 1.10361e-13 time 0.271565 copy! 0 SNES Function norm 3.227741616537e+00 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 1 SNES Function norm 7.734015655943e-01 116 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 2.393623504956e+00 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 1 SNES Function norm 4.969182786938e-01 117 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 2.596405006351e+00 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 1 SNES Function norm 2.596405006351e+00 0 SNES Function norm 4.974558747187e-01 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 1 SNES Function norm 2.909443551880e-01 118 TS dt 2.20722e-13 time 0.271565 copy! 0 SNES Function norm 1.088860095507e+00 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 1 SNES Function norm 6.203518870415e-01 119 TS dt 4.41444e-13 time 0.271565 copy! 0 SNES Function norm 1.828717001803e+00 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 1 SNES Function norm 9.407254842319e-01 120 TS dt 8.82888e-13 time 0.271565 copy! 0 SNES Function norm 2.970950728036e+00 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 1 SNES Function norm 2.277472194573e+00 121 TS dt 1.76578e-12 time 0.271565 copy! 0 SNES Function norm 6.655252327413e+00 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 1 SNES Function norm 3.860967926913e+00 122 TS dt 3.53155e-12 time 0.271565 copy! 0 SNES Function norm 8.141502067665e+00 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 1 SNES Function norm 6.426598300871e+00 123 TS dt 7.06311e-12 time 0.271565 copy! 0 SNES Function norm 1.224562198048e+01 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 1 SNES Function norm 5.283049961509e+00 124 TS dt 1.41262e-11 time 0.271565 copy! 0 SNES Function norm 5.780124419709e+00 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 1 SNES Function norm 1.227257133866e+00 125 TS dt 2.82524e-11 time 0.271565 copy! 0 SNES Function norm 5.063237678461e+00 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 1 SNES Function norm 4.907594073483e+00 126 TS dt 5.65048e-11 time 0.271565 copy! 0 SNES Function norm 8.280289521084e+00 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 1 SNES Function norm 3.515932015486e+00 127 TS dt 1.1301e-10 time 0.271565 copy! 0 SNES Function norm 2.638351398877e+00 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 1 SNES Function norm 6.341356656368e-01 128 TS dt 2.26019e-10 time 0.271565 copy! 0 SNES Function norm 4.327811744723e+00 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 1 SNES Function norm 1.736633786129e-01 129 TS dt 4.52039e-10 time 0.271565 copy! 0 SNES Function norm 3.450425343262e+00 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 1 SNES Function norm 1.373376592274e+00 130 TS dt 9.04078e-10 time 0.271565 copy! 0 SNES Function norm 4.603810494450e+00 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 1 SNES Function norm 1.142888930566e+00 ________________________________ From: Matthew Knepley > Sent: Tuesday, January 5, 2021 2:53 PM To: Sepideh Kavousi > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi > wrote: Dear Petsc Users I wrote a code in 2D solving PF equations for solidification of alloys. I have two variables in the equations 1-non-dimensional concentration (U) 2- order parameter (p). Everything is fine with this code. When I change the code to 3D, the code has convergence problems which I cannot solve it. I know the term which is causing problem is a term in the equation for U: (dp/dt)*(Pxx+Pyy+Pzz) where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, respectively. When I eliminate this term, the results of 2D and 3D are close to each other. But adding that term the time step should be reduced a lot to obtain convergence, otherwise the timestep does not converge. I have no idea how I should solve this problem. I solve the equations considering (I chose based on suggestions of Moose for solving Cahn-hillard) -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type ilu -sub_ksp_type preonly Would you please help me solve this issue? In order to see the solver you have we need the output of -ts_view and then to see what is happening during the run we need -ts_monitor -snes_monitor -snes_linesearch_monitor -ksp_monitor_true_residual Thanks, Matt Best, sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jan 5 20:52:32 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 5 Jan 2021 20:52:32 -0600 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: Ah, -snes_fd_color so it was already using finite differencing with coloring to compute the Jacobian which explains why the differences below are exactly zero. Implicit time-step schemes essentially add terms like I/dt to the Jacobian evaluation (and the function defining the ODE) so for tiny time-steps the nonlinear system gets easier and easier to solve (the nonlinear function becomes linear) But we didn't see that with your earlier run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps SNES still made no progress. It is hard to understand how this is possible, regardless of the problem you are solving. I would next run the code with valgrind to insure there are no issues of memory corruption or un-initialized data. How are you computing > (dp/dt)*(Pxx+Pyy+Pzz) That is, how are you computing Pxx etc? Are you using finite elements for the U and P model? Exactly what elements? Barry > On Jan 5, 2021, at 7:47 PM, Sepideh Kavousi wrote: > > The output for -snes_test_jacobian is given as : > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > ---------- Testing Jacobian ------------- > Run with -snes_test_jacobian_view and optionally -snes_test_jacobian to show difference > of hand-coded and finite difference Jacobian entries greater than . > Testing hand-coded Jacobian, if (for double precision runs) ||J - Jfd||_F/||J||_F is > O(1.e-8), the hand-coded Jacobian is probably correct. > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.673211216524e-03 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.067366268006e-18 true resid norm 1.921588081784e-14 ||r(i)||/||b|| 3.774355453546e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.668754123581e-03 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.604939763954e-18 true resid norm 4.960207918618e-14 ||r(i)||/||b|| 9.742768487084e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 4.172167491706e-04 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.588913006621e-20 true resid norm 1.948840699675e-14 ||r(i)||/||b|| 3.827884650536e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.043059511038e-04 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.104783140057e-19 true resid norm 5.333375230602e-14 ||r(i)||/||b|| 1.047573831159e-15 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 2.607659801680e-05 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.703990624142e-20 true resid norm 6.287819675470e-14 ||r(i)||/||b|| 1.235044425390e-15 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.519156394293e-06 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.557215852400e-21 true resid norm 1.840177917795e-14 ||r(i)||/||b|| 3.614450789618e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.629789529205e-06 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.173065981676e-22 true resid norm 1.530485325589e-14 ||r(i)||/||b|| 3.006157089528e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 4.074474092157e-07 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.739825180849e-22 true resid norm 2.686974828851e-14 ||r(i)||/||b|| 5.277717006551e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.018618539861e-07 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.045098765057e-22 true resid norm 5.010251971826e-14 ||r(i)||/||b|| 9.841064290920e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 2.546546360165e-08 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.287065233667e-24 true resid norm 1.149255127646e-14 ||r(i)||/||b|| 2.257350261311e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.366365906984e-09 true resid norm 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.164978605997e-24 true resid norm 2.417410255090e-14 ||r(i)||/||b|| 4.748242178566e-16 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.14.0, Sep 29, 2020 > [0]PETSC ERROR: /scratch/07065/tg863649/alloy-3D-cubic-9/one.out on a skylake named c456-001.stampede2.tacc.utexas.edu by tg863649 Tue Jan 5 18:35:45 2021 > [0]PETSC ERROR: Configure options --with-x=0 -with-pic --with-make-np=8 --download-petsc4py=1 --with-python-exec=/opt/apps/intel18/python3/3.7.0/bin/python3 --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" --with-packages-build-dir=/tmp/petsc-3.14/skylake --with-mpi=1 --with-mpi-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64 --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-plapack=1 --download-plapack --with-spai=1 --download-spai --with-sundials=1 --download-sundials --with-elemental=1 --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-fftw=1 --with-fftw-dir=/opt/apps/intel18/impi18_0/fftw3/3.3.8 --with-hdf5=1 --with-hdf5-dir=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64 --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 --with-debugging=no --LIBS= --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" > [0]PETSC ERROR: #1 TSStep() line 3773 in /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c > [0]PETSC ERROR: #2 TSSolve() line 4154 in /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c > [0]PETSC ERROR: #3 main() line 324 in /scratch/07065/tg863649/alloy-3D-cubic-9/one.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -ksp_gmres_restart 1001 > [0]PETSC ERROR: -ksp_monitor_true_residual > [0]PETSC ERROR: -ksp_type gmres > [0]PETSC ERROR: -pc_type lu > [0]PETSC ERROR: -snes_fd_color > [0]PETSC ERROR: -snes_linesearch_monitor > [0]PETSC ERROR: -snes_linesearch_type l2 > [0]PETSC ERROR: -snes_test_jacobian > [0]PETSC ERROR: -snes_type newtontr > [0]PETSC ERROR: -sub_ksp_type preonly > [0]PETSC ERROR: -sub_pc_type ilu > [0]PETSC ERROR: -ts_bdf_adapt > [0]PETSC ERROR: -ts_max_snes_failures -1 > [0]PETSC ERROR: -ts_monitor > [0]PETSC ERROR: -ts_type bdf > [0]PETSC ERROR: -ts_view > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov --------- > From: Matthew Knepley > > Sent: Tuesday, January 5, 2021 3:53 PM > To: Sepideh Kavousi > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 4:48 PM Sepideh Kavousi > wrote: > I have tried different pcs (lu,jacobi,sor,mg,asm,ksp,pbjacobi). they still do not solve the problem. the an example for lu: > -pc_type lu: > > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.761968940423e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.026000796457e-17 true resid norm 1.790914574372e-13 ||r(i)||/||b|| 3.050633943390e-15 > 1 SNES Function norm 5.808754845048e+01 > > So with a perfect linear solver, Newton is making no progress. That means that your initial guess is very far from the solution, but it could also mean > that your Jacobian does not match your residual, since it is not generating descent directions. Have you checked that it matches the finite difference > approximation? You can do this using -snes_test_jacobian (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESComputeJacobian.html ) > > Thanks, > > Matt > > 0 KSP preconditioned resid norm 1.191540511981e+06 true resid norm 5.808754845048e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.114882955273e-10 true resid norm 1.203029448994e-06 ||r(i)||/||b|| 2.071062527315e-08 > 2 SNES Function norm 5.794558475659e+01 > 0 KSP preconditioned resid norm 1.191526760369e+06 true resid norm 5.794558475659e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.166698266769e-09 true resid norm 1.059436818305e-06 ||r(i)||/||b|| 1.828330532439e-08 > 3 SNES Function norm 5.772046032228e+01 > 0 KSP preconditioned resid norm 1.191498793276e+06 true resid norm 5.772046032228e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.135175980555e-09 true resid norm 1.230437568859e-06 ||r(i)||/||b|| 2.131718219136e-08 > 4 SNES Function norm 5.750822623807e+01 > 0 KSP preconditioned resid norm 1.191443279124e+06 true resid norm 5.750822623807e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.905670672745e-10 true resid norm 1.110684454395e-06 ||r(i)||/||b|| 1.931348829639e-08 > 5 SNES Function norm 5.750395740069e+01 > 0 KSP preconditioned resid norm 1.191433462635e+06 true resid norm 5.750395740069e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.777205269843e-10 true resid norm 1.430076525401e-06 ||r(i)||/||b|| 2.486918448822e-08 > 6 SNES Function norm 5.750395740069e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 4.405989777785e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.185260514636e-18 true resid norm 2.397998208837e-14 ||r(i)||/||b|| 4.084736836000e-16 > 1 SNES Function norm 3.428067045216e+01 > 0 KSP preconditioned resid norm 1.895265798836e+06 true resid norm 3.428067045216e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.403775884750e-09 true resid norm 8.602063630559e-06 ||r(i)||/||b|| 2.509304373893e-07 > 2 SNES Function norm 3.428067045216e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.101564184740e-03 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.380439828314e-19 true resid norm 3.289971303359e-14 ||r(i)||/||b|| 5.604118853253e-16 > 1 SNES Function norm 3.330623063627e+01 > 0 KSP preconditioned resid norm 6.892935831006e-04 true resid norm 3.330623063627e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.234145873951e-19 true resid norm 1.823438085202e-14 ||r(i)||/||b|| 5.474765683081e-16 > 2 SNES Function norm 1.228168587373e+01 > 0 KSP preconditioned resid norm 2.442179927376e-04 true resid norm 1.228168587373e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.114351195683e-19 true resid norm 1.328341389902e-14 ||r(i)||/||b|| 1.081562745993e-15 > 3 SNES Function norm 8.377002138765e+00 > 0 KSP preconditioned resid norm 1.670901807768e-04 true resid norm 8.377002138765e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.056553283627e-19 true resid norm 1.539291322344e-14 ||r(i)||/||b|| 1.837520507750e-15 > 4 SNES Function norm 6.871259668846e-01 > 0 KSP preconditioned resid norm 7.849703293659e-06 true resid norm 6.871259668846e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.813692242300e-21 true resid norm 3.443983247611e-16 ||r(i)||/||b|| 5.012157033194e-16 > 5 SNES Function norm 4.598894985660e-01 > 0 SNES Function norm 4.609452800861e-01 > 0 KSP preconditioned resid norm 4.482807772478e+00 true resid norm 4.609452800861e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.779043025722e-15 true resid norm 9.363790606967e-11 ||r(i)||/||b|| 2.031432148566e-10 > 1 SNES Function norm 3.909615632676e-01 > 1 TS dt 1.25e-05 time 6.25e-06 > copy! > 0 SNES Function norm 1.083064424363e+00 > 0 KSP preconditioned resid norm 2.899372173019e+02 true resid norm 1.083064424363e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.843039617595e-13 true resid norm 2.031204561200e-09 ||r(i)||/||b|| 1.875423581006e-09 > 1 SNES Function norm 1.083064424363e+00 > 0 SNES Function norm 4.081254017393e-01 > 0 KSP preconditioned resid norm 1.134610736819e+01 true resid norm 4.081254017393e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.857284280292e-15 true resid norm 1.950178080787e-10 ||r(i)||/||b|| 4.778379567837e-10 > 1 SNES Function norm 3.852294258555e-01 > 2 TS dt 6.25e-06 time 9.375e-06 > copy! > 0 SNES Function norm 3.855090631551e-01 > 0 KSP preconditioned resid norm 5.807020984426e+01 true resid norm 3.855090631551e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.595611115758e-14 true resid norm 6.458636528724e-10 ||r(i)||/||b|| 1.675352707888e-09 > 1 SNES Function norm 3.855090631551e-01 > 0 SNES Function norm 3.894882915800e-01 > 0 KSP preconditioned resid norm 3.545691207399e+00 true resid norm 3.894882915800e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.936099793899e-15 true resid norm 1.237085076854e-10 ||r(i)||/||b|| 3.176180397711e-10 > 1 SNES Function norm 3.826734119825e-01 > 3 TS dt 3.125e-06 time 1.09375e-05 > copy! > 0 SNES Function norm 3.846635056055e-01 > 0 KSP preconditioned resid norm 1.270632337662e+01 true resid norm 3.846635056055e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.538849751267e-15 true resid norm 2.462963992263e-10 ||r(i)||/||b|| 6.402905283117e-10 > 1 SNES Function norm 3.846635056055e-01 > 0 SNES Function norm 3.838572028974e-01 > 0 KSP preconditioned resid norm 9.750671428246e-01 true resid norm 3.838572028974e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.289810958767e-15 true resid norm 5.932884665420e-11 ||r(i)||/||b|| 1.545596805436e-10 > 1 SNES Function norm 3.823876176903e-01 > 4 TS dt 1.5625e-06 time 1.17188e-05 > copy! > 0 SNES Function norm 3.842190105197e-01 > 0 KSP preconditioned resid norm 2.949895830433e+00 true resid norm 3.842190105197e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.544123722663e-15 true resid norm 1.184005133464e-10 ||r(i)||/||b|| 3.081589148499e-10 > 1 SNES Function norm 3.842190105197e-01 > 0 SNES Function norm 3.825899662960e-01 > 0 KSP preconditioned resid norm 2.548468293806e-01 true resid norm 3.825899662960e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.086200228021e-16 true resid norm 2.882610131465e-11 ||r(i)||/||b|| 7.534463486778e-11 > 1 SNES Function norm 3.825019685537e-01 > 5 TS dt 7.8125e-07 time 1.21094e-05 > copy! > 0 SNES Function norm 3.827182409606e-01 > 0 KSP preconditioned resid norm 7.091231452893e-01 true resid norm 3.827182409606e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.614097858359e-16 true resid norm 5.440482894192e-11 ||r(i)||/||b|| 1.421537390153e-10 > 1 SNES Function norm 3.827182409606e-01 > 0 SNES Function norm 3.825683488225e-01 > 0 KSP preconditioned resid norm 6.509635329694e-02 true resid norm 3.825683488225e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.255654660173e-17 true resid norm 1.410340803479e-11 ||r(i)||/||b|| 3.686506758386e-11 > 1 SNES Function norm 3.825462954733e-01 > 6 TS dt 3.90625e-07 time 1.23047e-05 > copy! > 0 SNES Function norm 3.826632731069e-01 > 0 KSP preconditioned resid norm 1.737397651071e-01 true resid norm 3.826632731069e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.095925875164e-16 true resid norm 2.780031944787e-11 ||r(i)||/||b|| 7.264956268774e-11 > 1 SNES Function norm 3.826632731069e-01 > 0 SNES Function norm 3.825751343810e-01 > 0 KSP preconditioned resid norm 1.644741829353e-02 true resid norm 3.825751343810e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.211213666062e-18 true resid norm 7.515828526583e-12 ||r(i)||/||b|| 1.964536597169e-11 > 1 SNES Function norm 3.825729851167e-01 > 7 TS dt 1.95313e-07 time 1.24023e-05 > copy! > 0 SNES Function norm 3.826356173729e-01 > 0 KSP preconditioned resid norm 4.299238585205e-02 true resid norm 3.826356173729e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.260847149806e-17 true resid norm 1.287166330766e-11 ||r(i)||/||b|| 3.363948028683e-11 > 1 SNES Function norm 3.826356173729e-01 > 0 SNES Function norm 3.825855754640e-01 > 0 KSP preconditioned resid norm 4.133474793122e-03 true resid norm 3.825855754640e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.077297138396e-18 true resid norm 3.792689943805e-12 ||r(i)||/||b|| 9.913311392374e-12 > 1 SNES Function norm 3.825855754640e-01 > 0 SNES Function norm 3.825755706234e-01 > 0 KSP preconditioned resid norm 3.705095771277e-04 true resid norm 3.825755706234e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.848267031641e-19 true resid norm 1.079462463835e-12 ||r(i)||/||b|| 2.821566630813e-12 > 1 SNES Function norm 3.825755706234e-01 > 0 SNES Function norm 3.825734391910e-01 > 0 KSP preconditioned resid norm 2.690993850525e-05 true resid norm 3.825734391910e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.585015168707e-21 true resid norm 3.091320932665e-13 ||r(i)||/||b|| 8.080333384362e-13 > 1 SNES Function norm 3.825734391910e-01 > 0 SNES Function norm 3.825729114845e-01 > 0 KSP preconditioned resid norm 1.757556306807e-06 true resid norm 3.825729114845e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.987700736384e-22 true resid norm 8.038113355816e-14 ||r(i)||/||b|| 2.101067042260e-13 > 1 SNES Function norm 3.825729114845e-01 > 0 SNES Function norm 3.825565841303e-01 > 0 KSP preconditioned resid norm 1.103248802240e-07 true resid norm 3.825565841303e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.504795613889e-22 true resid norm 1.879813159274e-14 ||r(i)||/||b|| 4.913817294630e-14 > 1 SNES Function norm 3.825565841303e-01 > 0 SNES Function norm 3.825547574266e-01 > 0 KSP preconditioned resid norm 6.887463339138e-09 true resid norm 3.825547574266e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.559974044704e-23 true resid norm 5.311723680631e-15 ||r(i)||/||b|| 1.388487158378e-14 > 1 SNES Function norm 3.825547574266e-01 > 0 SNES Function norm 3.825908198230e-01 > 0 KSP preconditioned resid norm 4.314157283909e-10 true resid norm 3.825908198230e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.248846716324e-24 true resid norm 1.634240020842e-15 ||r(i)||/||b|| 4.271508714187e-15 > 1 SNES Function norm 3.825908198230e-01 > 0 SNES Function norm 3.823046631992e-01 > 0 KSP preconditioned resid norm 1.477599397951e-11 true resid norm 3.823046631992e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.674742673343e-25 true resid norm 4.342823046268e-15 ||r(i)||/||b|| 1.135958690623e-14 > 1 SNES Function norm 3.823046631992e-01 > 0 SNES Function norm 4.050006682433e-01 > 0 KSP preconditioned resid norm 4.744618612191e-12 true resid norm 4.050006682433e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.185835350170e-26 true resid norm 2.742859197646e-15 ||r(i)||/||b|| 6.772480671558e-15 > 1 SNES Function norm 4.050006682433e-01 > 0 SNES Function norm 4.420555752951e-01 > 0 KSP preconditioned resid norm 2.570862477765e-13 true resid norm 4.420555752951e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.558628106533e-27 true resid norm 2.669513100096e-15 ||r(i)||/||b|| 6.038863096149e-15 > 1 SNES Function norm 3.053219162874e-01 > 8 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.938107171164e-01 > 0 KSP preconditioned resid norm 1.152686085246e-13 true resid norm 2.938107171164e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.059363368490e-28 true resid norm 1.486663802399e-15 ||r(i)||/||b|| 5.059937285439e-15 > 1 SNES Function norm 1.879261738365e-01 > 9 TS dt 7.45058e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.598740119900e-01 > 0 KSP preconditioned resid norm 4.386389243102e-12 true resid norm 2.598740119900e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.765692021861e-27 true resid norm 4.362311845294e-16 ||r(i)||/||b|| 1.678625658599e-15 > 1 SNES Function norm 2.598740119900e-01 > 0 SNES Function norm 2.359036347057e-01 > 0 KSP preconditioned resid norm 3.416321738725e-13 true resid norm 2.359036347057e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.983917914615e-27 true resid norm 2.031818641817e-15 ||r(i)||/||b|| 8.612917916048e-15 > 1 SNES Function norm 1.708032078859e-01 > 10 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 5.285427862819e-01 > 0 KSP preconditioned resid norm 1.592064460142e-12 true resid norm 5.285427862819e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.163339775654e-26 true resid norm 3.880080925123e-15 ||r(i)||/||b|| 7.341091442033e-15 > 1 SNES Function norm 5.285427862819e-01 > 0 SNES Function norm 1.646935190988e-01 > 0 KSP preconditioned resid norm 5.824112147206e-14 true resid norm 1.646935190988e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.820310421468e-29 true resid norm 7.168182199430e-17 ||r(i)||/||b|| 4.352437326407e-16 > 1 SNES Function norm 1.621698143957e-01 > > > > > From: Matthew Knepley > > Sent: Tuesday, January 5, 2021 3:25 PM > To: Sepideh Kavousi > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi > wrote: > Dear Matt, > following is the output for the first 130 timesteps. It continues the run with the same small timestep > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 > 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 > 1 SNES Function norm 5.870480277455e+01 > 0 SNES Function norm 5.870480198099e+01 > 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 > > You can see here that the linear solver is very bad, which might contribute to Newton stagnating. While you are starting out, I would recommend > > -pc_type lu > > Once you have the nonlinear system converging, then you can try different linear solvers to speed things up. > > Thanks, > > Matt > > 1 SNES Function norm 5.870165343582e+01 > 1 TS dt 0.0002 time 0.0001 > copy! > 0 SNES Function norm 5.869324856421e+01 > 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 > 1 SNES Function norm 5.869298823911e+01 > 2 TS dt 0.0004 time 0.0003 > copy! > 0 SNES Function norm 5.867574766362e+01 > 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 > 1 SNES Function norm 5.867531973582e+01 > 3 TS dt 0.0008 time 0.0007 > copy! > 0 SNES Function norm 5.864011322220e+01 > 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 > 1 SNES Function norm 5.863990526788e+01 > 4 TS dt 0.0016 time 0.0015 > copy! > 0 SNES Function norm 5.856903590475e+01 > 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 > 1 SNES Function norm 5.856893794226e+01 > 5 TS dt 0.0032 time 0.0031 > copy! > 0 SNES Function norm 5.842679564768e+01 > 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 > 1 SNES Function norm 5.842675271515e+01 > 6 TS dt 0.0064 time 0.0063 > copy! > 0 SNES Function norm 5.814352339598e+01 > 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 > 1 SNES Function norm 5.814350805660e+01 > 7 TS dt 0.0128 time 0.0127 > copy! > 0 SNES Function norm 5.760173270961e+01 > 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 > 1 SNES Function norm 5.760173134258e+01 > 8 TS dt 0.0256 time 0.0255 > copy! > 0 SNES Function norm 5.683398372141e+01 > 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 > 1 SNES Function norm 5.683398372141e+01 > 0 SNES Function norm 5.747024606058e+01 > 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 > 1 SNES Function norm 5.747023898114e+01 > 9 TS dt 0.00758395 time 0.0319 > copy! > 0 SNES Function norm 5.725811890845e+01 > 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 > 1 SNES Function norm 5.725811208900e+01 > 10 TS dt 0.0151679 time 0.039484 > copy! > 0 SNES Function norm 5.684339053584e+01 > 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 > 1 SNES Function norm 5.684338688259e+01 > 11 TS dt 0.0303358 time 0.0546519 > copy! > 0 SNES Function norm 5.605303343054e+01 > 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 > 1 SNES Function norm 5.605303164316e+01 > 12 TS dt 0.0606716 time 0.0849877 > copy! > 0 SNES Function norm 5.463652619423e+01 > 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 > 1 SNES Function norm 5.463652534643e+01 > 13 TS dt 0.121343 time 0.145659 > copy! > 0 SNES Function norm 5.251966690761e+01 > 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 > 1 SNES Function norm 5.251966654744e+01 > 14 TS dt 0.242686 time 0.267003 > copy! > 0 SNES Function norm 5.150151326391e+01 > 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 > 1 SNES Function norm 5.150151326391e+01 > 0 SNES Function norm 5.184908859688e+01 > 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 > 1 SNES Function norm 5.184908859688e+01 > 0 SNES Function norm 5.232682115621e+01 > 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 > 1 SNES Function norm 5.232682115621e+01 > 0 SNES Function norm 5.246983753576e+01 > 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 > 1 SNES Function norm 5.246983737815e+01 > 15 TS dt 0.00758395 time 0.270794 > copy! > 0 SNES Function norm 5.237345394852e+01 > 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 > 1 SNES Function norm 5.237345394852e+01 > 0 SNES Function norm 5.244535357001e+01 > 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 > 1 SNES Function norm 5.244535357001e+01 > 0 SNES Function norm 5.246369218939e+01 > 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 > 1 SNES Function norm 5.246369181668e+01 > 16 TS dt 0.000947994 time 0.271268 > copy! > 0 SNES Function norm 5.245144714661e+01 > 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 > 1 SNES Function norm 5.245144714661e+01 > 0 SNES Function norm 5.246062448755e+01 > 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 > 1 SNES Function norm 5.246062426384e+01 > 17 TS dt 0.000473997 time 0.271505 > copy! > 0 SNES Function norm 5.245449911597e+01 > 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 > 1 SNES Function norm 5.245449911597e+01 > 0 SNES Function norm 5.245909110130e+01 > 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 > 1 SNES Function norm 5.245909110130e+01 > 0 SNES Function norm 5.246024056684e+01 > 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 > 1 SNES Function norm 5.246024055663e+01 > 18 TS dt 5.92496e-05 time 0.271535 > copy! > 0 SNES Function norm 5.245947411014e+01 > 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 > 1 SNES Function norm 5.245947411014e+01 > 0 SNES Function norm 5.246004891756e+01 > 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 > 1 SNES Function norm 5.246004890902e+01 > 19 TS dt 2.96248e-05 time 0.27155 > copy! > 0 SNES Function norm 5.245966566276e+01 > 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 > 1 SNES Function norm 5.245966566276e+01 > 0 SNES Function norm 5.245995306375e+01 > 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 > 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 > 1 SNES Function norm 5.245977670896e+01 > 20 TS dt 1.48124e-05 time 0.271557 > copy! > 0 SNES Function norm 5.245977453999e+01 > 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 > 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 > 1 SNES Function norm 5.245977453999e+01 > 0 SNES Function norm 5.245973715661e+01 > 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 > 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 > 1 SNES Function norm 5.245972630220e+01 > 21 TS dt 7.4062e-06 time 0.271561 > copy! > 0 SNES Function norm 5.245962991861e+01 > 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 > 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 > 1 SNES Function norm 5.245962991861e+01 > 0 SNES Function norm 5.245970348930e+01 > 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 > 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 > 1 SNES Function norm 5.245969886119e+01 > 22 TS dt 3.7031e-06 time 0.271563 > copy! > 0 SNES Function norm 5.245964973088e+01 > 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 > 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 > 1 SNES Function norm 5.245964973088e+01 > 0 SNES Function norm 5.245968740629e+01 > 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 > 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 > 1 SNES Function norm 5.245968066039e+01 > 23 TS dt 1.85155e-06 time 0.271564 > copy! > 0 SNES Function norm 5.245966664362e+01 > 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 > 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 > 1 SNES Function norm 5.245966664362e+01 > 0 SNES Function norm 5.245967459813e+01 > 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 > 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 > 1 SNES Function norm 5.245967167916e+01 > 24 TS dt 9.25775e-07 time 0.271564 > copy! > 0 SNES Function norm 5.245972527444e+01 > 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 > 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 > 1 SNES Function norm 5.245972527444e+01 > 0 SNES Function norm 5.245966286541e+01 > 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 > 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 > 1 SNES Function norm 5.245964735626e+01 > 25 TS dt 4.62888e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245965045260e+01 > 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 > 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 > 1 SNES Function norm 5.245965045260e+01 > 0 SNES Function norm 5.245964737315e+01 > 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 > 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 > 1 SNES Function norm 5.245953907407e+01 > 26 TS dt 2.31444e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245952669603e+01 > 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 > 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 > 1 SNES Function norm 5.245952669603e+01 > 0 SNES Function norm 5.245955619349e+01 > 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 > 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 > 1 SNES Function norm 5.245916156502e+01 > 27 TS dt 1.15722e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245916087517e+01 > 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 > 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 > 1 SNES Function norm 5.245916087517e+01 > 0 SNES Function norm 5.245922267220e+01 > 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 > 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 > 1 SNES Function norm 5.245868047885e+01 > 28 TS dt 5.7861e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245971666219e+01 > 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 > 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 > 1 SNES Function norm 5.245971666219e+01 > 0 SNES Function norm 5.245866772896e+01 > 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 > 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 > 1 SNES Function norm 5.245862997528e+01 > 29 TS dt 2.89305e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245864268828e+01 > 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 > 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 > 1 SNES Function norm 5.245864268828e+01 > 0 SNES Function norm 5.245863432005e+01 > 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 > 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 > 1 SNES Function norm 5.245668231675e+01 > 30 TS dt 1.44652e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245690451230e+01 > 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 > 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 > 1 SNES Function norm 5.245690451230e+01 > 0 SNES Function norm 5.245696584491e+01 > 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 > 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 > 1 SNES Function norm 5.245647736964e+01 > 31 TS dt 7.23262e-09 time 0.271565 > copy! > 0 SNES Function norm 5.245669639739e+01 > 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 > 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 > 1 SNES Function norm 5.245669639739e+01 > 0 SNES Function norm 5.245652990872e+01 > 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 > 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 > 1 SNES Function norm 5.245350296758e+01 > 32 TS dt 3.61631e-09 time 0.271565 > copy! > 0 SNES Function norm 5.246640623267e+01 > 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 > 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 > 1 SNES Function norm 5.246640623267e+01 > 0 SNES Function norm 5.245276947886e+01 > 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 > 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 > 1 SNES Function norm 5.245276947886e+01 > 0 SNES Function norm 5.245277054267e+01 > 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 > 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 > 1 SNES Function norm 5.240788106409e+01 > 33 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 5.240539051979e+01 > 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 > 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 > 1 SNES Function norm 5.240539051979e+01 > 0 SNES Function norm 5.240989970820e+01 > 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 > 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 > 1 SNES Function norm 5.240989970820e+01 > 0 SNES Function norm 5.241000025070e+01 > 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 > 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 > 1 SNES Function norm 5.228672678486e+01 > 34 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 5.229325617508e+01 > 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 > 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 > 1 SNES Function norm 5.229325617508e+01 > 0 SNES Function norm 5.228920001144e+01 > 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 > 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 > 1 SNES Function norm 5.228920001144e+01 > 0 SNES Function norm 5.228865204069e+01 > 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 > 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 > 1 SNES Function norm 5.175201563991e+01 > 35 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.168638826028e+01 > 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 > 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 > 1 SNES Function norm 5.168638826028e+01 > 0 SNES Function norm 5.178402875676e+01 > 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 > 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 > 1 SNES Function norm 5.178402875676e+01 > 0 SNES Function norm 5.178806568852e+01 > 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 > 1 SNES Function norm 4.925818844220e+01 > 36 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 5.270383537204e+01 > 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 > 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 > 1 SNES Function norm 5.270383537204e+01 > 0 SNES Function norm 4.874520711594e+01 > 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 > 1 SNES Function norm 4.874520711594e+01 > 0 SNES Function norm 4.879679718326e+01 > 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 > 1 SNES Function norm 4.872964148564e+01 > 37 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.866691749862e+01 > 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 > 1 SNES Function norm 4.866382095049e+01 > 38 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.893392986820e+01 > 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 > 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 > 1 SNES Function norm 4.893392986820e+01 > 0 SNES Function norm 4.865470718806e+01 > 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 > 1 SNES Function norm 4.843333577010e+01 > 39 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.842612728466e+01 > 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 > 1 SNES Function norm 4.842612728466e+01 > 0 SNES Function norm 4.840969985006e+01 > 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 > 1 SNES Function norm 3.917148384566e+01 > 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 > 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 > 2 SNES Function norm 3.373009701057e-01 > 40 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 4.835053852293e+01 > 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 > 1 SNES Function norm 4.835053852293e+01 > 0 SNES Function norm 1.287943764600e+01 > 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 > 1 SNES Function norm 1.128956057544e+01 > 41 TS dt 2.75903e-14 time 0.271565 > copy! > 0 SNES Function norm 1.669600342271e+01 > 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 > 1 SNES Function norm 1.472401994522e+01 > 42 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 5.552587859898e+01 > 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 > 1 SNES Function norm 2.907234222906e+01 > 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 > 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 > 2 SNES Function norm 1.772805631039e-01 > 43 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 7.725490953363e+00 > 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 > 1 SNES Function norm 5.415206018520e+00 > 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 > 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 > 2 SNES Function norm 2.952018920094e-02 > 44 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.042546652708e+00 > 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 > 1 SNES Function norm 7.553554533053e-01 > 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 > 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 > 2 SNES Function norm 6.213938456039e-02 > 45 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.726791649984e+00 > 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 > 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 > 1 SNES Function norm 1.726791649984e+00 > 0 SNES Function norm 4.569198217932e-01 > 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 > 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 > 1 SNES Function norm 5.060166636556e-02 > 46 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.648499891442e-01 > 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 > 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 > 1 SNES Function norm 4.303888962028e-01 > 47 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.205705397633e+00 > 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 > 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 > 1 SNES Function norm 1.205705397633e+00 > 0 SNES Function norm 3.168320704593e-01 > 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 > 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 > 1 SNES Function norm 1.059925414019e-01 > 48 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.008300299382e+00 > 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 > 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 > 1 SNES Function norm 5.842489315290e-01 > 49 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.508490594861e+00 > 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 > 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 > 1 SNES Function norm 4.745080825960e-01 > 50 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.014687630993e+00 > 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 > 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 > 1 SNES Function norm 5.592988640286e-01 > 51 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.841734166744e-01 > 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 > 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 > 1 SNES Function norm 9.841734166744e-01 > 0 SNES Function norm 5.290699913962e-01 > 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 > 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 > 1 SNES Function norm 4.072990107379e-01 > 52 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.293320470683e-01 > 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 > 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 > 1 SNES Function norm 3.293320470683e-01 > 0 SNES Function norm 6.190385444978e-01 > 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 > 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 > 1 SNES Function norm 3.612736786105e-01 > 53 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.245269204940e+00 > 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 > 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 > 1 SNES Function norm 5.113533173009e-01 > 54 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.607180712938e+00 > 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 > 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 > 1 SNES Function norm 1.607180712938e+00 > 0 SNES Function norm 3.045128849808e-01 > 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 > 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 > 1 SNES Function norm 2.935346973548e-01 > 55 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 8.467066865826e-01 > 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 > 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 > 1 SNES Function norm 5.502623367952e-01 > 56 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.717470692129e+00 > 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 > 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 > 1 SNES Function norm 4.045528183667e-01 > 57 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.544462481147e+00 > 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 > 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 > 1 SNES Function norm 1.293824512618e+00 > 58 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.191314292345e+00 > 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 > 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 > 1 SNES Function norm 2.617175498914e+00 > 59 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.489780310948e+00 > 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 > 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 > 1 SNES Function norm 2.949802369100e+00 > 60 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 4.631512871139e+00 > 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 > 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 > 1 SNES Function norm 2.195569697365e+00 > 61 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.662194684551e+00 > 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 > 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 > 1 SNES Function norm 1.662194684551e+00 > 0 SNES Function norm 2.329147999080e+00 > 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 > 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 > 1 SNES Function norm 1.548533538885e-01 > 62 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.019371357242e+00 > 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 > 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 > 1 SNES Function norm 2.813130743975e-01 > 63 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.849189158431e+00 > 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 > 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 > 1 SNES Function norm 1.849189158431e+00 > 0 SNES Function norm 1.106437703298e-01 > 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 > 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 > 1 SNES Function norm 1.071406781366e-01 > 64 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.036597476193e-01 > 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 > 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 > 1 SNES Function norm 1.036597476193e-01 > 0 SNES Function norm 1.081080288554e-01 > 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 > 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 > 1 SNES Function norm 4.280304180299e-02 > 65 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 2.901310268658e-02 > 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 > 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 > 1 SNES Function norm 2.901310268658e-02 > 0 SNES Function norm 6.115306732380e-02 > 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 > 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 > 1 SNES Function norm 6.115306732380e-02 > 0 SNES Function norm 8.316482031597e-02 > 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 > 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 > 1 SNES Function norm 8.316482031597e-02 > 0 SNES Function norm 3.114039374287e-01 > 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 > 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 > 1 SNES Function norm 1.476629082037e-01 > 66 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.741199964540e-01 > 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 > 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 > 1 SNES Function norm 2.741199964540e-01 > 0 SNES Function norm 1.363976538331e+00 > 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 > 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 > 1 SNES Function norm 3.620782473843e-01 > 67 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 2.101847656775e+00 > 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 > 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 > 1 SNES Function norm 5.837297362402e-01 > 68 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.691478479373e+00 > 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 > 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 > 1 SNES Function norm 1.646871085300e+00 > 69 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 4.140741966573e+00 > 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 > 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 > 1 SNES Function norm 1.278992537889e+00 > 70 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 4.925810717003e+00 > 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 > 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 > 1 SNES Function norm 4.757993340113e-01 > 71 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.014381910385e+00 > 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 > 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 > 1 SNES Function norm 1.014381910385e+00 > 0 SNES Function norm 5.247873040702e-01 > 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 > 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 > 1 SNES Function norm 5.128575534549e-01 > 72 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.463622536847e+00 > 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 > 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 > 1 SNES Function norm 1.463622536847e+00 > 0 SNES Function norm 5.760430307939e-01 > 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 > 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 > 1 SNES Function norm 4.031598199606e-01 > 73 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.304273554580e+00 > 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 > 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 > 1 SNES Function norm 5.901656814576e-01 > 74 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.806967885919e+00 > 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 > 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 > 1 SNES Function norm 1.806967885919e+00 > 0 SNES Function norm 3.525254737467e-01 > 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 > 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 > 1 SNES Function norm 2.971700641320e-01 > 75 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 9.276360188622e-01 > 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 > 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 > 1 SNES Function norm 3.629560323025e-01 > 76 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.281038880923e+00 > 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 > 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 > 1 SNES Function norm 1.281038880923e+00 > 0 SNES Function norm 2.062116742478e-01 > 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 > 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 > 1 SNES Function norm 4.076142877095e-02 > 77 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970990313724e-01 > 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 > 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 > 1 SNES Function norm 2.970990313724e-01 > 0 SNES Function norm 3.372906354998e-01 > 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 > 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 > 1 SNES Function norm 2.538611106909e-01 > 78 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 6.929161829162e-01 > 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 > 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 > 1 SNES Function norm 4.533321906044e-01 > 79 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.430558276957e+00 > 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 > 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 > 1 SNES Function norm 3.940601459612e-01 > 80 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 7.524069011779e-01 > 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 > 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 > 1 SNES Function norm 6.651399131369e-02 > 81 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.121363306761e-01 > 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 > 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 > 1 SNES Function norm 9.121363306761e-01 > 0 SNES Function norm 1.548591006657e-01 > 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 > 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 > 1 SNES Function norm 1.548591006657e-01 > 0 SNES Function norm 3.363022877948e-01 > 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 > 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 > 1 SNES Function norm 2.439869058762e-01 > 82 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.602729745290e+00 > 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 > 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 > 1 SNES Function norm 1.007188504688e+00 > 83 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.461320490774e+00 > 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 > 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 > 1 SNES Function norm 2.147586456566e+00 > 84 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 6.341628072414e+00 > 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 > 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 > 1 SNES Function norm 3.965504408873e-01 > 85 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.612683816763e-01 > 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 > 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 > 1 SNES Function norm 9.612683816763e-01 > 0 SNES Function norm 7.029352448723e-01 > 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 > 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 > 1 SNES Function norm 2.593382931820e-01 > 86 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.105068885806e-01 > 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 > 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 > 1 SNES Function norm 3.105068885806e-01 > 0 SNES Function norm 4.981482205604e-01 > 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 > 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 > 1 SNES Function norm 4.981482205604e-01 > 0 SNES Function norm 7.138905144630e-01 > 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 > 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 > 1 SNES Function norm 3.696022475452e-01 > 87 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.789063973037e-01 > 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 > 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 > 1 SNES Function norm 2.343211520992e-01 > 88 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.552109100087e+00 > 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 > 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 > 1 SNES Function norm 2.944262139143e+00 > 89 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 7.582944894373e+00 > 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 > 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 > 1 SNES Function norm 2.445215958037e+00 > 90 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.577254475804e+00 > 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 > 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 > 1 SNES Function norm 9.478103979365e-02 > 91 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 3.398926029549e+00 > 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 > 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 > 1 SNES Function norm 2.854550344474e+00 > 92 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.846027110554e+00 > 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 > 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 > 1 SNES Function norm 4.478173098971e+00 > 93 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 9.479827241861e+00 > 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 > 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 > 1 SNES Function norm 2.980698125901e+00 > 94 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.583823756970e+00 > 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 > 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 > 1 SNES Function norm 1.929142014309e+00 > 95 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.257265283168e-01 > 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 > 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 > 1 SNES Function norm 1.257265283168e-01 > 0 SNES Function norm 2.389423098829e+00 > 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 > 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 > 1 SNES Function norm 1.440136831177e-01 > 96 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.101422245445e+00 > 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 > 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 > 1 SNES Function norm 7.435685625660e-01 > 97 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 3.062056964383e+00 > 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 > 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 > 1 SNES Function norm 3.062056964383e+00 > 0 SNES Function norm 1.640638339300e-01 > 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 > 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 > 1 SNES Function norm 7.185385907970e-03 > 98 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.778588005116e-01 > 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 > 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 > 1 SNES Function norm 1.179095579605e-01 > 99 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 2.512082795192e-01 > 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 > 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 > 1 SNES Function norm 2.512082795192e-01 > 0 SNES Function norm 8.541223283936e-02 > 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 > 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 > 1 SNES Function norm 6.589891795632e-02 > 100 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.174757993770e-01 > 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 > 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 > 1 SNES Function norm 2.174757993770e-01 > 0 SNES Function norm 3.006903173936e-02 > 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 > 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 > 1 SNES Function norm 3.006903173936e-02 > 0 SNES Function norm 5.071683999025e-02 > 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 > 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 > 1 SNES Function norm 4.513917663662e-02 > 101 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.261469809763e-01 > 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 > 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 > 1 SNES Function norm 9.629885791766e-02 > 102 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.223153986924e-01 > 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 > 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 > 1 SNES Function norm 1.444530545281e-01 > 103 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 3.103810286198e-01 > 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 > 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 > 1 SNES Function norm 3.103810286198e-01 > 0 SNES Function norm 1.063191577145e-01 > 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 > 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 > 1 SNES Function norm 1.280948192266e-02 > 104 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 1.183517461737e-01 > 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 > 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 > 1 SNES Function norm 3.338348178983e-02 > 105 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.506755043591e-01 > 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 > 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 > 1 SNES Function norm 1.506755043591e-01 > 0 SNES Function norm 1.678694983365e-02 > 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 > 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 > 1 SNES Function norm 1.678694983365e-02 > 0 SNES Function norm 2.211256020366e-02 > 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 > 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 > 1 SNES Function norm 2.211256020366e-02 > 0 SNES Function norm 6.319453516491e-02 > 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 > 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 > 1 SNES Function norm 6.319453516491e-02 > 0 SNES Function norm 1.993150558437e+00 > 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 > 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 > 1 SNES Function norm 7.832112166701e-01 > 106 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.629560288909e+00 > 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 > 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 > 1 SNES Function norm 1.672457508921e+00 > 107 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.062643571582e+00 > 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 > 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 > 1 SNES Function norm 9.036991011875e-01 > 108 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.105674791223e+00 > 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 > 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 > 1 SNES Function norm 6.003561441737e-01 > 109 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.053877992744e+00 > 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 > 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 > 1 SNES Function norm 3.053877992744e+00 > 0 SNES Function norm 5.829087412676e-01 > 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 > 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 > 1 SNES Function norm 4.254244459909e-01 > 110 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.180846267144e+00 > 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 > 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 > 1 SNES Function norm 3.911236882643e-01 > 111 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.532064854858e+00 > 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 > 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 > 1 SNES Function norm 1.532064854858e+00 > 0 SNES Function norm 1.764940070080e-01 > 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 > 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 > 1 SNES Function norm 1.706450714666e-01 > 112 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.407680008950e-01 > 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 > 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 > 1 SNES Function norm 4.407680008950e-01 > 0 SNES Function norm 1.064954076440e+00 > 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 > 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 > 1 SNES Function norm 2.964963371165e-01 > 113 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.622925709624e-01 > 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 > 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 > 1 SNES Function norm 8.622925709624e-01 > 0 SNES Function norm 2.612065167743e+00 > 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 > 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 > 1 SNES Function norm 1.331790654620e+00 > 114 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.606851714788e+00 > 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 > 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 > 1 SNES Function norm 9.013753678258e-01 > 115 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 3.227741616537e+00 > 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 > 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 > 1 SNES Function norm 7.734015655943e-01 > 116 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.393623504956e+00 > 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 > 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 > 1 SNES Function norm 4.969182786938e-01 > 117 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.596405006351e+00 > 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 > 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 > 1 SNES Function norm 2.596405006351e+00 > 0 SNES Function norm 4.974558747187e-01 > 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 > 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 > 1 SNES Function norm 2.909443551880e-01 > 118 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.088860095507e+00 > 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 > 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 > 1 SNES Function norm 6.203518870415e-01 > 119 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.828717001803e+00 > 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 > 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 > 1 SNES Function norm 9.407254842319e-01 > 120 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970950728036e+00 > 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 > 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 > 1 SNES Function norm 2.277472194573e+00 > 121 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 6.655252327413e+00 > 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 > 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 > 1 SNES Function norm 3.860967926913e+00 > 122 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.141502067665e+00 > 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 > 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 > 1 SNES Function norm 6.426598300871e+00 > 123 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.224562198048e+01 > 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 > 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 > 1 SNES Function norm 5.283049961509e+00 > 124 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 5.780124419709e+00 > 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 > 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 > 1 SNES Function norm 1.227257133866e+00 > 125 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 5.063237678461e+00 > 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 > 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 > 1 SNES Function norm 4.907594073483e+00 > 126 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 8.280289521084e+00 > 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 > 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 > 1 SNES Function norm 3.515932015486e+00 > 127 TS dt 1.1301e-10 time 0.271565 > copy! > 0 SNES Function norm 2.638351398877e+00 > 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 > 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 > 1 SNES Function norm 6.341356656368e-01 > 128 TS dt 2.26019e-10 time 0.271565 > copy! > 0 SNES Function norm 4.327811744723e+00 > 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 > 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 > 1 SNES Function norm 1.736633786129e-01 > 129 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 3.450425343262e+00 > 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 > 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 > 1 SNES Function norm 1.373376592274e+00 > 130 TS dt 9.04078e-10 time 0.271565 > copy! > 0 SNES Function norm 4.603810494450e+00 > 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 > 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 > 1 SNES Function norm 1.142888930566e+00 > > > From: Matthew Knepley > > Sent: Tuesday, January 5, 2021 2:53 PM > To: Sepideh Kavousi > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi > wrote: > Dear Petsc Users > I wrote a code in 2D solving PF equations for solidification of alloys. I have two variables in the equations 1-non-dimensional concentration (U) 2- order parameter (p). Everything is fine with this code. > When I change the code to 3D, the code has convergence problems which I cannot solve it. > I know the term which is causing problem is a term in the equation for U: > (dp/dt)*(Pxx+Pyy+Pzz) > where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, respectively. > > When I eliminate this term, the results of 2D and 3D are close to each other. But adding that term the time step should be reduced a lot to obtain convergence, otherwise the timestep does not converge. I have no idea how I should solve this problem. > I solve the equations considering (I chose based on suggestions of Moose for solving Cahn-hillard) > -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type ilu -sub_ksp_type preonly > > Would you please help me solve this issue? > > In order to see the solver you have we need the output of > > -ts_view > > and then to see what is happening during the run we need > > -ts_monitor -snes_monitor -snes_linesearch_monitor -ksp_monitor_true_residual > > Thanks, > > Matt > > Best, > sepideh > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 5 21:43:40 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jan 2021 22:43:40 -0500 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: On Tue, Jan 5, 2021 at 9:52 PM Barry Smith wrote: > > Ah, -snes_fd_color so it was already using finite differencing with > coloring to compute the Jacobian which explains why the differences below > are exactly zero. > > Implicit time-step schemes essentially add terms like I/dt to the > Jacobian evaluation (and the function defining the ODE) so for tiny > time-steps the nonlinear system gets easier and easier to solve (the > nonlinear function becomes linear) But we didn't see that with your earlier > run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps > SNES still made no progress. It is hard to understand how this is possible, > regardless of the problem you are solving. > > I would next run the code with valgrind to insure there are no issues of > memory corruption or un-initialized data. > > How are you computing > > (dp/dt)*(Pxx+Pyy+Pzz) > > > That is, how are you computing Pxx etc? > > Are you using finite elements for the U and P model? Exactly what elements? > I agree with Barry. This does not seem to make sense, so I would expect some kind of inconsistent discretization, or other mathematical problem which makes your system unsolvable. Matt > Barry > > > > > > On Jan 5, 2021, at 7:47 PM, Sepideh Kavousi wrote: > > The output for -snes_test_jacobian is given as : > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > ---------- Testing Jacobian ------------- > Run with -snes_test_jacobian_view and optionally -snes_test_jacobian > to show difference > of hand-coded and finite difference Jacobian entries greater than > . > Testing hand-coded Jacobian, if (for double precision runs) ||J - > Jfd||_F/||J||_F is > O(1.e-8), the hand-coded Jacobian is probably correct. > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.673211216524e-03 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.067366268006e-18 true resid norm > 1.921588081784e-14 ||r(i)||/||b|| 3.774355453546e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.668754123581e-03 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.604939763954e-18 true resid norm > 4.960207918618e-14 ||r(i)||/||b|| 9.742768487084e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 4.172167491706e-04 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.588913006621e-20 true resid norm > 1.948840699675e-14 ||r(i)||/||b|| 3.827884650536e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.043059511038e-04 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.104783140057e-19 true resid norm > 5.333375230602e-14 ||r(i)||/||b|| 1.047573831159e-15 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 2.607659801680e-05 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.703990624142e-20 true resid norm > 6.287819675470e-14 ||r(i)||/||b|| 1.235044425390e-15 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.519156394293e-06 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.557215852400e-21 true resid norm > 1.840177917795e-14 ||r(i)||/||b|| 3.614450789618e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.629789529205e-06 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.173065981676e-22 true resid norm > 1.530485325589e-14 ||r(i)||/||b|| 3.006157089528e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 4.074474092157e-07 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.739825180849e-22 true resid norm > 2.686974828851e-14 ||r(i)||/||b|| 5.277717006551e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 1.018618539861e-07 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.045098765057e-22 true resid norm > 5.010251971826e-14 ||r(i)||/||b|| 9.841064290920e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 2.546546360165e-08 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.287065233667e-24 true resid norm > 1.149255127646e-14 ||r(i)||/||b|| 2.257350261311e-16 > ---------- Testing Jacobian ------------- > ||J - Jfd||_F/||J||_F = 0., ||J - Jfd||_F = 0. > 0 KSP preconditioned resid norm 6.366365906984e-09 true resid norm > 5.091168824543e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.164978605997e-24 true resid norm > 2.417410255090e-14 ||r(i)||/||b|| 4.748242178566e-16 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: TSStep has failed due to DIVERGED_STEP_REJECTED > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.14.0, Sep 29, 2020 > [0]PETSC ERROR: /scratch/07065/tg863649/alloy-3D-cubic-9/one.out on a > skylake named c456-001.stampede2.tacc.utexas.edu by tg863649 Tue Jan 5 > 18:35:45 2021 > [0]PETSC ERROR: Configure options --with-x=0 -with-pic --with-make-np=8 > --download-petsc4py=1 > --with-python-exec=/opt/apps/intel18/python3/3.7.0/bin/python3 > --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" > --with-packages-build-dir=/tmp/petsc-3.14/skylake --with-mpi=1 > --with-mpi-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64 > --with-scalar-type=real --with-shared-libraries=1 --with-precision=double > --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre > --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis > --with-plapack=1 --download-plapack --with-spai=1 --download-spai > --with-sundials=1 --download-sundials --with-elemental=1 > --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 > --download-parmetis --with-metis=1 --download-metis --with-fftw=1 > --with-fftw-dir=/opt/apps/intel18/impi18_0/fftw3/3.3.8 --with-hdf5=1 > --with-hdf5-dir=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64 > --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis > --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack > --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles > --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 > --download-superlu_dist --with-superlu=1 --download-superlu > --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis > --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 > --with-debugging=no --LIBS= > --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl > COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" > FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" > CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -g" > [0]PETSC ERROR: #1 TSStep() line 3773 in > /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c > [0]PETSC ERROR: #2 TSSolve() line 4154 in > /home1/apps/intel18/impi18_0/petsc/3.14/src/ts/interface/ts.c > [0]PETSC ERROR: #3 main() line 324 in > /scratch/07065/tg863649/alloy-3D-cubic-9/one.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -ksp_gmres_restart 1001 > [0]PETSC ERROR: -ksp_monitor_true_residual > [0]PETSC ERROR: -ksp_type gmres > [0]PETSC ERROR: -pc_type lu > [0]PETSC ERROR: -snes_fd_color > [0]PETSC ERROR: -snes_linesearch_monitor > [0]PETSC ERROR: -snes_linesearch_type l2 > [0]PETSC ERROR: -snes_test_jacobian > [0]PETSC ERROR: -snes_type newtontr > [0]PETSC ERROR: -sub_ksp_type preonly > [0]PETSC ERROR: -sub_pc_type ilu > [0]PETSC ERROR: -ts_bdf_adapt > [0]PETSC ERROR: -ts_max_snes_failures -1 > [0]PETSC ERROR: -ts_monitor > [0]PETSC ERROR: -ts_type bdf > [0]PETSC ERROR: -ts_view > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov--------- > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 3:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 4:48 PM Sepideh Kavousi wrote: > > I have tried different pcs (lu,jacobi,sor,mg,asm,ksp,pbjacobi). they still > do not solve the problem. the an example for lu: > -pc_type lu: > > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.761968940423e-02 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.026000796457e-17 true resid norm > 1.790914574372e-13 ||r(i)||/||b|| 3.050633943390e-15 > 1 SNES Function norm 5.808754845048e+01 > > > So with a perfect linear solver, Newton is making no progress. That means > that your initial guess is very far from the solution, but it could also > mean > that your Jacobian does not match your residual, since it is not > generating descent directions. Have you checked that it matches the finite > difference > approximation? You can do this using -snes_test_jacobian ( > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESComputeJacobian.html > > ) > > Thanks, > > Matt > > > 0 KSP preconditioned resid norm 1.191540511981e+06 true resid norm > 5.808754845048e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.114882955273e-10 true resid norm > 1.203029448994e-06 ||r(i)||/||b|| 2.071062527315e-08 > 2 SNES Function norm 5.794558475659e+01 > 0 KSP preconditioned resid norm 1.191526760369e+06 true resid norm > 5.794558475659e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.166698266769e-09 true resid norm > 1.059436818305e-06 ||r(i)||/||b|| 1.828330532439e-08 > 3 SNES Function norm 5.772046032228e+01 > 0 KSP preconditioned resid norm 1.191498793276e+06 true resid norm > 5.772046032228e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.135175980555e-09 true resid norm > 1.230437568859e-06 ||r(i)||/||b|| 2.131718219136e-08 > 4 SNES Function norm 5.750822623807e+01 > 0 KSP preconditioned resid norm 1.191443279124e+06 true resid norm > 5.750822623807e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.905670672745e-10 true resid norm > 1.110684454395e-06 ||r(i)||/||b|| 1.931348829639e-08 > 5 SNES Function norm 5.750395740069e+01 > 0 KSP preconditioned resid norm 1.191433462635e+06 true resid norm > 5.750395740069e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.777205269843e-10 true resid norm > 1.430076525401e-06 ||r(i)||/||b|| 2.486918448822e-08 > 6 SNES Function norm 5.750395740069e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 4.405989777785e-03 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.185260514636e-18 true resid norm > 2.397998208837e-14 ||r(i)||/||b|| 4.084736836000e-16 > 1 SNES Function norm 3.428067045216e+01 > 0 KSP preconditioned resid norm 1.895265798836e+06 true resid norm > 3.428067045216e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.403775884750e-09 true resid norm > 8.602063630559e-06 ||r(i)||/||b|| 2.509304373893e-07 > 2 SNES Function norm 3.428067045216e+01 > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.101564184740e-03 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.380439828314e-19 true resid norm > 3.289971303359e-14 ||r(i)||/||b|| 5.604118853253e-16 > 1 SNES Function norm 3.330623063627e+01 > 0 KSP preconditioned resid norm 6.892935831006e-04 true resid norm > 3.330623063627e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.234145873951e-19 true resid norm > 1.823438085202e-14 ||r(i)||/||b|| 5.474765683081e-16 > 2 SNES Function norm 1.228168587373e+01 > 0 KSP preconditioned resid norm 2.442179927376e-04 true resid norm > 1.228168587373e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.114351195683e-19 true resid norm > 1.328341389902e-14 ||r(i)||/||b|| 1.081562745993e-15 > 3 SNES Function norm 8.377002138765e+00 > 0 KSP preconditioned resid norm 1.670901807768e-04 true resid norm > 8.377002138765e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.056553283627e-19 true resid norm > 1.539291322344e-14 ||r(i)||/||b|| 1.837520507750e-15 > 4 SNES Function norm 6.871259668846e-01 > 0 KSP preconditioned resid norm 7.849703293659e-06 true resid norm > 6.871259668846e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.813692242300e-21 true resid norm > 3.443983247611e-16 ||r(i)||/||b|| 5.012157033194e-16 > 5 SNES Function norm 4.598894985660e-01 > 0 SNES Function norm 4.609452800861e-01 > 0 KSP preconditioned resid norm 4.482807772478e+00 true resid norm > 4.609452800861e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.779043025722e-15 true resid norm > 9.363790606967e-11 ||r(i)||/||b|| 2.031432148566e-10 > 1 SNES Function norm 3.909615632676e-01 > 1 TS dt 1.25e-05 time 6.25e-06 > copy! > 0 SNES Function norm 1.083064424363e+00 > 0 KSP preconditioned resid norm 2.899372173019e+02 true resid norm > 1.083064424363e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.843039617595e-13 true resid norm > 2.031204561200e-09 ||r(i)||/||b|| 1.875423581006e-09 > 1 SNES Function norm 1.083064424363e+00 > 0 SNES Function norm 4.081254017393e-01 > 0 KSP preconditioned resid norm 1.134610736819e+01 true resid norm > 4.081254017393e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.857284280292e-15 true resid norm > 1.950178080787e-10 ||r(i)||/||b|| 4.778379567837e-10 > 1 SNES Function norm 3.852294258555e-01 > 2 TS dt 6.25e-06 time 9.375e-06 > copy! > 0 SNES Function norm 3.855090631551e-01 > 0 KSP preconditioned resid norm 5.807020984426e+01 true resid norm > 3.855090631551e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.595611115758e-14 true resid norm > 6.458636528724e-10 ||r(i)||/||b|| 1.675352707888e-09 > 1 SNES Function norm 3.855090631551e-01 > 0 SNES Function norm 3.894882915800e-01 > 0 KSP preconditioned resid norm 3.545691207399e+00 true resid norm > 3.894882915800e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.936099793899e-15 true resid norm > 1.237085076854e-10 ||r(i)||/||b|| 3.176180397711e-10 > 1 SNES Function norm 3.826734119825e-01 > 3 TS dt 3.125e-06 time 1.09375e-05 > copy! > 0 SNES Function norm 3.846635056055e-01 > 0 KSP preconditioned resid norm 1.270632337662e+01 true resid norm > 3.846635056055e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.538849751267e-15 true resid norm > 2.462963992263e-10 ||r(i)||/||b|| 6.402905283117e-10 > 1 SNES Function norm 3.846635056055e-01 > 0 SNES Function norm 3.838572028974e-01 > 0 KSP preconditioned resid norm 9.750671428246e-01 true resid norm > 3.838572028974e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.289810958767e-15 true resid norm > 5.932884665420e-11 ||r(i)||/||b|| 1.545596805436e-10 > 1 SNES Function norm 3.823876176903e-01 > 4 TS dt 1.5625e-06 time 1.17188e-05 > copy! > 0 SNES Function norm 3.842190105197e-01 > 0 KSP preconditioned resid norm 2.949895830433e+00 true resid norm > 3.842190105197e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.544123722663e-15 true resid norm > 1.184005133464e-10 ||r(i)||/||b|| 3.081589148499e-10 > 1 SNES Function norm 3.842190105197e-01 > 0 SNES Function norm 3.825899662960e-01 > 0 KSP preconditioned resid norm 2.548468293806e-01 true resid norm > 3.825899662960e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.086200228021e-16 true resid norm > 2.882610131465e-11 ||r(i)||/||b|| 7.534463486778e-11 > 1 SNES Function norm 3.825019685537e-01 > 5 TS dt 7.8125e-07 time 1.21094e-05 > copy! > 0 SNES Function norm 3.827182409606e-01 > 0 KSP preconditioned resid norm 7.091231452893e-01 true resid norm > 3.827182409606e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.614097858359e-16 true resid norm > 5.440482894192e-11 ||r(i)||/||b|| 1.421537390153e-10 > 1 SNES Function norm 3.827182409606e-01 > 0 SNES Function norm 3.825683488225e-01 > 0 KSP preconditioned resid norm 6.509635329694e-02 true resid norm > 3.825683488225e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.255654660173e-17 true resid norm > 1.410340803479e-11 ||r(i)||/||b|| 3.686506758386e-11 > 1 SNES Function norm 3.825462954733e-01 > 6 TS dt 3.90625e-07 time 1.23047e-05 > copy! > 0 SNES Function norm 3.826632731069e-01 > 0 KSP preconditioned resid norm 1.737397651071e-01 true resid norm > 3.826632731069e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.095925875164e-16 true resid norm > 2.780031944787e-11 ||r(i)||/||b|| 7.264956268774e-11 > 1 SNES Function norm 3.826632731069e-01 > 0 SNES Function norm 3.825751343810e-01 > 0 KSP preconditioned resid norm 1.644741829353e-02 true resid norm > 3.825751343810e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.211213666062e-18 true resid norm > 7.515828526583e-12 ||r(i)||/||b|| 1.964536597169e-11 > 1 SNES Function norm 3.825729851167e-01 > 7 TS dt 1.95313e-07 time 1.24023e-05 > copy! > 0 SNES Function norm 3.826356173729e-01 > 0 KSP preconditioned resid norm 4.299238585205e-02 true resid norm > 3.826356173729e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.260847149806e-17 true resid norm > 1.287166330766e-11 ||r(i)||/||b|| 3.363948028683e-11 > 1 SNES Function norm 3.826356173729e-01 > 0 SNES Function norm 3.825855754640e-01 > 0 KSP preconditioned resid norm 4.133474793122e-03 true resid norm > 3.825855754640e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.077297138396e-18 true resid norm > 3.792689943805e-12 ||r(i)||/||b|| 9.913311392374e-12 > 1 SNES Function norm 3.825855754640e-01 > 0 SNES Function norm 3.825755706234e-01 > 0 KSP preconditioned resid norm 3.705095771277e-04 true resid norm > 3.825755706234e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.848267031641e-19 true resid norm > 1.079462463835e-12 ||r(i)||/||b|| 2.821566630813e-12 > 1 SNES Function norm 3.825755706234e-01 > 0 SNES Function norm 3.825734391910e-01 > 0 KSP preconditioned resid norm 2.690993850525e-05 true resid norm > 3.825734391910e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.585015168707e-21 true resid norm > 3.091320932665e-13 ||r(i)||/||b|| 8.080333384362e-13 > 1 SNES Function norm 3.825734391910e-01 > 0 SNES Function norm 3.825729114845e-01 > 0 KSP preconditioned resid norm 1.757556306807e-06 true resid norm > 3.825729114845e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.987700736384e-22 true resid norm > 8.038113355816e-14 ||r(i)||/||b|| 2.101067042260e-13 > 1 SNES Function norm 3.825729114845e-01 > 0 SNES Function norm 3.825565841303e-01 > 0 KSP preconditioned resid norm 1.103248802240e-07 true resid norm > 3.825565841303e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.504795613889e-22 true resid norm > 1.879813159274e-14 ||r(i)||/||b|| 4.913817294630e-14 > 1 SNES Function norm 3.825565841303e-01 > 0 SNES Function norm 3.825547574266e-01 > 0 KSP preconditioned resid norm 6.887463339138e-09 true resid norm > 3.825547574266e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.559974044704e-23 true resid norm > 5.311723680631e-15 ||r(i)||/||b|| 1.388487158378e-14 > 1 SNES Function norm 3.825547574266e-01 > 0 SNES Function norm 3.825908198230e-01 > 0 KSP preconditioned resid norm 4.314157283909e-10 true resid norm > 3.825908198230e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.248846716324e-24 true resid norm > 1.634240020842e-15 ||r(i)||/||b|| 4.271508714187e-15 > 1 SNES Function norm 3.825908198230e-01 > 0 SNES Function norm 3.823046631992e-01 > 0 KSP preconditioned resid norm 1.477599397951e-11 true resid norm > 3.823046631992e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.674742673343e-25 true resid norm > 4.342823046268e-15 ||r(i)||/||b|| 1.135958690623e-14 > 1 SNES Function norm 3.823046631992e-01 > 0 SNES Function norm 4.050006682433e-01 > 0 KSP preconditioned resid norm 4.744618612191e-12 true resid norm > 4.050006682433e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.185835350170e-26 true resid norm > 2.742859197646e-15 ||r(i)||/||b|| 6.772480671558e-15 > 1 SNES Function norm 4.050006682433e-01 > 0 SNES Function norm 4.420555752951e-01 > 0 KSP preconditioned resid norm 2.570862477765e-13 true resid norm > 4.420555752951e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.558628106533e-27 true resid norm > 2.669513100096e-15 ||r(i)||/||b|| 6.038863096149e-15 > 1 SNES Function norm 3.053219162874e-01 > 8 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.938107171164e-01 > 0 KSP preconditioned resid norm 1.152686085246e-13 true resid norm > 2.938107171164e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.059363368490e-28 true resid norm > 1.486663802399e-15 ||r(i)||/||b|| 5.059937285439e-15 > 1 SNES Function norm 1.879261738365e-01 > 9 TS dt 7.45058e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 2.598740119900e-01 > 0 KSP preconditioned resid norm 4.386389243102e-12 true resid norm > 2.598740119900e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.765692021861e-27 true resid norm > 4.362311845294e-16 ||r(i)||/||b|| 1.678625658599e-15 > 1 SNES Function norm 2.598740119900e-01 > 0 SNES Function norm 2.359036347057e-01 > 0 KSP preconditioned resid norm 3.416321738725e-13 true resid norm > 2.359036347057e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.983917914615e-27 true resid norm > 2.031818641817e-15 ||r(i)||/||b|| 8.612917916048e-15 > 1 SNES Function norm 1.708032078859e-01 > 10 TS dt 3.72529e-13 time 1.24023e-05 > copy! > 0 SNES Function norm 5.285427862819e-01 > 0 KSP preconditioned resid norm 1.592064460142e-12 true resid norm > 5.285427862819e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.163339775654e-26 true resid norm > 3.880080925123e-15 ||r(i)||/||b|| 7.341091442033e-15 > 1 SNES Function norm 5.285427862819e-01 > 0 SNES Function norm 1.646935190988e-01 > 0 KSP preconditioned resid norm 5.824112147206e-14 true resid norm > 1.646935190988e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.820310421468e-29 true resid norm > 7.168182199430e-17 ||r(i)||/||b|| 4.352437326407e-16 > 1 SNES Function norm 1.621698143957e-01 > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 3:25 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 4:17 PM Sepideh Kavousi wrote: > > Dear Matt, > following is the output for the first 130 timesteps. It continues the run > with the same small timestep > 0 TS dt 0.0001 time 0. > copy! > Write output at step= 0! > 0 SNES Function norm 5.870630851181e+01 > 0 KSP preconditioned resid norm 1.714717261432e-02 true resid norm > 5.870630851181e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.199465935450e-03 true resid norm > 4.319055307029e+00 ||r(i)||/||b|| 7.357054831952 > 2 KSP preconditioned resid norm 3.319453031999e-08 true resid norm > 2.987572551099e-04 ||r(i)||/||b|| 5.089014497476 > 1 SNES Function norm 5.870480277455e+01 > 0 SNES Function norm 5.870480198099e+01 > 0 KSP preconditioned resid norm 1.242359546728e+03 true resid norm > 5.870480198099e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.490848214055e+02 true resid norm > 7.765825085110e+05 ||r(i)||/||b|| 1.322860281110 > > > You can see here that the linear solver is very bad, which might > contribute to Newton stagnating. While you are starting out, I would > recommend > > -pc_type lu > > Once you have the nonlinear system converging, then you can try different > linear solvers to speed things up. > > Thanks, > > Matt > > > 1 SNES Function norm 5.870165343582e+01 > 1 TS dt 0.0002 time 0.0001 > copy! > 0 SNES Function norm 5.869324856421e+01 > 0 KSP preconditioned resid norm 1.610036885378e+04 true resid norm > 5.869324856421e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.932166117910e+03 true resid norm > 2.795187905611e+06 ||r(i)||/||b|| 4.762367008112 > 1 SNES Function norm 5.869298823911e+01 > 2 TS dt 0.0004 time 0.0003 > copy! > 0 SNES Function norm 5.867574766362e+01 > 0 KSP preconditioned resid norm 6.440121822416e+04 true resid norm > 5.867574766362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.729183419474e+03 true resid norm > 5.589239290096e+06 ||r(i)||/||b|| 9.525637955461 > 1 SNES Function norm 5.867531973582e+01 > 3 TS dt 0.0008 time 0.0007 > copy! > 0 SNES Function norm 5.864011322220e+01 > 0 KSP preconditioned resid norm 2.575951235569e+05 true resid norm > 5.864011322220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.092020125512e+04 true resid norm > 1.117372172629e+07 ||r(i)||/||b|| 1.905474105063 > 1 SNES Function norm 5.863990526788e+01 > 4 TS dt 0.0016 time 0.0015 > copy! > 0 SNES Function norm 5.856903590475e+01 > 0 KSP preconditioned resid norm 1.030166914887e+06 true resid norm > 5.856903590475e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.236963623364e+05 true resid norm > 2.232651118139e+07 ||r(i)||/||b|| 3.811999094145 > 1 SNES Function norm 5.856893794226e+01 > 5 TS dt 0.0032 time 0.0031 > copy! > 0 SNES Function norm 5.842679564768e+01 > 0 KSP preconditioned resid norm 4.116839933215e+06 true resid norm > 5.842679564768e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.947221952957e+05 true resid norm > 4.455474620275e+07 ||r(i)||/||b|| 7.625738449088 > 1 SNES Function norm 5.842675271515e+01 > 6 TS dt 0.0064 time 0.0063 > copy! > 0 SNES Function norm 5.814352339598e+01 > 0 KSP preconditioned resid norm 1.640271206849e+07 true resid norm > 5.814352339598e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.975364210355e+06 true resid norm > 8.860074436391e+07 ||r(i)||/||b|| 1.523828264767 > 1 SNES Function norm 5.814350805660e+01 > 7 TS dt 0.0128 time 0.0127 > copy! > 0 SNES Function norm 5.760173270961e+01 > 0 KSP preconditioned resid norm 6.454729140249e+07 true resid norm > 5.760173270961e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.823848123455e+06 true resid norm > 1.742411872562e+08 ||r(i)||/||b|| 3.024929616867 > 1 SNES Function norm 5.760173134258e+01 > 8 TS dt 0.0256 time 0.0255 > copy! > 0 SNES Function norm 5.683398372141e+01 > 0 KSP preconditioned resid norm 2.407430690480e+08 true resid norm > 5.683398372141e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.988112927961e+07 true resid norm > 3.294875226513e+08 ||r(i)||/||b|| 5.797368072355 > 1 SNES Function norm 5.683398372141e+01 > 0 SNES Function norm 5.747024606058e+01 > 0 KSP preconditioned resid norm 2.515545687156e+07 true resid norm > 5.747024606058e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.051720581278e+06 true resid norm > 1.095447785949e+08 ||r(i)||/||b|| 1.906112921101 > 1 SNES Function norm 5.747023898114e+01 > 9 TS dt 0.00758395 time 0.0319 > copy! > 0 SNES Function norm 5.725811890845e+01 > 0 KSP preconditioned resid norm 2.611072328843e+07 true resid norm > 5.725811890845e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.176469515845e+06 true resid norm > 1.113946431934e+08 ||r(i)||/||b|| 1.945482061182 > 1 SNES Function norm 5.725811208900e+01 > 10 TS dt 0.0151679 time 0.039484 > copy! > 0 SNES Function norm 5.684339053584e+01 > 0 KSP preconditioned resid norm 8.519149364113e+07 true resid norm > 5.684339053584e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.044655731587e+07 true resid norm > 1.965090970240e+08 ||r(i)||/||b|| 3.457026316896 > 1 SNES Function norm 5.684338688259e+01 > 11 TS dt 0.0303358 time 0.0546519 > copy! > 0 SNES Function norm 5.605303343054e+01 > 0 KSP preconditioned resid norm 3.077821813947e+08 true resid norm > 5.605303343054e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.830523927105e+07 true resid norm > 3.551741213376e+08 ||r(i)||/||b|| 6.336394296622 > 1 SNES Function norm 5.605303164316e+01 > 12 TS dt 0.0606716 time 0.0849877 > copy! > 0 SNES Function norm 5.463652619423e+01 > 0 KSP preconditioned resid norm 9.913250744377e+08 true resid norm > 5.463652619423e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.270586572012e+08 true resid norm > 5.803799416601e+08 ||r(i)||/||b|| 1.062256300111 > 1 SNES Function norm 5.463652534643e+01 > 13 TS dt 0.121343 time 0.145659 > copy! > 0 SNES Function norm 5.251966690761e+01 > 0 KSP preconditioned resid norm 2.257468849129e+09 true resid norm > 5.251966690761e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.978974887824e+08 true resid norm > 7.294445370018e+08 ||r(i)||/||b|| 1.388897873791 > 1 SNES Function norm 5.251966654744e+01 > 14 TS dt 0.242686 time 0.267003 > copy! > 0 SNES Function norm 5.150151326391e+01 > 0 KSP preconditioned resid norm 1.934621627312e+09 true resid norm > 5.150151326391e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.373534863658e+08 true resid norm > 5.533546685373e+08 ||r(i)||/||b|| 1.074443513343 > 1 SNES Function norm 5.150151326391e+01 > 0 SNES Function norm 5.184908859688e+01 > 0 KSP preconditioned resid norm 9.594647854417e+08 true resid norm > 5.184908859688e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.327300773354e+08 true resid norm > 5.290096803217e+08 ||r(i)||/||b|| 1.020287327391 > 1 SNES Function norm 5.184908859688e+01 > 0 SNES Function norm 5.232682115621e+01 > 0 KSP preconditioned resid norm 1.266845619319e+08 true resid norm > 5.232682115621e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.745176236994e+07 true resid norm > 2.288152713580e+08 ||r(i)||/||b|| 4.372810468936 > 1 SNES Function norm 5.232682115621e+01 > 0 SNES Function norm 5.246983753576e+01 > 0 KSP preconditioned resid norm 1.016188047721e+07 true resid norm > 5.246983753576e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.386074172374e+06 true resid norm > 6.834728918642e+07 ||r(i)||/||b|| 1.302601502050 > 1 SNES Function norm 5.246983737815e+01 > 15 TS dt 0.00758395 time 0.270794 > copy! > 0 SNES Function norm 5.237345394852e+01 > 0 KSP preconditioned resid norm 1.520623505703e+07 true resid norm > 5.237345394852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.085613628230e+06 true resid norm > 8.323112079508e+07 ||r(i)||/||b|| 1.589185255509 > 1 SNES Function norm 5.237345394852e+01 > 0 SNES Function norm 5.244535357001e+01 > 0 KSP preconditioned resid norm 1.536584002766e+06 true resid norm > 5.244535357001e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.096310197197e+05 true resid norm > 2.689484472091e+07 ||r(i)||/||b|| 5.128165393146 > 1 SNES Function norm 5.244535357001e+01 > 0 SNES Function norm 5.246369218939e+01 > 0 KSP preconditioned resid norm 1.396784143620e+05 true resid norm > 5.246369218939e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.902693207785e+04 true resid norm > 8.150907015311e+06 ||r(i)||/||b|| 1.553628171248 > 1 SNES Function norm 5.246369181668e+01 > 16 TS dt 0.000947994 time 0.271268 > copy! > 0 SNES Function norm 5.245144714661e+01 > 0 KSP preconditioned resid norm 2.476071225856e+05 true resid norm > 5.245144714661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.375328023503e+04 true resid norm > 1.084434754065e+07 ||r(i)||/||b|| 2.067502067262 > 1 SNES Function norm 5.245144714661e+01 > 0 SNES Function norm 5.246062448755e+01 > 0 KSP preconditioned resid norm 2.428224536380e+04 true resid norm > 5.246062448755e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.307763409533e+03 true resid norm > 3.402894455924e+06 ||r(i)||/||b|| 6.486568715421 > 1 SNES Function norm 5.246062426384e+01 > 17 TS dt 0.000473997 time 0.271505 > copy! > 0 SNES Function norm 5.245449911597e+01 > 0 KSP preconditioned resid norm 6.205856465529e+04 true resid norm > 5.245449911597e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.456948933661e+03 true resid norm > 5.437111337200e+06 ||r(i)||/||b|| 1.036538605617 > 1 SNES Function norm 5.245449911597e+01 > 0 SNES Function norm 5.245909110130e+01 > 0 KSP preconditioned resid norm 6.073102282836e+03 true resid norm > 5.245909110130e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.273091803130e+02 true resid norm > 1.702595489649e+06 ||r(i)||/||b|| 3.245568030070 > 1 SNES Function norm 5.245909110130e+01 > 0 SNES Function norm 5.246024056684e+01 > 0 KSP preconditioned resid norm 5.469219328745e+02 true resid norm > 5.246024056684e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.449715612636e+01 true resid norm > 5.111045278569e+05 ||r(i)||/||b|| 9.742702708458 > 1 SNES Function norm 5.246024055663e+01 > 18 TS dt 5.92496e-05 time 0.271535 > copy! > 0 SNES Function norm 5.245947411014e+01 > 0 KSP preconditioned resid norm 9.721423022050e+02 true resid norm > 5.245947411014e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.324232130801e+02 true resid norm > 6.813881966795e+05 ||r(i)||/||b|| 1.298884916858 > 1 SNES Function norm 5.245947411014e+01 > 0 SNES Function norm 5.246004891756e+01 > 0 KSP preconditioned resid norm 9.496157296515e+01 true resid norm > 5.246004891756e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.293484266592e+01 true resid norm > 2.129900140686e+05 ||r(i)||/||b|| 4.060042231438 > 1 SNES Function norm 5.246004890902e+01 > 19 TS dt 2.96248e-05 time 0.27155 > copy! > 0 SNES Function norm 5.245966566276e+01 > 0 KSP preconditioned resid norm 2.431000935393e+02 true resid norm > 5.245966566276e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.311385157364e+01 true resid norm > 3.407887271653e+05 ||r(i)||/||b|| 6.496204710034 > 1 SNES Function norm 5.245966566276e+01 > 0 SNES Function norm 5.245995306375e+01 > 0 KSP preconditioned resid norm 2.374244377432e+01 true resid norm > 5.245995306375e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.233977517286e+00 true resid norm > 1.065053760904e+05 ||r(i)||/||b|| 2.030222481536 > 2 KSP preconditioned resid norm 7.230270716674e-06 true resid norm > 6.926453755200e-01 ||r(i)||/||b|| 1.320331672196 > 1 SNES Function norm 5.245977670896e+01 > 20 TS dt 1.48124e-05 time 0.271557 > copy! > 0 SNES Function norm 5.245977453999e+01 > 0 KSP preconditioned resid norm 6.168684412120e+01 true resid norm > 5.245977453999e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.402553599990e+00 true resid norm > 1.729515771051e+05 ||r(i)||/||b|| 3.296841792053 > 2 KSP preconditioned resid norm 3.005621555900e-05 true resid norm > 1.799678959759e+00 ||r(i)||/||b|| 3.430588437598 > 1 SNES Function norm 5.245977453999e+01 > 0 SNES Function norm 5.245973715661e+01 > 0 KSP preconditioned resid norm 5.969268501139e+00 true resid norm > 5.245973715661e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.130709135360e-01 true resid norm > 5.355456766116e+04 ||r(i)||/||b|| 1.020869919750 > 2 KSP preconditioned resid norm 9.089448541659e-07 true resid norm > 1.741374551917e-01 ||r(i)||/||b|| 3.319449631855 > 1 SNES Function norm 5.245972630220e+01 > 21 TS dt 7.4062e-06 time 0.271561 > copy! > 0 SNES Function norm 5.245962991861e+01 > 0 KSP preconditioned resid norm 1.536192095911e+01 true resid norm > 5.245962991861e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.092469114093e+00 true resid norm > 8.614027499757e+04 ||r(i)||/||b|| 1.642029788072 > 2 KSP preconditioned resid norm 3.742653666692e-06 true resid norm > 4.481544594898e-01 ||r(i)||/||b|| 8.542844472694 > 1 SNES Function norm 5.245962991861e+01 > 0 SNES Function norm 5.245970348930e+01 > 0 KSP preconditioned resid norm 1.495436965385e+00 true resid norm > 5.245970348930e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.036883393815e-01 true resid norm > 2.683283953695e+04 ||r(i)||/||b|| 5.114943042410 > 2 KSP preconditioned resid norm 1.138549307287e-07 true resid norm > 4.362396228098e-02 ||r(i)||/||b|| 8.315708892613 > 1 SNES Function norm 5.245969886119e+01 > 22 TS dt 3.7031e-06 time 0.271563 > copy! > 0 SNES Function norm 5.245964973088e+01 > 0 KSP preconditioned resid norm 3.852412585267e+00 true resid norm > 5.245964973088e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.247342254154e-01 true resid norm > 4.320350322949e+04 ||r(i)||/||b|| 8.235568375147 > 2 KSP preconditioned resid norm 4.692850698407e-07 true resid norm > 1.123826615441e-01 ||r(i)||/||b|| 2.142268622086 > 1 SNES Function norm 5.245964973088e+01 > 0 SNES Function norm 5.245968740629e+01 > 0 KSP preconditioned resid norm 3.748147030747e-01 true resid norm > 5.245968740629e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.104998787585e-02 true resid norm > 1.345019035411e+04 ||r(i)||/||b|| 2.563909740815 > 2 KSP preconditioned resid norm 1.426784946039e-08 true resid norm > 1.093337980248e-02 ||r(i)||/||b|| 2.084148866120 > 1 SNES Function norm 5.245968066039e+01 > 23 TS dt 1.85155e-06 time 0.271564 > copy! > 0 SNES Function norm 5.245966664362e+01 > 0 KSP preconditioned resid norm 9.837329252613e-01 true resid norm > 5.245966664362e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.339899075028e-01 true resid norm > 2.206396454765e+04 ||r(i)||/||b|| 4.205891108218 > 2 KSP preconditioned resid norm 5.991678882897e-08 true resid norm > 2.869654952283e-02 ||r(i)||/||b|| 5.470211947357 > 1 SNES Function norm 5.245966664362e+01 > 0 SNES Function norm 5.245967459813e+01 > 0 KSP preconditioned resid norm 9.462653470563e-02 true resid norm > 5.245967459813e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.288704436220e-02 true resid norm > 6.790773385904e+03 ||r(i)||/||b|| 1.294474934876 > 2 KSP preconditioned resid norm 1.800909995964e-09 true resid norm > 2.760039732652e-03 ||r(i)||/||b|| 5.261259727199 > 1 SNES Function norm 5.245967167916e+01 > 24 TS dt 9.25775e-07 time 0.271564 > copy! > 0 SNES Function norm 5.245972527444e+01 > 0 KSP preconditioned resid norm 2.662779884000e-01 true resid norm > 5.245972527444e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.626690181904e-02 true resid norm > 1.194411437916e+04 ||r(i)||/||b|| 2.276816036813 > 2 KSP preconditioned resid norm 8.108932355980e-09 true resid norm > 7.767274936648e-03 ||r(i)||/||b|| 1.480616777159 > 1 SNES Function norm 5.245972527444e+01 > 0 SNES Function norm 5.245966286541e+01 > 0 KSP preconditioned resid norm 2.455492468134e-02 true resid norm > 5.245966286541e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.343507759856e-03 true resid norm > 3.523728385966e+03 ||r(i)||/||b|| 6.717024459356 > 2 KSP preconditioned resid norm 2.336250050344e-10 true resid norm > 7.160969335303e-04 ||r(i)||/||b|| 1.365042957610 > 1 SNES Function norm 5.245964735626e+01 > 25 TS dt 4.62888e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245965045260e+01 > 0 KSP preconditioned resid norm 6.979335700381e-02 true resid norm > 5.245965045260e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.504921828211e-03 true resid norm > 6.260729925833e+03 ||r(i)||/||b|| 1.193437217331 > 2 KSP preconditioned resid norm 1.062619850597e-09 true resid norm > 2.035685556196e-03 ||r(i)||/||b|| 3.880478689113 > 1 SNES Function norm 5.245965045260e+01 > 0 SNES Function norm 5.245964737315e+01 > 0 KSP preconditioned resid norm 6.401992290600e-03 true resid norm > 5.245964737315e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.713916783411e-04 true resid norm > 1.836735746083e+03 ||r(i)||/||b|| 3.501235402934 > 2 KSP preconditioned resid norm 3.044475123624e-11 true resid norm > 1.866360513754e-04 ||r(i)||/||b|| 3.557706937065 > 1 SNES Function norm 5.245953907407e+01 > 26 TS dt 2.31444e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245952669603e+01 > 0 KSP preconditioned resid norm 2.305046937264e-02 true resid norm > 5.245952669603e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.138819663630e-03 true resid norm > 4.134990532267e+03 ||r(i)||/||b|| 7.882249026428 > 2 KSP preconditioned resid norm 1.754568090046e-10 true resid norm > 6.722522467136e-04 ||r(i)||/||b|| 1.281468379631 > 1 SNES Function norm 5.245952669603e+01 > 0 SNES Function norm 5.245955619349e+01 > 0 KSP preconditioned resid norm 1.849410943612e-03 true resid norm > 5.245955619349e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.515382952823e-04 true resid norm > 1.060363623027e+03 ||r(i)||/||b|| 2.021297357370 > 2 KSP preconditioned resid norm 4.394235662988e-12 true resid norm > 5.387607049056e-05 ||r(i)||/||b|| 1.027002025939 > 1 SNES Function norm 5.245916156502e+01 > 27 TS dt 1.15722e-07 time 0.271565 > copy! > 0 SNES Function norm 5.245916087517e+01 > 0 KSP preconditioned resid norm 1.050473288664e-02 true resid norm > 5.245916087517e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.430366094863e-03 true resid norm > 3.768622481624e+03 ||r(i)||/||b|| 7.183916819775 > 2 KSP preconditioned resid norm 3.997794865638e-11 true resid norm > 3.063457445171e-04 ||r(i)||/||b|| 5.839699671256 > 1 SNES Function norm 5.245916087517e+01 > 0 SNES Function norm 5.245922267220e+01 > 0 KSP preconditioned resid norm 6.766788166208e-04 true resid norm > 5.245922267220e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.191834279480e-05 true resid norm > 7.748447133999e+02 ||r(i)||/||b|| 1.477041926911 > 2 KSP preconditioned resid norm 8.028477267590e-13 true resid norm > 1.968655949470e-05 ||r(i)||/||b|| 3.752735647212 > 1 SNES Function norm 5.245868047885e+01 > 28 TS dt 5.7861e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245971666219e+01 > 0 KSP preconditioned resid norm 5.675077511356e-03 true resid norm > 5.245971666219e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.727222643129e-04 true resid norm > 4.071757048781e+03 ||r(i)||/||b|| 7.761683264515 > 2 KSP preconditioned resid norm 1.079847189588e-11 true resid norm > 1.654940997196e-04 ||r(i)||/||b|| 3.154689164360 > 1 SNES Function norm 5.245971666219e+01 > 0 SNES Function norm 5.245866772896e+01 > 0 KSP preconditioned resid norm 3.156888082592e-04 true resid norm > 5.245866772896e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.278792745648e-05 true resid norm > 7.210828864373e+02 ||r(i)||/||b|| 1.374573388259 > 2 KSP preconditioned resid norm 1.868277550800e-13 true resid norm > 9.162051759969e-06 ||r(i)||/||b|| 1.746527724895 > 1 SNES Function norm 5.245862997528e+01 > 29 TS dt 2.89305e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245864268828e+01 > 0 KSP preconditioned resid norm 9.946730650307e-04 true resid norm > 5.245864268828e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.352720009406e-04 true resid norm > 1.425435652676e+03 ||r(i)||/||b|| 2.717256069978 > 2 KSP preconditioned resid norm 9.451723279064e-13 true resid norm > 2.897067420077e-05 ||r(i)||/||b|| 5.522574110985 > 1 SNES Function norm 5.245864268828e+01 > 0 SNES Function norm 5.245863432005e+01 > 0 KSP preconditioned resid norm 8.830611952540e-05 true resid norm > 5.245863432005e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.171521190212e-05 true resid norm > 3.936108229308e+02 ||r(i)||/||b|| 7.503260960424 > 2 KSP preconditioned resid norm 2.554304840579e-14 true resid norm > 2.504584831831e-06 ||r(i)||/||b|| 4.774399608938 > 1 SNES Function norm 5.245668231675e+01 > 30 TS dt 1.44652e-08 time 0.271565 > copy! > 0 SNES Function norm 5.245690451230e+01 > 0 KSP preconditioned resid norm 7.890756591999e-04 true resid norm > 5.245690451230e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.073178641459e-04 true resid norm > 2.261548879199e+03 ||r(i)||/||b|| 4.311251112174 > 2 KSP preconditioned resid norm 3.749009906718e-13 true resid norm > 2.298213554527e-05 ||r(i)||/||b|| 4.381145963328 > 1 SNES Function norm 5.245690451230e+01 > 0 SNES Function norm 5.245696584491e+01 > 0 KSP preconditioned resid norm 4.617797594369e-05 true resid norm > 5.245696584491e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.815858852608e-06 true resid norm > 3.880872061891e+02 ||r(i)||/||b|| 7.398201553184 > 2 KSP preconditioned resid norm 6.320906527522e-15 true resid norm > 1.238878017978e-06 ||r(i)||/||b|| 2.361703537411 > 1 SNES Function norm 5.245647736964e+01 > 31 TS dt 7.23262e-09 time 0.271565 > copy! > 0 SNES Function norm 5.245669639739e+01 > 0 KSP preconditioned resid norm 1.920423917689e-04 true resid norm > 5.245669639739e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.595404589478e-05 true resid norm > 1.092735026689e+03 ||r(i)||/||b|| 2.083118270375 > 2 KSP preconditioned resid norm 4.531344162809e-14 true resid norm > 5.555122754515e-06 ||r(i)||/||b|| 1.058992108926 > 1 SNES Function norm 5.245669639739e+01 > 0 SNES Function norm 5.245652990872e+01 > 0 KSP preconditioned resid norm 2.148561771896e-05 true resid norm > 5.245652990872e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.901735215594e-06 true resid norm > 2.469549886027e+02 ||r(i)||/||b|| 4.707802613562 > 2 KSP preconditioned resid norm 1.021721996628e-15 true resid norm > 3.996560492854e-07 ||r(i)||/||b|| 7.618804560287 > 1 SNES Function norm 5.245350296758e+01 > 32 TS dt 3.61631e-09 time 0.271565 > copy! > 0 SNES Function norm 5.246640623267e+01 > 0 KSP preconditioned resid norm 1.868946199874e-04 true resid norm > 5.246640623267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.526564335433e-05 true resid norm > 2.127311290841e+03 ||r(i)||/||b|| 4.054615979236 > 2 KSP preconditioned resid norm 2.205384342972e-14 true resid norm > 5.407251237239e-06 ||r(i)||/||b|| 1.030612086000 > 1 SNES Function norm 5.246640623267e+01 > 0 SNES Function norm 5.245276947886e+01 > 0 KSP preconditioned resid norm 1.857577202281e-05 true resid norm > 5.245276947886e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.251209740085e-06 true resid norm > 3.217505532117e+02 ||r(i)||/||b|| 6.134100380368 > 2 KSP preconditioned resid norm 3.347148796890e-16 true resid norm > 2.616700562397e-07 ||r(i)||/||b|| 4.988679508051 > 1 SNES Function norm 5.245276947886e+01 > 0 SNES Function norm 5.245277054267e+01 > 0 KSP preconditioned resid norm 1.585651339750e-05 true resid norm > 5.245277054267e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.158396081841e-07 true resid norm > 9.815158167794e+01 ||r(i)||/||b|| 1.871237318114 > 2 KSP preconditioned resid norm 9.262934482642e-18 true resid norm > 2.410856399560e-08 ||r(i)||/||b|| 4.596242247296 > 1 SNES Function norm 5.240788106409e+01 > 33 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 5.240539051979e+01 > 0 KSP preconditioned resid norm 1.847576989221e-05 true resid norm > 5.240539051979e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258349698670e-06 true resid norm > 8.091283770820e+02 ||r(i)||/||b|| 1.543979291169 > 2 KSP preconditioned resid norm 1.346406659091e-16 true resid norm > 2.631533872893e-07 ||r(i)||/||b|| 5.021494634029 > 1 SNES Function norm 5.240539051979e+01 > 0 SNES Function norm 5.240989970820e+01 > 0 KSP preconditioned resid norm 1.577993726636e-05 true resid norm > 5.240989970820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.768519589340e-08 true resid norm > 1.783077376605e+02 ||r(i)||/||b|| 3.402176662295 > 2 KSP preconditioned resid norm 2.919605399798e-18 true resid norm > 1.824330465594e-08 ||r(i)||/||b|| 3.480889060562 > 1 SNES Function norm 5.240989970820e+01 > 0 SNES Function norm 5.241000025070e+01 > 0 KSP preconditioned resid norm 1.575274319792e-05 true resid norm > 5.241000025070e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.221250233678e-09 true resid norm > 5.572925549085e+01 ||r(i)||/||b|| 1.063332478998 > 2 KSP preconditioned resid norm 8.231796009516e-20 true resid norm > 1.712111363787e-09 ||r(i)||/||b|| 3.266764654833 > 1 SNES Function norm 5.228672678486e+01 > 34 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 5.229325617508e+01 > 0 KSP preconditioned resid norm 1.531451762057e-05 true resid norm > 5.229325617508e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.923728185961e-08 true resid norm > 3.519843808690e+02 ||r(i)||/||b|| 6.730970809898 > 2 KSP preconditioned resid norm 9.219320031850e-19 true resid norm > 1.440250259886e-08 ||r(i)||/||b|| 2.754179726472 > 1 SNES Function norm 5.229325617508e+01 > 0 SNES Function norm 5.228920001144e+01 > 0 KSP preconditioned resid norm 1.556134876385e-05 true resid norm > 5.228920001144e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.003575759749e-09 true resid norm > 8.140280605473e+01 ||r(i)||/||b|| 1.556780483100 > 2 KSP preconditioned resid norm 2.086030462329e-20 true resid norm > 1.041501558695e-09 ||r(i)||/||b|| 1.991810084046 > 1 SNES Function norm 5.228920001144e+01 > 0 SNES Function norm 5.228865204069e+01 > 0 KSP preconditioned resid norm 1.560838827547e-05 true resid norm > 5.228865204069e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.922970395207e-10 true resid norm > 2.669725885478e+01 ||r(i)||/||b|| 5.105746239932 > 2 KSP preconditioned resid norm 6.839542768926e-21 true resid norm > 1.026968182189e-10 ||r(i)||/||b|| 1.964036444065 > 1 SNES Function norm 5.175201563991e+01 > 35 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.168638826028e+01 > 0 KSP preconditioned resid norm 1.404721029851e-05 true resid norm > 5.168638826028e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.587829868981e-09 true resid norm > 1.459250372800e+02 ||r(i)||/||b|| 2.823277891757 > 2 KSP preconditioned resid norm 7.859657062564e-21 true resid norm > 7.465711379232e-10 ||r(i)||/||b|| 1.444425047004 > 1 SNES Function norm 5.168638826028e+01 > 0 SNES Function norm 5.178402875676e+01 > 0 KSP preconditioned resid norm 1.497434443016e-05 true resid norm > 5.178402875676e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736806352294e-10 true resid norm > 3.562053719761e+01 ||r(i)||/||b|| 6.878672450328 > 2 KSP preconditioned resid norm 2.152524613455e-21 true resid norm > 5.703543390249e-11 ||r(i)||/||b|| 1.101409744892 > 1 SNES Function norm 5.178402875676e+01 > 0 SNES Function norm 5.178806568852e+01 > 0 KSP preconditioned resid norm 1.513680827217e-05 true resid norm > 5.178806568852e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.964269630137e-11 true resid norm > 1.286074159038e+01 ||r(i)||/||b|| 2.483340788925 > 1 SNES Function norm 4.925818844220e+01 > 36 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 5.270383537204e+01 > 0 KSP preconditioned resid norm 1.970630708625e-06 true resid norm > 5.270383537204e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.088592363798e-10 true resid norm > 1.004963420137e+02 ||r(i)||/||b|| 1.906812688380 > 2 KSP preconditioned resid norm 7.410648384868e-22 true resid norm > 6.428135574044e-11 ||r(i)||/||b|| 1.219671306399 > 1 SNES Function norm 5.270383537204e+01 > 0 SNES Function norm 4.874520711594e+01 > 0 KSP preconditioned resid norm 8.155830254099e-06 true resid norm > 4.874520711594e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.736808340126e-11 true resid norm > 2.849743547961e+01 ||r(i)||/||b|| 5.846202563430 > 1 SNES Function norm 4.874520711594e+01 > 0 SNES Function norm 4.879679718326e+01 > 0 KSP preconditioned resid norm 9.914611839830e-06 true resid norm > 4.879679718326e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.917454881119e-12 true resid norm > 1.012614869841e+01 ||r(i)||/||b|| 2.075166667267 > 1 SNES Function norm 4.872964148564e+01 > 37 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.866691749862e+01 > 0 KSP preconditioned resid norm 7.212230332263e-06 true resid norm > 4.866691749862e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.134577224509e-12 true resid norm > 1.336615037832e+01 ||r(i)||/||b|| 2.746455100366 > 1 SNES Function norm 4.866382095049e+01 > 38 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.893392986820e+01 > 0 KSP preconditioned resid norm 1.505657886052e-06 true resid norm > 4.893392986820e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.933826297142e-11 true resid norm > 2.517043359347e+01 ||r(i)||/||b|| 5.143758872681 > 2 KSP preconditioned resid norm 1.275522938352e-21 true resid norm > 4.032007898216e-12 ||r(i)||/||b|| 8.239697708883 > 1 SNES Function norm 4.893392986820e+01 > 0 SNES Function norm 4.865470718806e+01 > 0 KSP preconditioned resid norm 6.064585739884e-06 true resid norm > 4.865470718806e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.189348282039e-12 true resid norm > 9.118770839991e+00 ||r(i)||/||b|| 1.874180601837 > 1 SNES Function norm 4.843333577010e+01 > 39 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.842612728466e+01 > 0 KSP preconditioned resid norm 1.689891330588e-06 true resid norm > 4.842612728466e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.404712448465e-12 true resid norm > 1.406935960874e+01 ||r(i)||/||b|| 2.905324129274 > 1 SNES Function norm 4.842612728466e+01 > 0 SNES Function norm 4.840969985006e+01 > 0 KSP preconditioned resid norm 4.866603210605e-06 true resid norm > 4.840969985006e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.824415550862e-13 true resid norm > 5.685261525942e+00 ||r(i)||/||b|| 1.174405448402 > 1 SNES Function norm 3.917148384566e+01 > 0 KSP preconditioned resid norm 4.873840219744e-12 true resid norm > 3.917148384566e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.760673202702e-16 true resid norm > 1.090795416316e-02 ||r(i)||/||b|| 2.784667082345 > 2 KSP preconditioned resid norm 2.716930489198e-27 true resid norm > 1.727082218528e-14 ||r(i)||/||b|| 4.409029347299 > 2 SNES Function norm 3.373009701057e-01 > 40 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 4.835053852293e+01 > 0 KSP preconditioned resid norm 2.102791183017e-05 true resid norm > 4.835053852293e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.264257339614e-12 true resid norm > 2.740749949067e+01 ||r(i)||/||b|| 5.668499323471 > 1 SNES Function norm 4.835053852293e+01 > 0 SNES Function norm 1.287943764600e+01 > 0 KSP preconditioned resid norm 2.829357402918e-06 true resid norm > 1.287943764600e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.855151020448e-14 true resid norm > 1.640255580411e+00 ||r(i)||/||b|| 1.273545961784 > 1 SNES Function norm 1.128956057544e+01 > 41 TS dt 2.75903e-14 time 0.271565 > copy! > 0 SNES Function norm 1.669600342271e+01 > 0 KSP preconditioned resid norm 5.647351770054e-06 true resid norm > 1.669600342271e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.899074559115e-13 true resid norm > 3.019191100477e+00 ||r(i)||/||b|| 1.808331625262 > 1 SNES Function norm 1.472401994522e+01 > 42 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 5.552587859898e+01 > 0 KSP preconditioned resid norm 7.077263385509e-06 true resid norm > 5.552587859898e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.106473736400e-13 true resid norm > 4.220553854228e+00 ||r(i)||/||b|| 7.601057310069 > 1 SNES Function norm 2.907234222906e+01 > 0 KSP preconditioned resid norm 5.805178419738e-12 true resid norm > 2.907234222906e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.606372758906e-15 true resid norm > 9.077259836980e-03 ||r(i)||/||b|| 3.122300833369 > 2 KSP preconditioned resid norm 5.428926972364e-26 true resid norm > 2.711140186441e-13 ||r(i)||/||b|| 9.325496257164 > 2 SNES Function norm 1.772805631039e-01 > 43 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 7.725490953363e+00 > 0 KSP preconditioned resid norm 1.415994950687e-05 true resid norm > 7.725490953363e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.015357130343e-13 true resid norm > 7.859026423913e-01 ||r(i)||/||b|| 1.017285046524 > 1 SNES Function norm 5.415206018520e+00 > 0 KSP preconditioned resid norm 2.165462061838e-12 true resid norm > 5.415206018520e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.235973695994e-15 true resid norm > 3.206254598880e-03 ||r(i)||/||b|| 5.920835860934 > 2 KSP preconditioned resid norm 1.611047004247e-26 true resid norm > 3.962511257349e-14 ||r(i)||/||b|| 7.317378588732 > 2 SNES Function norm 2.952018920094e-02 > 44 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.042546652708e+00 > 0 KSP preconditioned resid norm 8.091399616290e-06 true resid norm > 3.042546652708e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.205771643684e-14 true resid norm > 1.106405513744e-01 ||r(i)||/||b|| 3.636445517636 > 1 SNES Function norm 7.553554533053e-01 > 0 KSP preconditioned resid norm 6.277932115893e-13 true resid norm > 7.553554533053e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.929640368541e-15 true resid norm > 6.440202696056e-03 ||r(i)||/||b|| 8.526055736905 > 2 KSP preconditioned resid norm 5.620814878971e-28 true resid norm > 1.291358921256e-15 ||r(i)||/||b|| 1.709604287102 > 2 SNES Function norm 6.213938456039e-02 > 45 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.726791649984e+00 > 0 KSP preconditioned resid norm 4.535477432157e-12 true resid norm > 1.726791649984e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.852992097181e-13 true resid norm > 2.620032765602e-01 ||r(i)||/||b|| 1.517283666287 > 2 KSP preconditioned resid norm 5.405843783997e-26 true resid norm > 9.329259279290e-14 ||r(i)||/||b|| 5.402654848009 > 1 SNES Function norm 1.726791649984e+00 > 0 SNES Function norm 4.569198217932e-01 > 0 KSP preconditioned resid norm 1.754712299938e-13 true resid norm > 4.569198217932e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.123917412662e-15 true resid norm > 9.715310204164e-03 ||r(i)||/||b|| 2.126261488512 > 2 KSP preconditioned resid norm 6.198312815878e-28 true resid norm > 2.844892856386e-15 ||r(i)||/||b|| 6.226240842915 > 1 SNES Function norm 5.060166636556e-02 > 46 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.648499891442e-01 > 0 KSP preconditioned resid norm 1.215583828915e-13 true resid norm > 4.648499891442e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.136647831085e-14 true resid norm > 5.780523594107e-02 ||r(i)||/||b|| 1.243524519544 > 2 KSP preconditioned resid norm 2.652404498648e-27 true resid norm > 9.750234455621e-15 ||r(i)||/||b|| 2.097501276395 > 1 SNES Function norm 4.303888962028e-01 > 47 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.205705397633e+00 > 0 KSP preconditioned resid norm 4.272460039856e-12 true resid norm > 1.205705397633e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.187144280708e-13 true resid norm > 2.132926270869e-01 ||r(i)||/||b|| 1.769027720251 > 2 KSP preconditioned resid norm 4.626725670443e-26 true resid norm > 6.980033435000e-14 ||r(i)||/||b|| 5.789169932142 > 1 SNES Function norm 1.205705397633e+00 > 0 SNES Function norm 3.168320704593e-01 > 0 KSP preconditioned resid norm 1.467076946194e-13 true resid norm > 3.168320704593e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.064560855977e-15 true resid norm > 1.313310429467e-02 ||r(i)||/||b|| 4.145130976050 > 2 KSP preconditioned resid norm 4.892746399265e-28 true resid norm > 2.638623094303e-15 ||r(i)||/||b|| 8.328143961179 > 1 SNES Function norm 1.059925414019e-01 > 48 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.008300299382e+00 > 0 KSP preconditioned resid norm 7.125489114170e-13 true resid norm > 1.008300299382e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.171770889218e-14 true resid norm > 7.934022500962e-02 ||r(i)||/||b|| 7.868709853430 > 2 KSP preconditioned resid norm 5.099650177302e-27 true resid norm > 2.081198449279e-14 ||r(i)||/||b|| 2.064066082847 > 1 SNES Function norm 5.842489315290e-01 > 49 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.508490594861e+00 > 0 KSP preconditioned resid norm 1.603738895795e-12 true resid norm > 1.508490594861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.662801546410e-14 true resid norm > 6.022795565867e-02 ||r(i)||/||b|| 3.992597359495 > 2 KSP preconditioned resid norm 1.733582507049e-26 true resid norm > 2.177135518256e-14 ||r(i)||/||b|| 1.443254287214 > 1 SNES Function norm 4.745080825960e-01 > 50 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.014687630993e+00 > 0 KSP preconditioned resid norm 1.060889675722e-12 true resid norm > 1.014687630993e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.800896805065e-13 true resid norm > 1.186937932631e-01 ||r(i)||/||b|| 1.169756973848 > 2 KSP preconditioned resid norm 5.568109464029e-26 true resid norm > 5.204626017419e-14 ||r(i)||/||b|| 5.129288914587 > 1 SNES Function norm 5.592988640286e-01 > 51 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.841734166744e-01 > 0 KSP preconditioned resid norm 3.788698480562e-11 true resid norm > 9.841734166744e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.427253759379e-12 true resid norm > 7.591907014254e-01 ||r(i)||/||b|| 7.713993169931 > 2 KSP preconditioned resid norm 2.270789969554e-24 true resid norm > 9.656618497679e-13 ||r(i)||/||b|| 9.811907468817 > 1 SNES Function norm 9.841734166744e-01 > 0 SNES Function norm 5.290699913962e-01 > 0 KSP preconditioned resid norm 2.961203074770e-13 true resid norm > 5.290699913962e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.579385754203e-14 true resid norm > 1.070765043149e-01 ||r(i)||/||b|| 2.023862741342 > 2 KSP preconditioned resid norm 1.605527076496e-26 true resid norm > 2.353592595573e-14 ||r(i)||/||b|| 4.448546759119 > 1 SNES Function norm 4.072990107379e-01 > 52 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.293320470683e-01 > 0 KSP preconditioned resid norm 3.661504324948e-12 true resid norm > 3.293320470683e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.548679963821e-13 true resid norm > 1.203392185968e-01 ||r(i)||/||b|| 3.654039127625 > 2 KSP preconditioned resid norm 7.635768593448e-26 true resid norm > 7.754573523890e-14 ||r(i)||/||b|| 2.354636784644 > 1 SNES Function norm 3.293320470683e-01 > 0 SNES Function norm 6.190385444978e-01 > 0 KSP preconditioned resid norm 7.667894793585e-13 true resid norm > 6.190385444978e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.308949396243e-14 true resid norm > 4.695429803832e-02 ||r(i)||/||b|| 7.585036255926 > 2 KSP preconditioned resid norm 5.691126876098e-27 true resid norm > 1.280807005416e-14 ||r(i)||/||b|| 2.069026261450 > 1 SNES Function norm 3.612736786105e-01 > 53 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.245269204940e+00 > 0 KSP preconditioned resid norm 2.568176854554e-12 true resid norm > 1.245269204940e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.776800765706e-14 true resid norm > 6.420730342930e-02 ||r(i)||/||b|| 5.156098229569 > 2 KSP preconditioned resid norm 1.529785626210e-26 true resid norm > 2.418985312534e-14 ||r(i)||/||b|| 1.942540057152 > 1 SNES Function norm 5.113533173009e-01 > 54 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.607180712938e+00 > 0 KSP preconditioned resid norm 4.636162321951e-12 true resid norm > 1.607180712938e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.328071063591e-12 true resid norm > 4.130113477995e-01 ||r(i)||/||b|| 2.569787855683 > 2 KSP preconditioned resid norm 2.986094131796e-25 true resid norm > 2.815708124309e-13 ||r(i)||/||b|| 1.751954899435 > 1 SNES Function norm 1.607180712938e+00 > 0 SNES Function norm 3.045128849808e-01 > 0 KSP preconditioned resid norm 5.097361796388e-13 true resid norm > 3.045128849808e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.515506059329e-14 true resid norm > 3.781546284180e-02 ||r(i)||/||b|| 1.241834572753 > 2 KSP preconditioned resid norm 3.850751097418e-27 true resid norm > 7.793249956367e-15 ||r(i)||/||b|| 2.559251296331 > 1 SNES Function norm 2.935346973548e-01 > 55 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 8.467066865826e-01 > 0 KSP preconditioned resid norm 2.105464755264e-12 true resid norm > 8.467066865826e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.055931985432e-13 true resid norm > 6.965043278992e-02 ||r(i)||/||b|| 8.226040244353 > 2 KSP preconditioned resid norm 1.468324550496e-26 true resid norm > 2.541042013299e-14 ||r(i)||/||b|| 3.001088870048 > 1 SNES Function norm 5.502623367952e-01 > 56 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.717470692129e+00 > 0 KSP preconditioned resid norm 6.408790513630e-12 true resid norm > 1.717470692129e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.479693689356e-13 true resid norm > 5.024422519606e-02 ||r(i)||/||b|| 2.925477880136 > 2 KSP preconditioned resid norm 9.272060121777e-26 true resid norm > 6.739995179418e-14 ||r(i)||/||b|| 3.924372747848 > 1 SNES Function norm 4.045528183667e-01 > 57 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.544462481147e+00 > 0 KSP preconditioned resid norm 2.692153893932e-12 true resid norm > 1.544462481147e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.398074985269e-13 true resid norm > 3.283436461006e-01 ||r(i)||/||b|| 2.125941226211 > 2 KSP preconditioned resid norm 5.181047437358e-25 true resid norm > 2.315578147201e-13 ||r(i)||/||b|| 1.499277693998 > 1 SNES Function norm 1.293824512618e+00 > 58 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.191314292345e+00 > 0 KSP preconditioned resid norm 1.039062273406e-11 true resid norm > 3.191314292345e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.212715067065e-12 true resid norm > 7.787068321621e-01 ||r(i)||/||b|| 2.440081924961 > 2 KSP preconditioned resid norm 4.498885154283e-24 true resid norm > 9.194573610481e-13 ||r(i)||/||b|| 2.881124442220 > 1 SNES Function norm 2.617175498914e+00 > 59 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 5.489780310948e+00 > 0 KSP preconditioned resid norm 6.978639256294e-11 true resid norm > 5.489780310948e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.433511036300e-12 true resid norm > 4.377640181520e-01 ||r(i)||/||b|| 7.974162778044 > 2 KSP preconditioned resid norm 1.898195012657e-23 true resid norm > 2.066485446633e-12 ||r(i)||/||b|| 3.764240697413 > 1 SNES Function norm 2.949802369100e+00 > 60 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 4.631512871139e+00 > 0 KSP preconditioned resid norm 1.314555356832e-10 true resid norm > 4.631512871139e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.149855773442e-11 true resid norm > 2.985608081754e-01 ||r(i)||/||b|| 6.446291233171 > 2 KSP preconditioned resid norm 6.387204091106e-23 true resid norm > 3.106571956406e-12 ||r(i)||/||b|| 6.707466961313 > 1 SNES Function norm 2.195569697365e+00 > 61 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.662194684551e+00 > 0 KSP preconditioned resid norm 3.979057807951e-11 true resid norm > 1.662194684551e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.086175282186e-11 true resid norm > 5.322911899228e-01 ||r(i)||/||b|| 3.202339622849 > 2 KSP preconditioned resid norm 2.731031856638e-22 true resid norm > 6.309496904983e-12 ||r(i)||/||b|| 3.795883216103 > 1 SNES Function norm 1.662194684551e+00 > 0 SNES Function norm 2.329147999080e+00 > 0 KSP preconditioned resid norm 6.940116009840e-11 true resid norm > 2.329147999080e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.686978686498e-13 true resid norm > 1.844903435300e-02 ||r(i)||/||b|| 7.920936909243 > 2 KSP preconditioned resid norm 1.554849191003e-24 true resid norm > 1.343083437814e-13 ||r(i)||/||b|| 5.766415179903 > 1 SNES Function norm 1.548533538885e-01 > 62 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.019371357242e+00 > 0 KSP preconditioned resid norm 8.927861046269e-11 true resid norm > 2.019371357242e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.633454572537e-12 true resid norm > 3.342963950456e-02 ||r(i)||/||b|| 1.655447839481 > 2 KSP preconditioned resid norm 5.428564048405e-24 true resid norm > 3.971355417344e-13 ||r(i)||/||b|| 1.966629566722 > 1 SNES Function norm 2.813130743975e-01 > 63 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.849189158431e+00 > 0 KSP preconditioned resid norm 4.231820684255e-10 true resid norm > 1.849189158431e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103282896492e-11 true resid norm > 3.234539155275e-01 ||r(i)||/||b|| 1.749166190234 > 2 KSP preconditioned resid norm 2.526336324696e-22 true resid norm > 6.913891205901e-12 ||r(i)||/||b|| 3.738877212415 > 1 SNES Function norm 1.849189158431e+00 > 0 SNES Function norm 1.106437703298e-01 > 0 KSP preconditioned resid norm 5.126209904027e-13 true resid norm > 1.106437703298e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.846798499741e-14 true resid norm > 4.840531155629e-02 ||r(i)||/||b|| 4.374879074709 > 2 KSP preconditioned resid norm 1.059052122888e-24 true resid norm > 1.127634799619e-13 ||r(i)||/||b|| 1.019157966379 > 1 SNES Function norm 1.071406781366e-01 > 64 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.036597476193e-01 > 0 KSP preconditioned resid norm 2.324014615591e-11 true resid norm > 1.036597476193e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.412487586102e-12 true resid norm > 5.133394649546e-02 ||r(i)||/||b|| 4.952158159209 > 2 KSP preconditioned resid norm 9.523883793204e-24 true resid norm > 5.272794267194e-13 ||r(i)||/||b|| 5.086636219256 > 1 SNES Function norm 1.036597476193e-01 > 0 SNES Function norm 1.081080288554e-01 > 0 KSP preconditioned resid norm 1.084935227116e-12 true resid norm > 1.081080288554e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.229645091619e-14 true resid norm > 5.392749154177e-03 ||r(i)||/||b|| 4.988296624470 > 2 KSP preconditioned resid norm 8.081519853713e-26 true resid norm > 1.900661741517e-14 ||r(i)||/||b|| 1.758113399753 > 1 SNES Function norm 4.280304180299e-02 > 65 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 2.901310268658e-02 > 0 KSP preconditioned resid norm 9.917300891113e-12 true resid norm > 2.901310268658e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434546877621e-12 true resid norm > 6.205707705779e-02 ||r(i)||/||b|| 2.138932803161 > 2 KSP preconditioned resid norm 2.529678011006e-24 true resid norm > 3.065329426678e-13 ||r(i)||/||b|| 1.056532787890 > 1 SNES Function norm 2.901310268658e-02 > 0 SNES Function norm 6.115306732380e-02 > 0 KSP preconditioned resid norm 2.547917833733e-12 true resid norm > 6.115306732380e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.801385928681e-13 true resid norm > 3.828533392152e-02 ||r(i)||/||b|| 6.260574587176 > 2 KSP preconditioned resid norm 1.669209280590e-25 true resid norm > 6.101206994457e-14 ||r(i)||/||b|| 9.976943531143 > 1 SNES Function norm 6.115306732380e-02 > 0 SNES Function norm 8.316482031597e-02 > 0 KSP preconditioned resid norm 1.012769836049e-13 true resid norm > 8.316482031597e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.197763826639e-14 true resid norm > 1.249377989981e-02 ||r(i)||/||b|| 1.502291455972 > 2 KSP preconditioned resid norm 4.446090640241e-27 true resid norm > 6.863398747477e-15 ||r(i)||/||b|| 8.252766880757 > 1 SNES Function norm 8.316482031597e-02 > 0 SNES Function norm 3.114039374287e-01 > 0 KSP preconditioned resid norm 8.748442759635e-14 true resid norm > 3.114039374287e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.226759854372e-14 true resid norm > 2.120469361912e-02 ||r(i)||/||b|| 6.809385197312 > 2 KSP preconditioned resid norm 9.397777017132e-28 true resid norm > 4.097331860538e-15 ||r(i)||/||b|| 1.315761096141 > 1 SNES Function norm 1.476629082037e-01 > 66 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.741199964540e-01 > 0 KSP preconditioned resid norm 3.188829732248e-13 true resid norm > 2.741199964540e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.647675195755e-14 true resid norm > 3.703443776834e-02 ||r(i)||/||b|| 1.351030141814 > 2 KSP preconditioned resid norm 2.098519894127e-27 true resid norm > 6.898284400613e-15 ||r(i)||/||b|| 2.516519951061 > 1 SNES Function norm 2.741199964540e-01 > 0 SNES Function norm 1.363976538331e+00 > 0 KSP preconditioned resid norm 1.251170838500e-13 true resid norm > 1.363976538331e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.636645431797e-15 true resid norm > 5.264845256515e-02 ||r(i)||/||b|| 3.859923619329 > 2 KSP preconditioned resid norm 7.635303466452e-28 true resid norm > 9.257181979862e-15 ||r(i)||/||b|| 6.786907046941 > 1 SNES Function norm 3.620782473843e-01 > 67 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 2.101847656775e+00 > 0 KSP preconditioned resid norm 4.257244883447e-13 true resid norm > 2.101847656775e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.680981244445e-14 true resid norm > 5.619073143068e-02 ||r(i)||/||b|| 2.673396963361 > 2 KSP preconditioned resid norm 1.997340575290e-27 true resid norm > 1.283993693662e-14 ||r(i)||/||b|| 6.108880867380 > 1 SNES Function norm 5.837297362402e-01 > 68 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.691478479373e+00 > 0 KSP preconditioned resid norm 8.133079044346e-13 true resid norm > 1.691478479373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.698893362948e-13 true resid norm > 2.106548414754e-01 ||r(i)||/||b|| 1.245388836123 > 2 KSP preconditioned resid norm 1.128206894627e-26 true resid norm > 4.076997707073e-14 ||r(i)||/||b|| 2.410316038182 > 1 SNES Function norm 1.646871085300e+00 > 69 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 4.140741966573e+00 > 0 KSP preconditioned resid norm 8.449944920915e-12 true resid norm > 4.140741966573e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.434715389913e-13 true resid norm > 1.581407737712e-01 ||r(i)||/||b|| 3.819140990862 > 2 KSP preconditioned resid norm 4.610158546072e-26 true resid norm > 5.765801763725e-14 ||r(i)||/||b|| 1.392456185454 > 1 SNES Function norm 1.278992537889e+00 > 70 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 4.925810717003e+00 > 0 KSP preconditioned resid norm 1.428864783873e-11 true resid norm > 4.925810717003e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.733848835145e-13 true resid norm > 5.645045312253e-02 ||r(i)||/||b|| 1.146013445618 > 2 KSP preconditioned resid norm 1.251375367698e-25 true resid norm > 5.322015357344e-14 ||r(i)||/||b|| 1.080434402194 > 1 SNES Function norm 4.757993340113e-01 > 71 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.014381910385e+00 > 0 KSP preconditioned resid norm 3.946219055429e-11 true resid norm > 1.014381910385e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.966786301027e-12 true resid norm > 1.033679382452e+00 ||r(i)||/||b|| 1.019023872438 > 2 KSP preconditioned resid norm 2.903362877451e-24 true resid norm > 1.280286864393e-12 ||r(i)||/||b|| 1.262134952611 > 1 SNES Function norm 1.014381910385e+00 > 0 SNES Function norm 5.247873040702e-01 > 0 KSP preconditioned resid norm 1.818706815696e-12 true resid norm > 5.247873040702e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.258546236990e-13 true resid norm > 6.756656786828e-02 ||r(i)||/||b|| 1.287503858120 > 2 KSP preconditioned resid norm 2.177562327102e-26 true resid norm > 2.997964765485e-14 ||r(i)||/||b|| 5.712723501946 > 1 SNES Function norm 5.128575534549e-01 > 72 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.463622536847e+00 > 0 KSP preconditioned resid norm 9.747931941457e-12 true resid norm > 1.463622536847e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.765688549276e-13 true resid norm > 2.251123750537e-01 ||r(i)||/||b|| 1.538049390375 > 2 KSP preconditioned resid norm 1.869288517804e-25 true resid norm > 1.493803600598e-13 ||r(i)||/||b|| 1.020620797365 > 1 SNES Function norm 1.463622536847e+00 > 0 SNES Function norm 5.760430307939e-01 > 0 KSP preconditioned resid norm 6.381755075677e-13 true resid norm > 5.760430307939e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.512645640892e-14 true resid norm > 5.054200634193e-02 ||r(i)||/||b|| 8.773998406383 > 2 KSP preconditioned resid norm 5.897788241013e-27 true resid norm > 1.444726065102e-14 ||r(i)||/||b|| 2.508017609572 > 1 SNES Function norm 4.031598199606e-01 > 73 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.304273554580e+00 > 0 KSP preconditioned resid norm 2.483318267074e-12 true resid norm > 1.304273554580e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.092864274592e-13 true resid norm > 7.365253445649e-02 ||r(i)||/||b|| 5.647015857820 > 2 KSP preconditioned resid norm 1.918045490471e-26 true resid norm > 3.067490789471e-14 ||r(i)||/||b|| 2.351876857964 > 1 SNES Function norm 5.901656814576e-01 > 74 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.806967885919e+00 > 0 KSP preconditioned resid norm 1.254495981226e-12 true resid norm > 1.806967885919e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.165988305016e-13 true resid norm > 5.816785021255e-01 ||r(i)||/||b|| 3.219085998475 > 2 KSP preconditioned resid norm 1.946561591545e-25 true resid norm > 1.737891724525e-13 ||r(i)||/||b|| 9.617723358940 > 1 SNES Function norm 1.806967885919e+00 > 0 SNES Function norm 3.525254737467e-01 > 0 KSP preconditioned resid norm 5.330467525907e-13 true resid norm > 3.525254737467e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.630807147216e-14 true resid norm > 3.919219364897e-02 ||r(i)||/||b|| 1.111754938797 > 2 KSP preconditioned resid norm 5.994572328389e-27 true resid norm > 9.229714104420e-15 ||r(i)||/||b|| 2.618169406688 > 1 SNES Function norm 2.971700641320e-01 > 75 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 9.276360188622e-01 > 0 KSP preconditioned resid norm 1.891036138299e-12 true resid norm > 9.276360188622e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.946049427296e-14 true resid norm > 4.561178088188e-02 ||r(i)||/||b|| 4.916991142477 > 2 KSP preconditioned resid norm 1.256902512171e-26 true resid norm > 1.744854906145e-14 ||r(i)||/||b|| 1.880969335672 > 1 SNES Function norm 3.629560323025e-01 > 76 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 1.281038880923e+00 > 0 KSP preconditioned resid norm 4.415052525028e-12 true resid norm > 1.281038880923e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.136240774791e-12 true resid norm > 3.697121999437e-01 ||r(i)||/||b|| 2.886034182486 > 2 KSP preconditioned resid norm 2.562962622803e-25 true resid norm > 2.399035397213e-13 ||r(i)||/||b|| 1.872726451116 > 1 SNES Function norm 1.281038880923e+00 > 0 SNES Function norm 2.062116742478e-01 > 0 KSP preconditioned resid norm 1.923072172220e-13 true resid norm > 2.062116742478e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.958504624205e-15 true resid norm > 6.829230759572e-03 ||r(i)||/||b|| 3.311757583310 > 2 KSP preconditioned resid norm 1.391184745224e-27 true resid norm > 3.773822554237e-15 ||r(i)||/||b|| 1.830072214874 > 1 SNES Function norm 4.076142877095e-02 > 77 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970990313724e-01 > 0 KSP preconditioned resid norm 9.040615152309e-13 true resid norm > 2.970990313724e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.232085752735e-14 true resid norm > 4.914824284390e-02 ||r(i)||/||b|| 1.654271392837 > 2 KSP preconditioned resid norm 7.636608912666e-27 true resid norm > 1.683518570269e-14 ||r(i)||/||b|| 5.666523254863 > 1 SNES Function norm 2.970990313724e-01 > 0 SNES Function norm 3.372906354998e-01 > 0 KSP preconditioned resid norm 1.370644308318e-13 true resid norm > 3.372906354998e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.614825882833e-14 true resid norm > 3.675107445605e-02 ||r(i)||/||b|| 1.089596644200 > 2 KSP preconditioned resid norm 8.601703784137e-28 true resid norm > 4.442878941035e-15 ||r(i)||/||b|| 1.317225701938 > 1 SNES Function norm 2.538611106909e-01 > 78 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 6.929161829162e-01 > 0 KSP preconditioned resid norm 7.330013379242e-13 true resid norm > 6.929161829162e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.054917531295e-14 true resid norm > 5.498057443165e-02 ||r(i)||/||b|| 7.934664507367 > 2 KSP preconditioned resid norm 7.310884687286e-27 true resid norm > 1.292973689106e-14 ||r(i)||/||b|| 1.865988587052 > 1 SNES Function norm 4.533321906044e-01 > 79 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.430558276957e+00 > 0 KSP preconditioned resid norm 1.636116329936e-12 true resid norm > 1.430558276957e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.817922999930e-14 true resid norm > 4.712735639247e-02 ||r(i)||/||b|| 3.294333209041 > 2 KSP preconditioned resid norm 1.146093777731e-26 true resid norm > 1.932373607025e-14 ||r(i)||/||b|| 1.350782864390 > 1 SNES Function norm 3.940601459612e-01 > 80 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 7.524069011779e-01 > 0 KSP preconditioned resid norm 2.481886990312e-12 true resid norm > 7.524069011779e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.339301105663e-14 true resid norm > 7.656029456180e-03 ||r(i)||/||b|| 1.017538441526 > 2 KSP preconditioned resid norm 2.346841971396e-26 true resid norm > 8.847412314801e-15 ||r(i)||/||b|| 1.175881335079 > 1 SNES Function norm 6.651399131369e-02 > 81 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.121363306761e-01 > 0 KSP preconditioned resid norm 1.415277696966e-11 true resid norm > 9.121363306761e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.506895452970e-12 true resid norm > 4.337437183567e-01 ||r(i)||/||b|| 4.755250983537 > 2 KSP preconditioned resid norm 1.384562439012e-24 true resid norm > 5.421357345910e-13 ||r(i)||/||b|| 5.943582295304 > 1 SNES Function norm 9.121363306761e-01 > 0 SNES Function norm 1.548591006657e-01 > 0 KSP preconditioned resid norm 5.616471772316e-13 true resid norm > 1.548591006657e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.954607379149e-14 true resid norm > 5.523767371683e-02 ||r(i)||/||b|| 3.566963354390 > 2 KSP preconditioned resid norm 1.470440737347e-26 true resid norm > 2.184049715244e-14 ||r(i)||/||b|| 1.410346376710 > 1 SNES Function norm 1.548591006657e-01 > 0 SNES Function norm 3.363022877948e-01 > 0 KSP preconditioned resid norm 8.467251086447e-14 true resid norm > 3.363022877948e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.434324574579e-14 true resid norm > 2.672034782109e-02 ||r(i)||/||b|| 7.945336321172 > 2 KSP preconditioned resid norm 1.110644590522e-27 true resid norm > 5.785593809905e-15 ||r(i)||/||b|| 1.720355174460 > 1 SNES Function norm 2.439869058762e-01 > 82 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.602729745290e+00 > 0 KSP preconditioned resid norm 9.709062010424e-13 true resid norm > 1.602729745290e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.355854428092e-14 true resid norm > 1.378584285322e-01 ||r(i)||/||b|| 8.601476882629 > 2 KSP preconditioned resid norm 8.483606474061e-27 true resid norm > 3.154414053953e-14 ||r(i)||/||b|| 1.968150939497 > 1 SNES Function norm 1.007188504688e+00 > 83 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.461320490774e+00 > 0 KSP preconditioned resid norm 7.492674005874e-12 true resid norm > 3.461320490774e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.176957625211e-13 true resid norm > 2.789428168044e-01 ||r(i)||/||b|| 8.058855501763 > 2 KSP preconditioned resid norm 7.444156283855e-26 true resid norm > 1.021992286550e-13 ||r(i)||/||b|| 2.952608084902 > 1 SNES Function norm 2.147586456566e+00 > 84 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 6.341628072414e+00 > 0 KSP preconditioned resid norm 1.890014038061e-11 true resid norm > 6.341628072414e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.459710305760e-13 true resid norm > 4.739692900452e-02 ||r(i)||/||b|| 7.473937049493 > 2 KSP preconditioned resid norm 8.252858530640e-26 true resid norm > 4.602796303332e-14 ||r(i)||/||b|| 7.258067251459 > 1 SNES Function norm 3.965504408873e-01 > 85 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 9.612683816763e-01 > 0 KSP preconditioned resid norm 2.356191998211e-11 true resid norm > 9.612683816763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.862087577004e-12 true resid norm > 6.694050681105e-01 ||r(i)||/||b|| 6.963768712991 > 2 KSP preconditioned resid norm 1.951534640097e-24 true resid norm > 8.287711294991e-13 ||r(i)||/||b|| 8.621641419785 > 1 SNES Function norm 9.612683816763e-01 > 0 SNES Function norm 7.029352448723e-01 > 0 KSP preconditioned resid norm 8.948344944644e-13 true resid norm > 7.029352448723e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.386128932384e-14 true resid norm > 3.219048667133e-02 ||r(i)||/||b|| 4.579438419990 > 2 KSP preconditioned resid norm 1.108923120335e-26 true resid norm > 1.627610218719e-14 ||r(i)||/||b|| 2.315448301379 > 1 SNES Function norm 2.593382931820e-01 > 86 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.105068885806e-01 > 0 KSP preconditioned resid norm 1.618419501580e-12 true resid norm > 3.105068885806e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.981247261898e-13 true resid norm > 1.027739572583e-01 ||r(i)||/||b|| 3.309876883187 > 2 KSP preconditioned resid norm 6.583406103694e-26 true resid norm > 6.399058569206e-14 ||r(i)||/||b|| 2.060842707374 > 1 SNES Function norm 3.105068885806e-01 > 0 SNES Function norm 4.981482205604e-01 > 0 KSP preconditioned resid norm 8.626925542249e-13 true resid norm > 4.981482205604e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.669162705870e-14 true resid norm > 7.232605345260e-02 ||r(i)||/||b|| 1.451898259744 > 2 KSP preconditioned resid norm 9.172627809051e-27 true resid norm > 1.659330498905e-14 ||r(i)||/||b|| 3.330997543339 > 1 SNES Function norm 4.981482205604e-01 > 0 SNES Function norm 7.138905144630e-01 > 0 KSP preconditioned resid norm 5.742811613241e-14 true resid norm > 7.138905144630e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.084547511423e-14 true resid norm > 6.979230429959e-02 ||r(i)||/||b|| 9.776331648290 > 2 KSP preconditioned resid norm 6.986277174013e-28 true resid norm > 8.250974203380e-15 ||r(i)||/||b|| 1.155775855852 > 1 SNES Function norm 3.696022475452e-01 > 87 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.789063973037e-01 > 0 KSP preconditioned resid norm 2.073370392578e-13 true resid norm > 8.789063973037e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.600619418517e-15 true resid norm > 2.560720526085e-02 ||r(i)||/||b|| 2.913530421375 > 2 KSP preconditioned resid norm 3.498888144974e-28 true resid norm > 1.974995118370e-15 ||r(i)||/||b|| 2.247105180288 > 1 SNES Function norm 2.343211520992e-01 > 88 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 3.552109100087e+00 > 0 KSP preconditioned resid norm 3.072072938848e-12 true resid norm > 3.552109100087e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.936295929537e-13 true resid norm > 4.080829783621e-01 ||r(i)||/||b|| 1.148846971936 > 2 KSP preconditioned resid norm 2.809147978825e-26 true resid norm > 7.254932274516e-14 ||r(i)||/||b|| 2.042429460947 > 1 SNES Function norm 2.944262139143e+00 > 89 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 7.582944894373e+00 > 0 KSP preconditioned resid norm 8.393978853130e-12 true resid norm > 7.582944894373e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.262222672686e-13 true resid norm > 2.986218296929e-01 ||r(i)||/||b|| 3.938072000424 > 2 KSP preconditioned resid norm 6.640484585438e-26 true resid norm > 1.074847335850e-13 ||r(i)||/||b|| 1.417453708054 > 1 SNES Function norm 2.445215958037e+00 > 90 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 3.577254475804e+00 > 0 KSP preconditioned resid norm 1.115580563317e-11 true resid norm > 3.577254475804e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.650683554983e-14 true resid norm > 1.178080901429e-02 ||r(i)||/||b|| 3.293254392153 > 2 KSP preconditioned resid norm 7.673544276365e-26 true resid norm > 2.812768618506e-14 ||r(i)||/||b|| 7.862925708896 > 1 SNES Function norm 9.478103979365e-02 > 91 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 3.398926029549e+00 > 0 KSP preconditioned resid norm 5.123364982057e-12 true resid norm > 3.398926029549e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.115609694640e-12 true resid norm > 9.139384215290e-01 ||r(i)||/||b|| 2.688903534774 > 2 KSP preconditioned resid norm 1.387793573464e-24 true resid norm > 5.108329430216e-13 ||r(i)||/||b|| 1.502924566703 > 1 SNES Function norm 2.854550344474e+00 > 92 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.846027110554e+00 > 0 KSP preconditioned resid norm 5.947065097708e-11 true resid norm > 8.846027110554e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.761881698641e-12 true resid norm > 6.337664376005e-01 ||r(i)||/||b|| 7.164418893136 > 2 KSP preconditioned resid norm 7.117755314975e-24 true resid norm > 1.565174980493e-12 ||r(i)||/||b|| 1.769353587698 > 1 SNES Function norm 4.478173098971e+00 > 93 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 9.479827241861e+00 > 0 KSP preconditioned resid norm 1.701559584628e-10 true resid norm > 9.479827241861e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.315086094503e-12 true resid norm > 3.627411941645e-01 ||r(i)||/||b|| 3.826453635808 > 2 KSP preconditioned resid norm 1.798626239608e-23 true resid norm > 2.081618631614e-12 ||r(i)||/||b|| 2.195840260065 > 1 SNES Function norm 2.980698125901e+00 > 94 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.583823756970e+00 > 0 KSP preconditioned resid norm 4.104881353739e-11 true resid norm > 2.583823756970e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.478889497974e-12 true resid norm > 4.729547726201e-01 ||r(i)||/||b|| 1.830445173918 > 2 KSP preconditioned resid norm 5.263394780419e-23 true resid norm > 2.707360752891e-12 ||r(i)||/||b|| 1.047811695975 > 1 SNES Function norm 1.929142014309e+00 > 95 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 1.257265283168e-01 > 0 KSP preconditioned resid norm 1.888094035320e-10 true resid norm > 1.257265283168e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.700866752609e-11 true resid norm > 2.919506838417e-01 ||r(i)||/||b|| 2.322108848072 > 2 KSP preconditioned resid norm 2.762793483719e-22 true resid norm > 5.974718530884e-12 ||r(i)||/||b|| 4.752154227810 > 1 SNES Function norm 1.257265283168e-01 > 0 SNES Function norm 2.389423098829e+00 > 0 KSP preconditioned resid norm 7.163702375816e-11 true resid norm > 2.389423098829e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.293759314015e-13 true resid norm > 1.716421726476e-02 ||r(i)||/||b|| 7.183414805513 > 2 KSP preconditioned resid norm 1.175729523003e-24 true resid norm > 1.264669953401e-13 ||r(i)||/||b|| 5.292783659876 > 1 SNES Function norm 1.440136831177e-01 > 96 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.101422245445e+00 > 0 KSP preconditioned resid norm 7.152285194626e-11 true resid norm > 2.101422245445e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.097479577508e-12 true resid norm > 9.195764624290e-02 ||r(i)||/||b|| 4.375971865827 > 2 KSP preconditioned resid norm 1.748601892473e-23 true resid norm > 1.041205209420e-12 ||r(i)||/||b|| 4.954764382443 > 1 SNES Function norm 7.435685625660e-01 > 97 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 3.062056964383e+00 > 0 KSP preconditioned resid norm 6.084026768000e-10 true resid norm > 3.062056964383e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.895899408610e-11 true resid norm > 4.033100253552e-01 ||r(i)||/||b|| 1.317121235975 > 2 KSP preconditioned resid norm 3.441051874886e-22 true resid norm > 8.780304816060e-12 ||r(i)||/||b|| 2.867453126507 > 1 SNES Function norm 3.062056964383e+00 > 0 SNES Function norm 1.640638339300e-01 > 0 KSP preconditioned resid norm 5.417809531981e-12 true resid norm > 1.640638339300e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.578652530864e-14 true resid norm > 8.330361033090e-04 ||r(i)||/||b|| 5.077512108271 > 2 KSP preconditioned resid norm 5.700346043787e-25 true resid norm > 1.892267222941e-14 ||r(i)||/||b|| 1.153372548729 > 1 SNES Function norm 7.185385907970e-03 > 98 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 1.778588005116e-01 > 0 KSP preconditioned resid norm 1.467933269180e-11 true resid norm > 1.778588005116e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.320665418771e-13 true resid norm > 1.500689112501e-02 ||r(i)||/||b|| 8.437530828862 > 2 KSP preconditioned resid norm 7.093249924252e-24 true resid norm > 2.009990248879e-13 ||r(i)||/||b|| 1.130104466631 > 1 SNES Function norm 1.179095579605e-01 > 99 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 2.512082795192e-01 > 0 KSP preconditioned resid norm 2.545260780417e-10 true resid norm > 2.512082795192e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.103551821095e-11 true resid norm > 3.332276483102e-01 ||r(i)||/||b|| 1.326499464699 > 2 KSP preconditioned resid norm 2.716418948880e-22 true resid norm > 6.784375663172e-12 ||r(i)||/||b|| 2.700697475480 > 1 SNES Function norm 2.512082795192e-01 > 0 SNES Function norm 8.541223283936e-02 > 0 KSP preconditioned resid norm 4.670001273164e-12 true resid norm > 8.541223283936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.562541098196e-13 true resid norm > 8.434243733496e-03 ||r(i)||/||b|| 9.874749146717 > 2 KSP preconditioned resid norm 6.370478068503e-25 true resid norm > 5.778616836687e-14 ||r(i)||/||b|| 6.765561143396 > 1 SNES Function norm 6.589891795632e-02 > 100 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 2.174757993770e-01 > 0 KSP preconditioned resid norm 2.859280269411e-12 true resid norm > 2.174757993770e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.471810140751e-12 true resid norm > 6.201078658085e-02 ||r(i)||/||b|| 2.851387913436 > 2 KSP preconditioned resid norm 5.722426147432e-24 true resid norm > 3.941686459620e-13 ||r(i)||/||b|| 1.812471305272 > 1 SNES Function norm 2.174757993770e-01 > 0 SNES Function norm 3.006903173936e-02 > 0 KSP preconditioned resid norm 6.535808425657e-13 true resid norm > 3.006903173936e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.524559373094e-13 true resid norm > 1.019789011252e-02 ||r(i)||/||b|| 3.391492682875 > 2 KSP preconditioned resid norm 1.371358314749e-25 true resid norm > 3.214755423120e-14 ||r(i)||/||b|| 1.069125022377 > 1 SNES Function norm 3.006903173936e-02 > 0 SNES Function norm 5.071683999025e-02 > 0 KSP preconditioned resid norm 3.218558385971e-13 true resid norm > 5.071683999025e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.425007105445e-14 true resid norm > 5.633853826000e-03 ||r(i)||/||b|| 1.110844805607 > 2 KSP preconditioned resid norm 1.798053049205e-26 true resid norm > 6.703474165418e-15 ||r(i)||/||b|| 1.321745236238 > 1 SNES Function norm 4.513917663662e-02 > 101 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 1.261469809763e-01 > 0 KSP preconditioned resid norm 1.032315057640e-12 true resid norm > 1.261469809763e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.604301669904e-14 true resid norm > 1.296411054050e-02 ||r(i)||/||b|| 1.027698835134 > 2 KSP preconditioned resid norm 1.310670285844e-25 true resid norm > 2.515529585425e-14 ||r(i)||/||b|| 1.994125872817 > 1 SNES Function norm 9.629885791766e-02 > 102 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 3.223153986924e-01 > 0 KSP preconditioned resid norm 2.305970581681e-12 true resid norm > 3.223153986924e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.976815022230e-13 true resid norm > 1.959093829944e-02 ||r(i)||/||b|| 6.078188748945 > 2 KSP preconditioned resid norm 6.568144168886e-25 true resid norm > 8.678245455582e-14 ||r(i)||/||b|| 2.692470012537 > 1 SNES Function norm 1.444530545281e-01 > 103 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 3.103810286198e-01 > 0 KSP preconditioned resid norm 6.997537490850e-11 true resid norm > 3.103810286198e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.436018618177e-12 true resid norm > 3.621194384608e-01 ||r(i)||/||b|| 1.166693209540 > 2 KSP preconditioned resid norm 1.688134668219e-23 true resid norm > 1.831353422483e-12 ||r(i)||/||b|| 5.900339433201 > 1 SNES Function norm 3.103810286198e-01 > 0 SNES Function norm 1.063191577145e-01 > 0 KSP preconditioned resid norm 8.698969335864e-13 true resid norm > 1.063191577145e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.309099373219e-15 true resid norm > 8.571707973797e-04 ||r(i)||/||b|| 8.062242175410 > 2 KSP preconditioned resid norm 7.499864860003e-27 true resid norm > 2.462681494673e-15 ||r(i)||/||b|| 2.316310199980 > 1 SNES Function norm 1.280948192266e-02 > 104 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 1.183517461737e-01 > 0 KSP preconditioned resid norm 1.090570879027e-12 true resid norm > 1.183517461737e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.694477819390e-14 true resid norm > 4.063412999276e-03 ||r(i)||/||b|| 3.433335908127 > 2 KSP preconditioned resid norm 4.916221176514e-26 true resid norm > 1.355899448286e-14 ||r(i)||/||b|| 1.145652254506 > 1 SNES Function norm 3.338348178983e-02 > 105 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.506755043591e-01 > 0 KSP preconditioned resid norm 3.066688763024e-11 true resid norm > 1.506755043591e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.742427906882e-12 true resid norm > 1.608109396920e-01 ||r(i)||/||b|| 1.067266642816 > 2 KSP preconditioned resid norm 5.677157047324e-24 true resid norm > 8.112232502417e-13 ||r(i)||/||b|| 5.383909306905 > 1 SNES Function norm 1.506755043591e-01 > 0 SNES Function norm 1.678694983365e-02 > 0 KSP preconditioned resid norm 4.701859124640e-13 true resid norm > 1.678694983365e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.091062630452e-14 true resid norm > 6.973109079933e-03 ||r(i)||/||b|| 4.153886887751 > 2 KSP preconditioned resid norm 4.322767172602e-26 true resid norm > 1.155415302284e-14 ||r(i)||/||b|| 6.882818580704 > 1 SNES Function norm 1.678694983365e-02 > 0 SNES Function norm 2.211256020366e-02 > 0 KSP preconditioned resid norm 2.163088052431e-13 true resid norm > 2.211256020366e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.334586688160e-14 true resid norm > 1.543819424718e-02 ||r(i)||/||b|| 6.981640346024 > 2 KSP preconditioned resid norm 4.802892641462e-27 true resid norm > 7.248676397051e-15 ||r(i)||/||b|| 3.278081022862 > 1 SNES Function norm 2.211256020366e-02 > 0 SNES Function norm 6.319453516491e-02 > 0 KSP preconditioned resid norm 6.072060969357e-14 true resid norm > 6.319453516491e-02 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.964391819297e-15 true resid norm > 1.554133427814e-02 ||r(i)||/||b|| 2.459284531104 > 2 KSP preconditioned resid norm 5.645620505335e-28 true resid norm > 2.227910476902e-15 ||r(i)||/||b|| 3.525479649606 > 1 SNES Function norm 6.319453516491e-02 > 0 SNES Function norm 1.993150558437e+00 > 0 KSP preconditioned resid norm 1.408921226669e-13 true resid norm > 1.993150558437e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.137246523173e-14 true resid norm > 9.300777627086e-02 ||r(i)||/||b|| 4.666369827264 > 2 KSP preconditioned resid norm 3.373579898708e-28 true resid norm > 6.738838901468e-15 ||r(i)||/||b|| 3.380998426306 > 1 SNES Function norm 7.832112166701e-01 > 106 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.629560288909e+00 > 0 KSP preconditioned resid norm 2.855866937659e-13 true resid norm > 3.629560288909e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.469091767778e-14 true resid norm > 2.028655945992e-01 ||r(i)||/||b|| 5.589260914581 > 2 KSP preconditioned resid norm 1.191764462912e-27 true resid norm > 1.834164299648e-14 ||r(i)||/||b|| 5.053406345813 > 1 SNES Function norm 1.672457508921e+00 > 107 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 4.062643571582e+00 > 0 KSP preconditioned resid norm 1.085854715813e-12 true resid norm > 4.062643571582e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.178769832441e-14 true resid norm > 1.118410859325e-01 ||r(i)||/||b|| 2.752914056129 > 2 KSP preconditioned resid norm 4.297181573967e-27 true resid norm > 2.628615092202e-14 ||r(i)||/||b|| 6.470208488357 > 1 SNES Function norm 9.036991011875e-01 > 108 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.105674791223e+00 > 0 KSP preconditioned resid norm 1.963614457879e-12 true resid norm > 4.105674791223e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.051107635909e-14 true resid norm > 8.277420683114e-02 ||r(i)||/||b|| 2.016092628868 > 2 KSP preconditioned resid norm 5.700654872204e-27 true resid norm > 2.220252793344e-14 ||r(i)||/||b|| 5.407765851524 > 1 SNES Function norm 6.003561441737e-01 > 109 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 3.053877992744e+00 > 0 KSP preconditioned resid norm 1.003248415294e-11 true resid norm > 3.053877992744e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.233646686238e-13 true resid norm > 5.577994771593e-01 ||r(i)||/||b|| 1.826528363230 > 2 KSP preconditioned resid norm 1.029901304608e-25 true resid norm > 1.936015096597e-13 ||r(i)||/||b|| 6.339529939299 > 1 SNES Function norm 3.053877992744e+00 > 0 SNES Function norm 5.829087412676e-01 > 0 KSP preconditioned resid norm 1.889341282348e-13 true resid norm > 5.829087412676e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.662933221169e-14 true resid norm > 5.989286100184e-02 ||r(i)||/||b|| 1.027482635989 > 2 KSP preconditioned resid norm 1.456379026302e-27 true resid norm > 7.668848909744e-15 ||r(i)||/||b|| 1.315617414326 > 1 SNES Function norm 4.254244459909e-01 > 110 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.180846267144e+00 > 0 KSP preconditioned resid norm 9.105704411362e-13 true resid norm > 1.180846267144e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.670199723048e-14 true resid norm > 5.057066847784e-02 ||r(i)||/||b|| 4.282578510423 > 2 KSP preconditioned resid norm 4.325921784021e-27 true resid norm > 1.275361754352e-14 ||r(i)||/||b|| 1.080040467449 > 1 SNES Function norm 3.911236882643e-01 > 111 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.532064854858e+00 > 0 KSP preconditioned resid norm 4.779679137769e-12 true resid norm > 1.532064854858e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.157872991170e-13 true resid norm > 2.098875091408e-01 ||r(i)||/||b|| 1.369964910266 > 2 KSP preconditioned resid norm 4.366911735365e-26 true resid norm > 7.239962529738e-14 ||r(i)||/||b|| 4.725624053563 > 1 SNES Function norm 1.532064854858e+00 > 0 SNES Function norm 1.764940070080e-01 > 0 KSP preconditioned resid norm 2.464402583943e-14 true resid norm > 1.764940070080e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.215297019902e-15 true resid norm > 2.047617209569e-02 ||r(i)||/||b|| 1.160162457797 > 2 KSP preconditioned resid norm 3.631831094940e-28 true resid norm > 2.355763139437e-15 ||r(i)||/||b|| 1.334755315136 > 1 SNES Function norm 1.706450714666e-01 > 112 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 4.407680008950e-01 > 0 KSP preconditioned resid norm 4.930548470884e-13 true resid norm > 4.407680008950e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.356345294181e-14 true resid norm > 7.551962516450e-02 ||r(i)||/||b|| 1.713364514011 > 2 KSP preconditioned resid norm 4.584264258499e-27 true resid norm > 1.292315985984e-14 ||r(i)||/||b|| 2.931964170176 > 1 SNES Function norm 4.407680008950e-01 > 0 SNES Function norm 1.064954076440e+00 > 0 KSP preconditioned resid norm 8.497424886833e-14 true resid norm > 1.064954076440e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.766159625691e-15 true resid norm > 2.490017627469e-02 ||r(i)||/||b|| 2.338145543133 > 2 KSP preconditioned resid norm 6.635425384718e-28 true resid norm > 8.234479883376e-15 ||r(i)||/||b|| 7.732239413465 > 1 SNES Function norm 2.964963371165e-01 > 113 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 8.622925709624e-01 > 0 KSP preconditioned resid norm 4.463457031339e-13 true resid norm > 8.622925709624e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.715977525563e-14 true resid norm > 1.329341326415e-01 ||r(i)||/||b|| 1.541636065508 > 2 KSP preconditioned resid norm 2.945289610486e-27 true resid norm > 1.273616756823e-14 ||r(i)||/||b|| 1.477012327036 > 1 SNES Function norm 8.622925709624e-01 > 0 SNES Function norm 2.612065167743e+00 > 0 KSP preconditioned resid norm 1.216980364706e-13 true resid norm > 2.612065167743e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 9.591070115731e-15 true resid norm > 1.117348786244e-01 ||r(i)||/||b|| 4.277645137044 > 2 KSP preconditioned resid norm 6.045627145118e-28 true resid norm > 1.388069047834e-14 ||r(i)||/||b|| 5.314067447381 > 1 SNES Function norm 1.331790654620e+00 > 114 TS dt 5.51805e-14 time 0.271565 > copy! > 0 SNES Function norm 3.606851714788e+00 > 0 KSP preconditioned resid norm 4.067733329812e-13 true resid norm > 3.606851714788e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.823877861556e-14 true resid norm > 1.077790820977e-01 ||r(i)||/||b|| 2.988176133103 > 2 KSP preconditioned resid norm 4.727810534551e-28 true resid norm > 7.519886149541e-15 ||r(i)||/||b|| 2.084889189846 > 1 SNES Function norm 9.013753678258e-01 > 115 TS dt 1.10361e-13 time 0.271565 > copy! > 0 SNES Function norm 3.227741616537e+00 > 0 KSP preconditioned resid norm 7.976107877122e-13 true resid norm > 3.227741616537e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 3.326079645062e-14 true resid norm > 1.010611637550e-01 ||r(i)||/||b|| 3.131017775316 > 2 KSP preconditioned resid norm 3.388023427292e-27 true resid norm > 1.918813178565e-14 ||r(i)||/||b|| 5.944754588577 > 1 SNES Function norm 7.734015655943e-01 > 116 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 2.393623504956e+00 > 0 KSP preconditioned resid norm 1.608960463713e-12 true resid norm > 2.393623504956e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 4.457509300959e-14 true resid norm > 6.395509588463e-02 ||r(i)||/||b|| 2.671894546164 > 2 KSP preconditioned resid norm 1.031673495966e-26 true resid norm > 2.303431837034e-14 ||r(i)||/||b|| 9.623200274667 > 1 SNES Function norm 4.969182786938e-01 > 117 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 2.596405006351e+00 > 0 KSP preconditioned resid norm 4.993386282508e-12 true resid norm > 2.596405006351e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.113373830574e-12 true resid norm > 7.480031452187e-01 ||r(i)||/||b|| 2.880918590856 > 2 KSP preconditioned resid norm 1.249282305831e-25 true resid norm > 2.418510317361e-13 ||r(i)||/||b|| 9.314842297119 > 1 SNES Function norm 2.596405006351e+00 > 0 SNES Function norm 4.974558747187e-01 > 0 KSP preconditioned resid norm 1.517572543094e-13 true resid norm > 4.974558747187e-01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.284261373562e-14 true resid norm > 3.075576674000e-02 ||r(i)||/||b|| 6.182612027126 > 2 KSP preconditioned resid norm 8.899015955837e-28 true resid norm > 5.037510039676e-15 ||r(i)||/||b|| 1.012654648520 > 1 SNES Function norm 2.909443551880e-01 > 118 TS dt 2.20722e-13 time 0.271565 > copy! > 0 SNES Function norm 1.088860095507e+00 > 0 KSP preconditioned resid norm 8.259380793768e-13 true resid norm > 1.088860095507e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.146289565587e-14 true resid norm > 8.801960855155e-02 ||r(i)||/||b|| 8.083647193494 > 2 KSP preconditioned resid norm 6.542880856158e-27 true resid norm > 1.745113143419e-14 ||r(i)||/||b|| 1.602697307597 > 1 SNES Function norm 6.203518870415e-01 > 119 TS dt 4.41444e-13 time 0.271565 > copy! > 0 SNES Function norm 1.828717001803e+00 > 0 KSP preconditioned resid norm 4.269088444122e-12 true resid norm > 1.828717001803e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.812126037081e-13 true resid norm > 1.188943176912e-01 ||r(i)||/||b|| 6.501515410749 > 2 KSP preconditioned resid norm 2.087655349967e-26 true resid norm > 4.228142299992e-14 ||r(i)||/||b|| 2.312081254685 > 1 SNES Function norm 9.407254842319e-01 > 120 TS dt 8.82888e-13 time 0.271565 > copy! > 0 SNES Function norm 2.970950728036e+00 > 0 KSP preconditioned resid norm 1.617443380237e-11 true resid norm > 2.970950728036e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.861956154914e-13 true resid norm > 2.920843259110e-01 ||r(i)||/||b|| 9.831341972610 > 2 KSP preconditioned resid norm 2.385942331865e-25 true resid norm > 2.019781312789e-13 ||r(i)||/||b|| 6.798434230930 > 1 SNES Function norm 2.277472194573e+00 > 121 TS dt 1.76578e-12 time 0.271565 > copy! > 0 SNES Function norm 6.655252327413e+00 > 0 KSP preconditioned resid norm 1.940972430816e-11 true resid norm > 6.655252327413e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.363154535742e-12 true resid norm > 6.207663299839e-01 ||r(i)||/||b|| 9.327464977202 > 2 KSP preconditioned resid norm 1.739471177685e-24 true resid norm > 6.857043712083e-13 ||r(i)||/||b|| 1.030320621179 > 1 SNES Function norm 3.860967926913e+00 > 122 TS dt 3.53155e-12 time 0.271565 > copy! > 0 SNES Function norm 8.141502067665e+00 > 0 KSP preconditioned resid norm 2.861146567034e-11 true resid norm > 8.141502067665e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.627735131533e-12 true resid norm > 1.790291714119e+00 ||r(i)||/||b|| 2.198969796040 > 2 KSP preconditioned resid norm 9.558838466264e-24 true resid norm > 2.239279225704e-12 ||r(i)||/||b|| 2.750449741453 > 1 SNES Function norm 6.426598300871e+00 > 123 TS dt 7.06311e-12 time 0.271565 > copy! > 0 SNES Function norm 1.224562198048e+01 > 0 KSP preconditioned resid norm 1.860092428595e-10 true resid norm > 1.224562198048e+01 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 1.412376301911e-11 true resid norm > 6.881778695473e-01 ||r(i)||/||b|| 5.619786978926 > 2 KSP preconditioned resid norm 3.119097349134e-23 true resid norm > 3.694807748646e-12 ||r(i)||/||b|| 3.017247923002 > 1 SNES Function norm 5.283049961509e+00 > 124 TS dt 1.41262e-11 time 0.271565 > copy! > 0 SNES Function norm 5.780124419709e+00 > 0 KSP preconditioned resid norm 2.355254806679e-10 true resid norm > 5.780124419709e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.026309613126e-12 true resid norm > 1.462739352259e-01 ||r(i)||/||b|| 2.530636446633 > 2 KSP preconditioned resid norm 4.007977791166e-23 true resid norm > 1.760862449704e-12 ||r(i)||/||b|| 3.046409249773 > 1 SNES Function norm 1.227257133866e+00 > 125 TS dt 2.82524e-11 time 0.271565 > copy! > 0 SNES Function norm 5.063237678461e+00 > 0 KSP preconditioned resid norm 1.132388220275e-09 true resid norm > 5.063237678461e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.141022667213e-11 true resid norm > 8.476518026367e-01 ||r(i)||/||b|| 1.674129986515 > 2 KSP preconditioned resid norm 6.201902296105e-22 true resid norm > 1.808797506267e-11 ||r(i)||/||b|| 3.572412794212 > 1 SNES Function norm 4.907594073483e+00 > 126 TS dt 5.65048e-11 time 0.271565 > copy! > 0 SNES Function norm 8.280289521084e+00 > 0 KSP preconditioned resid norm 1.016908346818e-09 true resid norm > 8.280289521084e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 7.542353161709e-11 true resid norm > 4.553108039388e-01 ||r(i)||/||b|| 5.498730482544 > 2 KSP preconditioned resid norm 1.286724699239e-21 true resid norm > 1.963867830074e-11 ||r(i)||/||b|| 2.371738119873 > 1 SNES Function norm 3.515932015486e+00 > 127 TS dt 1.1301e-10 time 0.271565 > copy! > 0 SNES Function norm 2.638351398877e+00 > 0 KSP preconditioned resid norm 8.318417737715e-10 true resid norm > 2.638351398877e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 2.886536821595e-11 true resid norm > 7.583100218372e-02 ||r(i)||/||b|| 2.874181286693 > 2 KSP preconditioned resid norm 1.101294596184e-21 true resid norm > 7.150867353793e-12 ||r(i)||/||b|| 2.710354411788 > 1 SNES Function norm 6.341356656368e-01 > 128 TS dt 2.26019e-10 time 0.271565 > copy! > 0 SNES Function norm 4.327811744723e+00 > 0 KSP preconditioned resid norm 9.707711632195e-09 true resid norm > 4.327811744723e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 8.220441545084e-10 true resid norm > 1.079734109206e+00 ||r(i)||/||b|| 2.494873097293 > 2 KSP preconditioned resid norm 5.565090370575e-20 true resid norm > 1.824401176064e-10 ||r(i)||/||b|| 4.215528039751 > 1 SNES Function norm 1.736633786129e-01 > 129 TS dt 4.52039e-10 time 0.271565 > copy! > 0 SNES Function norm 3.450425343262e+00 > 0 KSP preconditioned resid norm 5.403348982018e-08 true resid norm > 3.450425343262e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 6.555040648017e-09 true resid norm > 4.397639385930e+00 ||r(i)||/||b|| 1.274520949864 > 2 KSP preconditioned resid norm 8.635871880972e-19 true resid norm > 1.425658172221e-09 ||r(i)||/||b|| 4.131833123140 > 1 SNES Function norm 1.373376592274e+00 > 130 TS dt 9.04078e-10 time 0.271565 > copy! > 0 SNES Function norm 4.603810494450e+00 > 0 KSP preconditioned resid norm 4.317990603404e-07 true resid norm > 4.603810494450e+00 ||r(i)||/||b|| 1.000000000000 > 1 KSP preconditioned resid norm 5.666497114486e-08 true resid norm > 1.908034902688e+01 ||r(i)||/||b|| 4.144468815536 > 2 KSP preconditioned resid norm 1.483356956179e-17 true resid norm > 1.226697895774e-08 ||r(i)||/||b|| 2.664527345886 > 1 SNES Function norm 1.142888930566e+00 > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, January 5, 2021 2:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] convergence problem- 3D Cahn Hillard > > On Tue, Jan 5, 2021 at 3:14 PM Sepideh Kavousi wrote: > > Dear Petsc Users > I wrote a code in 2D solving PF equations for solidification of alloys. I > have two variables in the equations 1-non-dimensional concentration (U) 2- > order parameter (p). Everything is fine with this code. > When I change the code to 3D, the code has convergence problems which I > cannot solve it. > I know the term which is causing problem is a term in the equation for U: > (dp/dt)*(Pxx+Pyy+Pzz) > where Pxx,Pyy,Pzz are the second derivative of order parameter in x,y,z, > respectively. > > When I eliminate this term, the results of 2D and 3D are close to each > other. But adding that term the time step should be reduced a lot to obtain > convergence, otherwise the timestep does not converge. I have no idea how I > should solve this problem. > I solve the equations considering (I chose based on suggestions of Moose > for solving Cahn-hillard) > -ts_type bdf -ts_bdf_adapt -pc_type bjacobi -snes_linesearch_type l2 > -snes_type newtontr -ksp_type gmres -ksp_gmres_restart 1001 -sub_pc_type > ilu -sub_ksp_type preonly > > Would you please help me solve this issue? > > > In order to see the solver you have we need the output of > > -ts_view > > and then to see what is happening during the run we need > > -ts_monitor -snes_monitor -snes_linesearch_monitor > -ksp_monitor_true_residual > > Thanks, > > Matt > > > Best, > sepideh > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jan 5 22:31:45 2021 From: jed at jedbrown.org (Jed Brown) Date: Tue, 05 Jan 2021 21:31:45 -0700 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: Message-ID: <87o8i2sn2m.fsf@jedbrown.org> Matthew Knepley writes: > On Tue, Jan 5, 2021 at 9:52 PM Barry Smith wrote: > >> >> Ah, -snes_fd_color so it was already using finite differencing with >> coloring to compute the Jacobian which explains why the differences below >> are exactly zero. >> >> Implicit time-step schemes essentially add terms like I/dt to the >> Jacobian evaluation (and the function defining the ODE) so for tiny >> time-steps the nonlinear system gets easier and easier to solve (the >> nonlinear function becomes linear) But we didn't see that with your earlier >> run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps >> SNES still made no progress. It is hard to understand how this is possible, >> regardless of the problem you are solving. >> >> I would next run the code with valgrind to insure there are no issues of >> memory corruption or un-initialized data. >> >> How are you computing >> >> (dp/dt)*(Pxx+Pyy+Pzz) >> >> >> That is, how are you computing Pxx etc? >> >> Are you using finite elements for the U and P model? Exactly what elements? >> > > I agree with Barry. This does not seem to make sense, so I would expect > some kind of inconsistent discretization, or other > mathematical problem which makes your system unsolvable. Try -mat_fd_type ds before ruling out sensitivity to differencing parameter. From roland.richter at ntnu.no Wed Jan 6 01:41:42 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Wed, 6 Jan 2021 08:41:42 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: Hei, I added one additional function to the code: /void test_scaling_petsc_pointer(const Mat &in_mat,// //??? ??? ??? ??? ??? ??? ??? ??? Mat &out_mat,// //??? ??? ??? ??? ??? ??? ??? ??? const PetscScalar &scaling_factor) {// //??? MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);// //??? PetscScalar *mat_ptr;// //??? MatDenseGetArray (out_mat, &mat_ptr);// //??? PetscInt r_0, r_1;// //??? MatGetLocalSize (out_mat, &r_0, &r_1);// //??? for(int i = 0; i < r_0 * r_1; ++i)// //??? ??? *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);// // //??? MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);// //??? MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);// //}/ When replacing test function /test_scaling_petsc()/ with /test_scaling_petsc_pointer()/ everything works as it should, but I do not understand why. Do you have any suggestions? Thanks! *//* Am 05.01.21 um 15:24 schrieb Roland Richter: > > Hei, > > the code I attached to the original mail should work out of the box, > but requires armadillo and PETSc to compile/run. Armadillo stores the > data in column-major order, and therefore I am transposing the > matrices before and after transferring using .st(). > > Thank you for your help! > > Regards, > > Roland > > Am 05.01.21 um 15:21 schrieb Matthew Knepley: >> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter > > wrote: >> >> Hei, >> >> I would like to scale a given matrix with a fixed scalar value, and >> therefore would like to use MatScale(). Nevertheless, I observed an >> interesting behavior depending on the size of the matrix, and >> currently >> I am not sure why. >> >> When running the attached code, I intend to divide all elements >> in the >> matrix by a constant factor of 10. If I have three or fewer rows and >> 1024 columns, I get the expected result. If I have four or more rows >> (with the same number of columns), suddenly my scaling factor >> seems to >> be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based >> matrix >> still behaves as expected. >> >> >> 1) It looks like you assume the storage in your armadillo matrix is >> row major. I would be surprised if this was true. >> >> 2) I think it is unlikely that there is a problem?with MatScale, so I >> would guess either you have a memory overwrite >> or are misinterpreting your output. If you send something I can run, >> I will figure out which it is. >> >> ? Thanks, >> >> ? ? ?Matt >> ? >> >> I currently do not understand that behavior, but do not see any >> problems >> with the code either. Are there any possible explanations for >> that behavior? >> >> Thank you very much, >> >> regards, >> >> Roland Richter >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 6 08:26:02 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Jan 2021 09:26:02 -0500 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: On Wed, Jan 6, 2021 at 2:41 AM Roland Richter wrote: > Hei, > > I added one additional function to the code: > > *void test_scaling_petsc_pointer(const Mat &in_mat,* > * Mat &out_mat,* > * const PetscScalar &scaling_factor) {* > * MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);* > * PetscScalar *mat_ptr;* > * MatDenseGetArray (out_mat, &mat_ptr);* > * PetscInt r_0, r_1;* > * MatGetLocalSize (out_mat, &r_0, &r_1);* > * for(int i = 0; i < r_0 * r_1; ++i)* > * *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);* > > * MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);* > *}* > > When replacing test function *test_scaling_petsc()* with > *test_scaling_petsc_pointer()* everything works as it should, but I do > not understand why. > > Do you have any suggestions? > The easiest explanation is that you have a memory overwrite in the code somewhere. Barry's suggestion to use valgrind is good. Matt > Thanks! > > > Am 05.01.21 um 15:24 schrieb Roland Richter: > > Hei, > > the code I attached to the original mail should work out of the box, but > requires armadillo and PETSc to compile/run. Armadillo stores the data in > column-major order, and therefore I am transposing the matrices before and > after transferring using .st(). > > Thank you for your help! > > Regards, > > Roland > Am 05.01.21 um 15:21 schrieb Matthew Knepley: > > On Tue, Jan 5, 2021 at 7:57 AM Roland Richter > wrote: > >> Hei, >> >> I would like to scale a given matrix with a fixed scalar value, and >> therefore would like to use MatScale(). Nevertheless, I observed an >> interesting behavior depending on the size of the matrix, and currently >> I am not sure why. >> >> When running the attached code, I intend to divide all elements in the >> matrix by a constant factor of 10. If I have three or fewer rows and >> 1024 columns, I get the expected result. If I have four or more rows >> (with the same number of columns), suddenly my scaling factor seems to >> be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based matrix >> still behaves as expected. >> > > 1) It looks like you assume the storage in your armadillo matrix is row > major. I would be surprised if this was true. > > 2) I think it is unlikely that there is a problem with MatScale, so I > would guess either you have a memory overwrite > or are misinterpreting your output. If you send something I can run, I > will figure out which it is. > > Thanks, > > Matt > > >> I currently do not understand that behavior, but do not see any problems >> with the code either. Are there any possible explanations for that >> behavior? >> >> Thank you very much, >> >> regards, >> >> Roland Richter >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Wed Jan 6 10:05:25 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Wed, 6 Jan 2021 17:05:25 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: Hei, I ran the program in both versions using "valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all -malloc_debug". I got ==3059== LEAK SUMMARY: ==3059==??? definitely lost: 12,916 bytes in 32 blocks ==3059==??? indirectly lost: 2,415 bytes in 2 blocks ==3059==????? possibly lost: 0 bytes in 0 blocks ==3059==??? still reachable: 103,511 bytes in 123 blocks ==3059==???????? suppressed: 0 bytes in 0 blocks but none of the leaks is related to the scaling-function itself. Did I miss something here? Thanks! Am 06.01.21 um 15:26 schrieb Matthew Knepley: > On Wed, Jan 6, 2021 at 2:41 AM Roland Richter > wrote: > > Hei, > > I added one additional function to the code: > > /void test_scaling_petsc_pointer(const Mat &in_mat,// > //??? ??? ??? ??? ??? ??? ??? ??? Mat &out_mat,// > //??? ??? ??? ??? ??? ??? ??? ??? const PetscScalar > &scaling_factor) {// > //??? MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);// > //??? PetscScalar *mat_ptr;// > //??? MatDenseGetArray (out_mat, &mat_ptr);// > //??? PetscInt r_0, r_1;// > //??? MatGetLocalSize (out_mat, &r_0, &r_1);// > //??? for(int i = 0; i < r_0 * r_1; ++i)// > //??? ??? *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);// > // > //??? MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);// > //??? MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);// > //}/ > > When replacing test function /test_scaling_petsc()/ with > /test_scaling_petsc_pointer()/ everything works as it should, but > I do not understand why. > > Do you have any suggestions? > > The easiest explanation is that you have a memory overwrite in the > code somewhere. Barry's suggestion to use > valgrind is good. > > ? ?Matt? > > Thanks! > > > Am 05.01.21 um 15:24 schrieb Roland Richter: >> >> Hei, >> >> the code I attached to the original mail should work out of the >> box, but requires armadillo and PETSc to compile/run. Armadillo >> stores the data in column-major order, and therefore I am >> transposing the matrices before and after transferring using .st(). >> >> Thank you for your help! >> >> Regards, >> >> Roland >> >> Am 05.01.21 um 15:21 schrieb Matthew Knepley: >>> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter >>> > wrote: >>> >>> Hei, >>> >>> I would like to scale a given matrix with a fixed scalar >>> value, and >>> therefore would like to use MatScale(). Nevertheless, I >>> observed an >>> interesting behavior depending on the size of the matrix, >>> and currently >>> I am not sure why. >>> >>> When running the attached code, I intend to divide all >>> elements in the >>> matrix by a constant factor of 10. If I have three or fewer >>> rows and >>> 1024 columns, I get the expected result. If I have four or >>> more rows >>> (with the same number of columns), suddenly my scaling >>> factor seems to >>> be 0.01 instead of 0.1 for the PETSc-matrix. The >>> armadillo-based matrix >>> still behaves as expected. >>> >>> >>> 1) It looks like you assume the storage in your armadillo matrix >>> is row major. I would be surprised if this was true. >>> >>> 2) I think it is unlikely that there is a problem?with MatScale, >>> so I would guess either you have a memory overwrite >>> or are misinterpreting your output. If you send something I can >>> run, I will figure out which it is. >>> >>> ? Thanks, >>> >>> ? ? ?Matt >>> ? >>> >>> I currently do not understand that behavior, but do not see >>> any problems >>> with the code either. Are there any possible explanations >>> for that behavior? >>> >>> Thank you very much, >>> >>> regards, >>> >>> Roland Richter >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 6 10:36:32 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Jan 2021 11:36:32 -0500 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: On Wed, Jan 6, 2021 at 11:05 AM Roland Richter wrote: > Hei, > > I ran the program in both versions using "valgrind --tool=memcheck > --leak-check=full --show-leak-kinds=all -malloc_debug". I got > > ==3059== LEAK SUMMARY: > ==3059== definitely lost: 12,916 bytes in 32 blocks > ==3059== indirectly lost: 2,415 bytes in 2 blocks > ==3059== possibly lost: 0 bytes in 0 blocks > ==3059== still reachable: 103,511 bytes in 123 blocks > ==3059== suppressed: 0 bytes in 0 blocks > > but none of the leaks is related to the scaling-function itself. > > Did I miss something here? > > Here is my analysis. It is certainly the case that MatScale() does not mysteriously scale by other numbers. It is used all over the place in tests, and in the code. Your test requires another package. Thus, it seems reasonable to guess that a bad interaction with that package (memory overwrite, conflicting layout or format, etc.) is responsible for the behavior you see. Thanks, Matt > Thanks! > Am 06.01.21 um 15:26 schrieb Matthew Knepley: > > On Wed, Jan 6, 2021 at 2:41 AM Roland Richter > wrote: > >> Hei, >> >> I added one additional function to the code: >> >> *void test_scaling_petsc_pointer(const Mat &in_mat,* >> * Mat &out_mat,* >> * const PetscScalar &scaling_factor) {* >> * MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);* >> * PetscScalar *mat_ptr;* >> * MatDenseGetArray (out_mat, &mat_ptr);* >> * PetscInt r_0, r_1;* >> * MatGetLocalSize (out_mat, &r_0, &r_1);* >> * for(int i = 0; i < r_0 * r_1; ++i)* >> * *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);* >> >> * MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);* >> *}* >> >> When replacing test function *test_scaling_petsc()* with >> *test_scaling_petsc_pointer()* everything works as it should, but I do >> not understand why. >> >> Do you have any suggestions? >> > The easiest explanation is that you have a memory overwrite in the code > somewhere. Barry's suggestion to use > valgrind is good. > > Matt > >> Thanks! >> >> >> Am 05.01.21 um 15:24 schrieb Roland Richter: >> >> Hei, >> >> the code I attached to the original mail should work out of the box, but >> requires armadillo and PETSc to compile/run. Armadillo stores the data in >> column-major order, and therefore I am transposing the matrices before and >> after transferring using .st(). >> >> Thank you for your help! >> >> Regards, >> >> Roland >> Am 05.01.21 um 15:21 schrieb Matthew Knepley: >> >> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter >> wrote: >> >>> Hei, >>> >>> I would like to scale a given matrix with a fixed scalar value, and >>> therefore would like to use MatScale(). Nevertheless, I observed an >>> interesting behavior depending on the size of the matrix, and currently >>> I am not sure why. >>> >>> When running the attached code, I intend to divide all elements in the >>> matrix by a constant factor of 10. If I have three or fewer rows and >>> 1024 columns, I get the expected result. If I have four or more rows >>> (with the same number of columns), suddenly my scaling factor seems to >>> be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based matrix >>> still behaves as expected. >>> >> >> 1) It looks like you assume the storage in your armadillo matrix is row >> major. I would be surprised if this was true. >> >> 2) I think it is unlikely that there is a problem with MatScale, so I >> would guess either you have a memory overwrite >> or are misinterpreting your output. If you send something I can run, I >> will figure out which it is. >> >> Thanks, >> >> Matt >> >> >>> I currently do not understand that behavior, but do not see any problems >>> with the code either. Are there any possible explanations for that >>> behavior? >>> >>> Thank you very much, >>> >>> regards, >>> >>> Roland Richter >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Wed Jan 6 11:31:04 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Wed, 6 Jan 2021 18:31:04 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: Hei, I removed all dependencies to armadillo and other not directly necessary packages, and attached both CMakeLists.txt and main-file. Even though I am only having PETSc as main dependence I still have the same issues. For a scaling factor of 0.1 and a matrix size of [1024, 1024] it works fine, for a scaling factor of 0.01 on the same matrix the apparent scaling factor is suddenly 1e-8. Thank you for your help! Regards, Roland Am 06.01.21 um 17:36 schrieb Matthew Knepley: > On Wed, Jan 6, 2021 at 11:05 AM Roland Richter > wrote: > > Hei, > > I ran the program in both versions using "valgrind --tool=memcheck > --leak-check=full --show-leak-kinds=all -malloc_debug". I got > > ==3059== LEAK SUMMARY: > ==3059==??? definitely lost: 12,916 bytes in 32 blocks > ==3059==??? indirectly lost: 2,415 bytes in 2 blocks > ==3059==????? possibly lost: 0 bytes in 0 blocks > ==3059==??? still reachable: 103,511 bytes in 123 blocks > ==3059==???????? suppressed: 0 bytes in 0 blocks > > but none of the leaks is related to the scaling-function itself. > > Did I miss something here? > > Here is my analysis. It is certainly the case that MatScale() does not > mysteriously scale by other numbers. > It is used all over the place in tests, and in the code. Your test > requires another package. Thus, it seems > reasonable to guess that a bad interaction with that package (memory > overwrite, conflicting layout or format, etc.) > is responsible for the behavior you see. > > ? Thanks, > > ? ? ?Matt? > > Thanks! > > Am 06.01.21 um 15:26 schrieb Matthew Knepley: >> On Wed, Jan 6, 2021 at 2:41 AM Roland Richter >> > wrote: >> >> Hei, >> >> I added one additional function to the code: >> >> /void test_scaling_petsc_pointer(const Mat &in_mat,// >> //??? ??? ??? ??? ??? ??? ??? ??? Mat &out_mat,// >> //??? ??? ??? ??? ??? ??? ??? ??? const PetscScalar >> &scaling_factor) {// >> //??? MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);// >> //??? PetscScalar *mat_ptr;// >> //??? MatDenseGetArray (out_mat, &mat_ptr);// >> //??? PetscInt r_0, r_1;// >> //??? MatGetLocalSize (out_mat, &r_0, &r_1);// >> //??? for(int i = 0; i < r_0 * r_1; ++i)// >> //??? ??? *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);// >> // >> //??? MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);// >> //??? MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);// >> //}/ >> >> When replacing test function /test_scaling_petsc()/ with >> /test_scaling_petsc_pointer()/ everything works as it should, >> but I do not understand why. >> >> Do you have any suggestions? >> >> The easiest explanation is that you have a memory overwrite in >> the code somewhere. Barry's suggestion to use >> valgrind is good. >> >> ? ?Matt? >> >> Thanks! >> >> >> Am 05.01.21 um 15:24 schrieb Roland Richter: >>> >>> Hei, >>> >>> the code I attached to the original mail should work out of >>> the box, but requires armadillo and PETSc to compile/run. >>> Armadillo stores the data in column-major order, and >>> therefore I am transposing the matrices before and after >>> transferring using .st(). >>> >>> Thank you for your help! >>> >>> Regards, >>> >>> Roland >>> >>> Am 05.01.21 um 15:21 schrieb Matthew Knepley: >>>> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter >>>> > wrote: >>>> >>>> Hei, >>>> >>>> I would like to scale a given matrix with a fixed >>>> scalar value, and >>>> therefore would like to use MatScale(). Nevertheless, I >>>> observed an >>>> interesting behavior depending on the size of the >>>> matrix, and currently >>>> I am not sure why. >>>> >>>> When running the attached code, I intend to divide all >>>> elements in the >>>> matrix by a constant factor of 10. If I have three or >>>> fewer rows and >>>> 1024 columns, I get the expected result. If I have four >>>> or more rows >>>> (with the same number of columns), suddenly my scaling >>>> factor seems to >>>> be 0.01 instead of 0.1 for the PETSc-matrix. The >>>> armadillo-based matrix >>>> still behaves as expected. >>>> >>>> >>>> 1) It looks like you assume the storage in your armadillo >>>> matrix is row major. I would be surprised if this was true. >>>> >>>> 2) I think it is unlikely that there is a problem?with >>>> MatScale, so I would guess either you have a memory overwrite >>>> or are misinterpreting your output. If you send something I >>>> can run, I will figure out which it is. >>>> >>>> ? Thanks, >>>> >>>> ? ? ?Matt >>>> ? >>>> >>>> I currently do not understand that behavior, but do not >>>> see any problems >>>> with the code either. Are there any possible >>>> explanations for that behavior? >>>> >>>> Thank you very much, >>>> >>>> regards, >>>> >>>> Roland Richter >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: main.cpp Type: text/x-c++src Size: 2648 bytes Desc: not available URL: -------------- next part -------------- cmake_minimum_required(VERSION 3.12) set(PROJECT_NAME "armadillo_with_PETSc") project(${PROJECT_NAME} LANGUAGES CXX C) include(GNUInstallDirs) include(CheckLanguage) if(NOT WIN32) string(ASCII 27 Esc) set(ColourReset "${Esc}[m") set(ColourBold "${Esc}[1m") set(Red "${Esc}[31m") set(Green "${Esc}[32m") set(Yellow "${Esc}[33m") set(Blue "${Esc}[34m") set(Magenta "${Esc}[35m") set(Cyan "${Esc}[36m") set(White "${Esc}[37m") set(BoldRed "${Esc}[1;31m") set(BoldGreen "${Esc}[1;32m") set(BoldYellow "${Esc}[1;33m") set(BoldBlue "${Esc}[1;34m") set(BoldMagenta "${Esc}[1;35m") set(BoldCyan "${Esc}[1;36m") set(BoldWhite "${Esc}[1;37m") endif() check_language(CUDA) if(CMAKE_CUDA_COMPILER) message(STATUS "${BoldGreen}CUDA-support enabled${ColourReset}") enable_language (CUDA) set(CMAKE_CUDA_STANDARD 14) set(CMAKE_CUDA_STANDARD_REQUIRED ON) add_compile_definitions (USE_CUDA) else(CMAKE_CUDA_COMPILER) message(STATUS "${BoldRed}No CUDA support${ColourReset}") remove_definitions (USE_CUDA) endif() set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) cmake_policy (SET CMP0074 NEW) set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/") set(INCLUDE_PATH ${CMAKE_SOURCE_DIR}/include) set(SOURCE_PATH ${CMAKE_SOURCE_DIR}/source) set(EXECUTABLE_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/bin) set(LIBRARY_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/lib) if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native -fopenmp-simd") elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -xHost") endif() configure_file(downloadFindFFTW.cmake.in findFFTW-download/CMakeLists.txt) execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" . RESULT_VARIABLE result WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-download ) if(result) message(FATAL_ERROR "CMake step for findFFTW failed: ${result}") else() message("CMake step for findFFTW completed (${result}).") endif() execute_process(COMMAND ${CMAKE_COMMAND} --build . RESULT_VARIABLE result WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-download ) if(result) message(FATAL_ERROR "Build step for findFFTW failed: ${result}") endif() set(findFFTW_DIR ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-src) set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${findFFTW_DIR}") #Packages set(ENV{PETSC_DIR} /opt/petsc) find_package (PETSc REQUIRED) set(BOOST_ROOT "/opt/boost") find_package(Boost 1.72 REQUIRED COMPONENTS system filesystem mpi program_options serialization) pkg_check_modules(PETSc REQUIRED) find_package(GSL REQUIRED QUIET) find_package(MKL REQUIRED QUIET) set(FFTW_ROOT "/opt/fftw3") find_package(FFTW REQUIRED QUIET) find_package(OpenMP REQUIRED QUIET) set(EIGEN3_INCLUDE_DIRS "/usr/include/eigen3") set(PROJECT_SRC ${CMAKE_SOURCE_DIR}/source/main.cpp) set(PROJECT_INC "") INCLUDE_DIRECTORIES( ${INCLUDE_DIRS} ${UPPE_INCLUDE_PATH}) add_executable(${PROJECT_NAME} ${PROJECT_SRC} ${PROJECT_INC}) target_include_directories (${PROJECT_NAME} PRIVATE ${INCLUDE_PATH} ${PETSC_INCLUDE_DIRS} ${FFTW_INCLUDE_DIRS} ${GSL_INCLUDE_DIRS} ${MKL_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS}) target_compile_options(${PROJECT_NAME} PUBLIC -Wall -Wextra -pedantic -Werror -fPIC -flto -fopenmp -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -mavx -march=native) set(PROJECT_LINK_LIBRARIES ${EIGEN3_LIBRARIES} ${GSL_LIBRARIES} ${FFTW_LIBRARIES} /opt/fftw3/lib64/libfftw3_mpi.so gfortran /opt/intel/mkl/lib/intel64/libmkl_rt.so ${PETSC_LIBRARY_DIRS}/libpetsc.so Boost::filesystem Boost::mpi Boost::program_options Boost::serialization ${X11_LIBRARIES} OpenMP::OpenMP_CXX ) target_link_libraries(${PROJECT_NAME} ${PROJECT_LINK_LIBRARIES} ) From bsmith at petsc.dev Wed Jan 6 12:44:59 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 6 Jan 2021 12:44:59 -0600 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: $ ./main -start_in_debugger noxterm PETSC: Attaching lldb to ./main of pid 17914 on Barry-Smiths-MacBook-Pro.local (lldb) process attach --pid 17914 warning: (x86_64) /Users/barrysmith/soft/clang-ifort/lib/libmpifort.12.dylib empty dSYM file detected, dSYM was created with an executable with no debug info. warning: (x86_64) /Users/barrysmith/soft/clang-ifort/lib/libmpi.12.dylib empty dSYM file detected, dSYM was created with an executable with no debug info. warning: (x86_64) /Users/barrysmith/soft/clang-ifort/lib/libpmpi.12.dylib empty dSYM file detected, dSYM was created with an executable with no debug info. Process 17914 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x00007fff733cb756 libsystem_kernel.dylib`__semwait_signal + 10 libsystem_kernel.dylib`__semwait_signal: -> 0x7fff733cb756 <+10>: jae 0x7fff733cb760 ; <+20> 0x7fff733cb758 <+12>: movq %rax, %rdi 0x7fff733cb75b <+15>: jmp 0x7fff733ca22d ; cerror 0x7fff733cb760 <+20>: retq Target 0: (main) stopped. Executable module set to "/Users/barrysmith/Src/petsc/src/ksp/ksp/tutorials/main". Architecture set to: x86_64h-apple-macosx-. (lldb) b MatScale Breakpoint 1: where = libpetsc.3.014.dylib`MatScale + 48 at matrix.c:5281:3, address = 0x0000000102f62090 (lldb) c 1.0230000000000000e+03 Process 17914 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 frame #0: 0x0000000102f62090 libpetsc.3.014.dylib`MatScale(mat=0x00007fa5040c9870, a=0.01 + 0i) at matrix.c:5281:3 5278 { 5279 PetscErrorCode ierr; 5280 Target 0: (main) stopped. (lldb) c 0001e+01 1.0230000000000000e+01 [ 0]32 bytes PetscPushErrorHandler() line 161 in /Users/barrysmith/Src/petsc/src/sys/error/err.c Process 17914 exited with status = 0 (0x00000000) I see no issue. Code is, of course, built with complex. Barry > On Jan 6, 2021, at 11:31 AM, Roland Richter wrote: > > Hei, > > I removed all dependencies to armadillo and other not directly necessary packages, and attached both CMakeLists.txt and main-file. Even though I am only having PETSc as main dependence I still have the same issues. For a scaling factor of 0.1 and a matrix size of [1024, 1024] it works fine, for a scaling factor of 0.01 on the same matrix the apparent scaling factor is suddenly 1e-8. > > Thank you for your help! > > Regards, > > Roland > > Am 06.01.21 um 17:36 schrieb Matthew Knepley: >> On Wed, Jan 6, 2021 at 11:05 AM Roland Richter > wrote: >> Hei, >> >> I ran the program in both versions using "valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all -malloc_debug". I got >> >> ==3059== LEAK SUMMARY: >> ==3059== definitely lost: 12,916 bytes in 32 blocks >> ==3059== indirectly lost: 2,415 bytes in 2 blocks >> ==3059== possibly lost: 0 bytes in 0 blocks >> ==3059== still reachable: 103,511 bytes in 123 blocks >> ==3059== suppressed: 0 bytes in 0 blocks >> >> but none of the leaks is related to the scaling-function itself. >> >> Did I miss something here? >> >> Here is my analysis. It is certainly the case that MatScale() does not mysteriously scale by other numbers. >> It is used all over the place in tests, and in the code. Your test requires another package. Thus, it seems >> reasonable to guess that a bad interaction with that package (memory overwrite, conflicting layout or format, etc.) >> is responsible for the behavior you see. >> >> Thanks, >> >> Matt >> Thanks! >> >> Am 06.01.21 um 15:26 schrieb Matthew Knepley: >>> On Wed, Jan 6, 2021 at 2:41 AM Roland Richter > wrote: >>> Hei, >>> >>> I added one additional function to the code: >>> >>> void test_scaling_petsc_pointer(const Mat &in_mat, >>> Mat &out_mat, >>> const PetscScalar &scaling_factor) { >>> MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN); >>> PetscScalar *mat_ptr; >>> MatDenseGetArray (out_mat, &mat_ptr); >>> PetscInt r_0, r_1; >>> MatGetLocalSize (out_mat, &r_0, &r_1); >>> for(int i = 0; i < r_0 * r_1; ++i) >>> *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor); >>> >>> MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY); >>> } >>> >>> When replacing test function test_scaling_petsc() with test_scaling_petsc_pointer() everything works as it should, but I do not understand why. >>> >>> Do you have any suggestions? >>> >>> The easiest explanation is that you have a memory overwrite in the code somewhere. Barry's suggestion to use >>> valgrind is good. >>> >>> Matt >>> Thanks! >>> >>> >>> >>> Am 05.01.21 um 15:24 schrieb Roland Richter: >>>> Hei, >>>> >>>> the code I attached to the original mail should work out of the box, but requires armadillo and PETSc to compile/run. Armadillo stores the data in column-major order, and therefore I am transposing the matrices before and after transferring using .st(). >>>> >>>> Thank you for your help! >>>> >>>> Regards, >>>> >>>> Roland >>>> >>>> Am 05.01.21 um 15:21 schrieb Matthew Knepley: >>>>> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter > wrote: >>>>> Hei, >>>>> >>>>> I would like to scale a given matrix with a fixed scalar value, and >>>>> therefore would like to use MatScale(). Nevertheless, I observed an >>>>> interesting behavior depending on the size of the matrix, and currently >>>>> I am not sure why. >>>>> >>>>> When running the attached code, I intend to divide all elements in the >>>>> matrix by a constant factor of 10. If I have three or fewer rows and >>>>> 1024 columns, I get the expected result. If I have four or more rows >>>>> (with the same number of columns), suddenly my scaling factor seems to >>>>> be 0.01 instead of 0.1 for the PETSc-matrix. The armadillo-based matrix >>>>> still behaves as expected. >>>>> >>>>> 1) It looks like you assume the storage in your armadillo matrix is row major. I would be surprised if this was true. >>>>> >>>>> 2) I think it is unlikely that there is a problem with MatScale, so I would guess either you have a memory overwrite >>>>> or are misinterpreting your output. If you send something I can run, I will figure out which it is. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> I currently do not understand that behavior, but do not see any problems >>>>> with the code either. Are there any possible explanations for that behavior? >>>>> >>>>> Thank you very much, >>>>> >>>>> regards, >>>>> >>>>> Roland Richter >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Thu Jan 7 02:15:16 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Thu, 7 Jan 2021 09:15:16 +0100 Subject: [petsc-users] MatScale returns different results depending on matrix size In-Reply-To: References: <6880198c-b274-3766-b539-fd8dd1b3cc60@ntnu.no> Message-ID: <738536b7-c3f1-92d3-d5b4-11eeffea413c@ntnu.no> Hei, I think I found a solution, but no explanation for it (yet). In my CMakeLists-file I was linking the following libraries: ??? ${EIGEN3_LIBRARIES} ??? ${GSL_LIBRARIES} ??? ${FFTW_LIBRARIES} ??? /opt/fftw3/lib64/libfftw3_mpi.so ??? gfortran ??? /opt/intel/mkl/lib/intel64/libmkl_rt.so ??? ${PETSC_LIBRARY_DIRS}/libpetsc.so ??? Boost::filesystem ??? Boost::mpi ??? Boost::program_options ??? Boost::serialization ??? ${X11_LIBRARIES} ??? OpenMP::OpenMP_CXX with ??? ${FFTW_LIBRARIES} containing ??? libfftw3.so ??? libfftw3_omp.so Now, apparently the issue was a clash between the OpenMP-libraries and mkl_rt. When removing mkl_rt, the program works as expected, and similarly for when removing OpenMP::OpenMP_CXX and libfftw3_omp.so. As long as both options are present in the link list, I will obtain wrong results. Therefore, I'll now take a deeper look at what causes that behavior. Nevertheless, for the moment it looks as if I could solve the problem. Thank you very much for your help! Regards, Roland Am 06.01.21 um 19:44 schrieb Barry Smith: > > $ ./main -start_in_debugger noxterm > PETSC: Attaching lldb to ./main of pid 17914 on > Barry-Smiths-MacBook-Pro.local > (lldb) process attach --pid 17914 > warning: (x86_64) > /Users/barrysmith/soft/clang-ifort/lib/libmpifort.12.dylib empty dSYM > file detected, dSYM was created with an executable with no debug info. > warning: (x86_64) > /Users/barrysmith/soft/clang-ifort/lib/libmpi.12.dylib empty dSYM file > detected, dSYM was created with an executable with no debug info. > warning: (x86_64) > /Users/barrysmith/soft/clang-ifort/lib/libpmpi.12.dylib empty dSYM > file detected, dSYM was created with an executable with no debug info. > Process 17914 stopped > * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP > ? ? frame #0: > 0x00007fff733cb756libsystem_kernel.dylib`__semwait_signal + 10 > libsystem_kernel.dylib`__semwait_signal: > ->?0x7fff733cb756 <+10>: jae? ? 0x7fff733cb760? ? ? ? ? ? ; <+20> > ? ?0x7fff733cb758 <+12>: movq ? %rax, %rdi > ? ?0x7fff733cb75b <+15>: jmp? ? 0x7fff733ca22d? ? ? ? ? ? ; cerror > ? ?0x7fff733cb760 <+20>: retq? ? > Target 0: (main) stopped. > > Executable module set to > "/Users/barrysmith/Src/petsc/src/ksp/ksp/tutorials/main". > Architecture set to: x86_64h-apple-macosx-. > (lldb) b MatScale > Breakpoint 1: where = libpetsc.3.014.dylib`MatScale + 48 at > matrix.c:5281:3, address = 0x0000000102f62090 > (lldb) c > > > 1.0230000000000000e+03? > Process 17914 stopped > * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 > ? ? frame #0: > 0x0000000102f62090libpetsc.3.014.dylib`MatScale(mat=0x00007fa5040c9870, > a=0.01 + 0i) at matrix.c:5281:3 > ??5278{ > ??5279? PetscErrorCode ierr; > ??5280 > > Target 0: (main) stopped. > (lldb) c > > 0001e+01 1.0230000000000000e+01? > [ 0]32 bytes PetscPushErrorHandler() line 161 in > /Users/barrysmith/Src/petsc/src/sys/error/err.c > Process 17914 exited with status = 0 (0x00000000)? > > I see no issue. Code is, of course, built with complex. > > Barry > > >> On Jan 6, 2021, at 11:31 AM, Roland Richter > > wrote: >> >> Hei, >> >> I removed all dependencies to armadillo and other not directly >> necessary packages, and attached both CMakeLists.txt and main-file. >> Even though I am only having PETSc as main dependence I still have >> the same issues. For a scaling factor of 0.1 and a matrix size of >> [1024, 1024] it works fine, for a scaling factor of 0.01 on the same >> matrix the apparent scaling factor is suddenly 1e-8. >> >> Thank you for your help! >> >> Regards, >> >> Roland >> >> Am 06.01.21 um 17:36 schrieb Matthew Knepley: >>> On Wed, Jan 6, 2021 at 11:05 AM Roland Richter >>> > wrote: >>> >>> Hei, >>> >>> I ran the program in both versions using "valgrind >>> --tool=memcheck --leak-check=full --show-leak-kinds=all >>> -malloc_debug". I got >>> >>> ==3059== LEAK SUMMARY: >>> ==3059==??? definitely lost: 12,916 bytes in 32 blocks >>> ==3059==??? indirectly lost: 2,415 bytes in 2 blocks >>> ==3059==????? possibly lost: 0 bytes in 0 blocks >>> ==3059==??? still reachable: 103,511 bytes in 123 blocks >>> ==3059==???????? suppressed: 0 bytes in 0 blocks >>> >>> but none of the leaks is related to the scaling-function itself. >>> >>> Did I miss something here? >>> >>> Here is my analysis. It is certainly the case that MatScale() does >>> not mysteriously scale by other numbers. >>> It is used all over the place in tests, and in the code. Your test >>> requires another package. Thus, it seems >>> reasonable to guess that a bad interaction with that package (memory >>> overwrite, conflicting layout or format, etc.) >>> is responsible for the behavior you see. >>> >>> ? Thanks, >>> >>> ? ? ?Matt? >>> >>> Thanks! >>> >>> Am 06.01.21 um 15:26 schrieb Matthew Knepley: >>>> On Wed, Jan 6, 2021 at 2:41 AM Roland Richter >>>> > wrote: >>>> >>>> Hei, >>>> >>>> I added one additional function to the code: >>>> >>>> /void test_scaling_petsc_pointer(const Mat &in_mat,// >>>> //??? ??? ??? ??? ??? ??? ??? ??? Mat &out_mat,// >>>> //??? ??? ??? ??? ??? ??? ??? ??? const PetscScalar >>>> &scaling_factor) {// >>>> //??? MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);// >>>> //??? PetscScalar *mat_ptr;// >>>> //??? MatDenseGetArray (out_mat, &mat_ptr);// >>>> //??? PetscInt r_0, r_1;// >>>> //??? MatGetLocalSize (out_mat, &r_0, &r_1);// >>>> //??? for(int i = 0; i < r_0 * r_1; ++i)// >>>> //??? ??? *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);// >>>> // >>>> //??? MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);// >>>> //??? MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);// >>>> //}/ >>>> >>>> When replacing test function /test_scaling_petsc()/ with >>>> /test_scaling_petsc_pointer()/ everything works as it >>>> should, but I do not understand why. >>>> >>>> Do you have any suggestions? >>>> >>>> The easiest explanation is that you have a memory overwrite in >>>> the code somewhere. Barry's suggestion to use >>>> valgrind is good. >>>> >>>> ? ?Matt? >>>> >>>> Thanks! >>>> >>>> >>>> Am 05.01.21 um 15:24 schrieb Roland Richter: >>>>> >>>>> Hei, >>>>> >>>>> the code I attached to the original mail should work out >>>>> of the box, but requires armadillo and PETSc to >>>>> compile/run. Armadillo stores the data in column-major >>>>> order, and therefore I am transposing the matrices before >>>>> and after transferring using .st(). >>>>> >>>>> Thank you for your help! >>>>> >>>>> Regards, >>>>> >>>>> Roland >>>>> >>>>> Am 05.01.21 um 15:21 schrieb Matthew Knepley: >>>>>> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter >>>>>> > >>>>>> wrote: >>>>>> >>>>>> Hei, >>>>>> >>>>>> I would like to scale a given matrix with a fixed >>>>>> scalar value, and >>>>>> therefore would like to use MatScale(). Nevertheless, >>>>>> I observed an >>>>>> interesting behavior depending on the size of the >>>>>> matrix, and currently >>>>>> I am not sure why. >>>>>> >>>>>> When running the attached code, I intend to divide >>>>>> all elements in the >>>>>> matrix by a constant factor of 10. If I have three or >>>>>> fewer rows and >>>>>> 1024 columns, I get the expected result. If I have >>>>>> four or more rows >>>>>> (with the same number of columns), suddenly my >>>>>> scaling factor seems to >>>>>> be 0.01 instead of 0.1 for the PETSc-matrix. The >>>>>> armadillo-based matrix >>>>>> still behaves as expected. >>>>>> >>>>>> >>>>>> 1) It looks like you assume the storage in your armadillo >>>>>> matrix is row major. I would be surprised if this was true. >>>>>> >>>>>> 2) I think it is unlikely that there is a problem?with >>>>>> MatScale, so I would guess either you have a memory overwrite >>>>>> or are misinterpreting your output. If you send something >>>>>> I can run, I will figure out which it is. >>>>>> >>>>>> ? Thanks, >>>>>> >>>>>> ? ? ?Matt >>>>>> ? >>>>>> >>>>>> I currently do not understand that behavior, but do >>>>>> not see any problems >>>>>> with the code either. Are there any possible >>>>>> explanations for that behavior? >>>>>> >>>>>> Thank you very much, >>>>>> >>>>>> regards, >>>>>> >>>>>> Roland Richter >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they >>>>>> begin their experiments is infinitely more interesting >>>>>> than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Thu Jan 7 12:48:34 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 7 Jan 2021 18:48:34 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: <87o8i2sn2m.fsf@jedbrown.org> References: , <87o8i2sn2m.fsf@jedbrown.org> Message-ID: I use finite difference and, as an example, the discretization of Pxx is: Pxx=((aY[k][j][i+1].p+aY[k][j][i-1].p-2.0*aY[k][j][i].p)/hx2); I ran the code with valgrind and it seems there is a memory leak problem. I am trying to figure out what is causing the memory error. Best, Sepideh ________________________________ From: Jed Brown Sent: Tuesday, January 5, 2021 10:31 PM To: Matthew Knepley ; Barry Smith Cc: petsc-users at mcs.anl.gov ; Sepideh Kavousi Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard Matthew Knepley writes: > On Tue, Jan 5, 2021 at 9:52 PM Barry Smith wrote: > >> >> Ah, -snes_fd_color so it was already using finite differencing with >> coloring to compute the Jacobian which explains why the differences below >> are exactly zero. >> >> Implicit time-step schemes essentially add terms like I/dt to the >> Jacobian evaluation (and the function defining the ODE) so for tiny >> time-steps the nonlinear system gets easier and easier to solve (the >> nonlinear function becomes linear) But we didn't see that with your earlier >> run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps >> SNES still made no progress. It is hard to understand how this is possible, >> regardless of the problem you are solving. >> >> I would next run the code with valgrind to insure there are no issues of >> memory corruption or un-initialized data. >> >> How are you computing >> >> (dp/dt)*(Pxx+Pyy+Pzz) >> >> >> That is, how are you computing Pxx etc? >> >> Are you using finite elements for the U and P model? Exactly what elements? >> > > I agree with Barry. This does not seem to make sense, so I would expect > some kind of inconsistent discretization, or other > mathematical problem which makes your system unsolvable. Try -mat_fd_type ds before ruling out sensitivity to differencing parameter. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: valgrind-out.txt URL: From bsmith at petsc.dev Thu Jan 7 13:38:02 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 7 Jan 2021 13:38:02 -0600 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: <87o8i2sn2m.fsf@jedbrown.org> Message-ID: <810AC3BE-7496-4AA7-BD2A-CB6A0E478F17@petsc.dev> It looks like valgrind is being run on bash, not on your program, so these leaks are not relevant. by 0x41C482: main (in /usr/bin/bash) So the entire code is finite difference based? Barry > On Jan 7, 2021, at 12:48 PM, Sepideh Kavousi wrote: > > I use finite difference and, as an example, the discretization of Pxx is: > Pxx=((aY[k][j][i+1].p+aY[k][j][i-1].p-2.0*aY[k][j][i].p)/hx2); > > I ran the code with valgrind and it seems there is a memory leak problem. > I am trying to figure out what is causing the memory error. > Best, > Sepideh > > From: Jed Brown > > Sent: Tuesday, January 5, 2021 10:31 PM > To: Matthew Knepley >; Barry Smith > > Cc: petsc-users at mcs.anl.gov >; Sepideh Kavousi > > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard > > Matthew Knepley > writes: > > > On Tue, Jan 5, 2021 at 9:52 PM Barry Smith > wrote: > > > >> > >> Ah, -snes_fd_color so it was already using finite differencing with > >> coloring to compute the Jacobian which explains why the differences below > >> are exactly zero. > >> > >> Implicit time-step schemes essentially add terms like I/dt to the > >> Jacobian evaluation (and the function defining the ODE) so for tiny > >> time-steps the nonlinear system gets easier and easier to solve (the > >> nonlinear function becomes linear) But we didn't see that with your earlier > >> run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps > >> SNES still made no progress. It is hard to understand how this is possible, > >> regardless of the problem you are solving. > >> > >> I would next run the code with valgrind to insure there are no issues of > >> memory corruption or un-initialized data. > >> > >> How are you computing > >> > >> (dp/dt)*(Pxx+Pyy+Pzz) > >> > >> > >> That is, how are you computing Pxx etc? > >> > >> Are you using finite elements for the U and P model? Exactly what elements? > >> > > > > I agree with Barry. This does not seem to make sense, so I would expect > > some kind of inconsistent discretization, or other > > mathematical problem which makes your system unsolvable. > > Try -mat_fd_type ds before ruling out sensitivity to differencing parameter. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jan 8 11:28:41 2021 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 8 Jan 2021 12:28:41 -0500 Subject: [petsc-users] [petsc-dev] [petsc-maint] Proposal to improve MatSetValuesBlocked In-Reply-To: <18D5CDE9-9F68-4369-8AE7-7F44EAD3940E@nsc.liu.se> References: <7AB688E5-5247-486F-86AB-2FDD0C12D1D2@nsc.liu.se> <19356970-0463-4543-9E93-1E7FC6BFC875@nsc.liu.se> <06080FF8-1CDA-4831-9D8D-485D9AD35E5E@nsc.liu.se> <18D5CDE9-9F68-4369-8AE7-7F44EAD3940E@nsc.liu.se> Message-ID: please keep this in the list. Satish, Frank would like to make an MR. I assume you need to give him permission: bramkamp Thanks, Mark On Fri, Jan 8, 2021 at 11:28 AM Frank Bramkamp wrote: > > Hello Mark, > > my gitlab user name is: > > bramkamp > > real name: Frank Bramkamp > email: bramkamp at nsc.liu.se > > > Greetings, Frank > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Sun Jan 10 15:25:41 2021 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Sun, 10 Jan 2021 21:25:41 +0000 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: <810AC3BE-7496-4AA7-BD2A-CB6A0E478F17@petsc.dev> References: <87o8i2sn2m.fsf@jedbrown.org> , <810AC3BE-7496-4AA7-BD2A-CB6A0E478F17@petsc.dev> Message-ID: Yes, the entire code is finite difference. Is it possible that the large Laplacian term makes the equation stiff, such that the time step is reduced to solve the problem? Best, Sepideh ________________________________ From: Barry Smith Sent: Thursday, January 7, 2021 1:38 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard It looks like valgrind is being run on bash, not on your program, so these leaks are not relevant. by 0x41C482: main (in /usr/bin/bash) So the entire code is finite difference based? Barry On Jan 7, 2021, at 12:48 PM, Sepideh Kavousi > wrote: I use finite difference and, as an example, the discretization of Pxx is: Pxx=((aY[k][j][i+1].p+aY[k][j][i-1].p-2.0*aY[k][j][i].p)/hx2); I ran the code with valgrind and it seems there is a memory leak problem. I am trying to figure out what is causing the memory error. Best, Sepideh ________________________________ From: Jed Brown > Sent: Tuesday, January 5, 2021 10:31 PM To: Matthew Knepley >; Barry Smith > Cc: petsc-users at mcs.anl.gov >; Sepideh Kavousi > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard Matthew Knepley > writes: > On Tue, Jan 5, 2021 at 9:52 PM Barry Smith > wrote: > >> >> Ah, -snes_fd_color so it was already using finite differencing with >> coloring to compute the Jacobian which explains why the differences below >> are exactly zero. >> >> Implicit time-step schemes essentially add terms like I/dt to the >> Jacobian evaluation (and the function defining the ODE) so for tiny >> time-steps the nonlinear system gets easier and easier to solve (the >> nonlinear function becomes linear) But we didn't see that with your earlier >> run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps >> SNES still made no progress. It is hard to understand how this is possible, >> regardless of the problem you are solving. >> >> I would next run the code with valgrind to insure there are no issues of >> memory corruption or un-initialized data. >> >> How are you computing >> >> (dp/dt)*(Pxx+Pyy+Pzz) >> >> >> That is, how are you computing Pxx etc? >> >> Are you using finite elements for the U and P model? Exactly what elements? >> > > I agree with Barry. This does not seem to make sense, so I would expect > some kind of inconsistent discretization, or other > mathematical problem which makes your system unsolvable. Try -mat_fd_type ds before ruling out sensitivity to differencing parameter. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: valgrind-5.txt URL: From bsmith at petsc.dev Sun Jan 10 20:15:11 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 10 Jan 2021 20:15:11 -0600 Subject: [petsc-users] convergence problem- 3D Cahn Hillard In-Reply-To: References: <87o8i2sn2m.fsf@jedbrown.org> <810AC3BE-7496-4AA7-BD2A-CB6A0E478F17@petsc.dev> Message-ID: <6D48C767-B83D-4CF8-A33A-0D83F1F5F661@petsc.dev> > On Jan 10, 2021, at 3:25 PM, Sepideh Kavousi wrote: > > Yes, the entire code is finite difference. > Is it possible that the large Laplacian term makes the equation stiff, such that the time step is reduced to solve the problem? Yes, a bit, but no way near to this degree. What are your boundary conditions? Is it possible there is a null space lurking around, unlikely since the direct solver has no issues but if so it could cause difficulties. Barry > Best, > Sepideh > From: Barry Smith > > Sent: Thursday, January 7, 2021 1:38 PM > To: Sepideh Kavousi > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard > > > It looks like valgrind is being run on bash, not on your program, so these leaks are not relevant. by 0x41C482: main (in /usr/bin/bash) > > So the entire code is finite difference based? > > Barry > > >> On Jan 7, 2021, at 12:48 PM, Sepideh Kavousi > wrote: >> >> I use finite difference and, as an example, the discretization of Pxx is: >> Pxx=((aY[k][j][i+1].p+aY[k][j][i-1].p-2.0*aY[k][j][i].p)/hx2); >> >> I ran the code with valgrind and it seems there is a memory leak problem. >> I am trying to figure out what is causing the memory error. >> Best, >> Sepideh >> >> From: Jed Brown > >> Sent: Tuesday, January 5, 2021 10:31 PM >> To: Matthew Knepley >; Barry Smith > >> Cc: petsc-users at mcs.anl.gov >; Sepideh Kavousi > >> Subject: Re: [petsc-users] convergence problem- 3D Cahn Hillard >> >> Matthew Knepley > writes: >> >> > On Tue, Jan 5, 2021 at 9:52 PM Barry Smith > wrote: >> > >> >> >> >> Ah, -snes_fd_color so it was already using finite differencing with >> >> coloring to compute the Jacobian which explains why the differences below >> >> are exactly zero. >> >> >> >> Implicit time-step schemes essentially add terms like I/dt to the >> >> Jacobian evaluation (and the function defining the ODE) so for tiny >> >> time-steps the nonlinear system gets easier and easier to solve (the >> >> nonlinear function becomes linear) But we didn't see that with your earlier >> >> run where dt 3.72529e-13 (which is absurdly small). for tiny time-steps >> >> SNES still made no progress. It is hard to understand how this is possible, >> >> regardless of the problem you are solving. >> >> >> >> I would next run the code with valgrind to insure there are no issues of >> >> memory corruption or un-initialized data. >> >> >> >> How are you computing >> >> >> >> (dp/dt)*(Pxx+Pyy+Pzz) >> >> >> >> >> >> That is, how are you computing Pxx etc? >> >> >> >> Are you using finite elements for the U and P model? Exactly what elements? >> >> >> > >> > I agree with Barry. This does not seem to make sense, so I would expect >> > some kind of inconsistent discretization, or other >> > mathematical problem which makes your system unsolvable. >> >> Try -mat_fd_type ds before ruling out sensitivity to differencing parameter. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Mon Jan 11 05:41:05 2021 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Mon, 11 Jan 2021 12:41:05 +0100 Subject: [petsc-users] DMPlex with dof in cells AND at nodes Message-ID: Dear all, I haven't been on this mailing list since december, so first of all, happy new year to everyone ! I am facing another challenge with the DMPlex structure : although I am doing cell-centered finite volume (using PetscFV), I also need to do some manipulations and store information that are vertex-based. The problem I have is when I run the code in parallel : when I visualize the nodal data using a vtk viewer, it shows NaN at all the vertices in close proximity to the MPI partition line. Here is how I proceed to work on the vertex-based data : First comes the creation of the DM that I'll be using for the cell-centered FVM call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) ... dm = dmDist ... call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) ... dm = dmGhost ... I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in MPI the neighbouring relationships are conserved for all strata. Is that true ? Like, does doing this actually enable to get all the transitive closure of a given vertex, even if part of it is in another MPI block ? Then I need a vertex-based DM, so I figure it's pretty much what's done in CreateMassMatrix in TS ex11.c. call DMClone(dm, dmNodal, ierr) call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, ierr); call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); CHKERRA(ierr) call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); CHKERRA(ierr) do inode = vstart, vend-1 ! 3 dof per node, ndim = 3 call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) end do call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) >From what I understood, this new dmNodal basically has all the characteristics of the ori dm, except it's primary layer is vertex-based with 3 DOF per vertex. Then I get a local vector and work on it : call DMGetLocalVector(dmNodal, nodalvec, ierr) call VecGetArrayF90(nodalvec, nodalvecarray, ierr) call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) ! For each node ... do inode = vstart, vend-1 A = 0 ! matrix, ndim x ndim b = 0 ! vector, ndim call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, vertexclosure, ierr); ! For each face attached to that node do iface = 1, size(vertexclosure) if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= fend)) cycle A = A + ... b = b + ... end do nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, b) ... end do Then I want to visualize it : call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, ierr) call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); CHKERRA(ierr) call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) In the toto.vtk file, I get vertex-based values as expected, and the whole shebang works in sequential. Now when I run it in parallel, I get vertex-based values inside the MPI blocks, but in the neighbourhood of the join between the MPI blocks, all the vertex-based values are NaN (see attachment). All in all, I think I do not understand fully the consequences of the DMPlexDistribute with useCone=true & useClosure=true ... but I cannot figure out where the NaN are coming from : is it because the vertices do not actually know all their closure through the MPI join, or is it some Global / Local error I made ? Thank you very much in advance for your support !! Thibault -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nodal_velocity_00010.vtu Type: application/octet-stream Size: 20533 bytes Desc: not available URL: From knepley at gmail.com Mon Jan 11 09:24:42 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 11 Jan 2021 10:24:42 -0500 Subject: [petsc-users] DMPlex with dof in cells AND at nodes In-Reply-To: References: Message-ID: On Mon, Jan 11, 2021 at 6:41 AM Thibault Bridel-Bertomeu < thibault.bridelbertomeu at gmail.com> wrote: > Dear all, > > I haven't been on this mailing list since december, so first of all, happy > new year to everyone ! > > I am facing another challenge with the DMPlex structure : although I am > doing cell-centered finite volume (using PetscFV), I also need to do some > manipulations and store information that are vertex-based. > The problem I have is when I run the code in parallel : when I visualize > the nodal data using a vtk viewer, it shows NaN at all the vertices in > close proximity to the MPI partition line. > Here is how I proceed to work on the vertex-based data : > > First comes the creation of the DM that I'll be using for the > cell-centered FVM > > call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) > call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) > call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) > ... > dm = dmDist > ... > call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) > ... > dm = dmGhost > ... > > I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in MPI > the neighbouring relationships are conserved for all strata. Is that true ? > Like, does doing this actually enable to get all the transitive closure of > a given vertex, even if part of it is in another MPI block ? > What you say should be true, but it is not. When I designed everything, I was very much stuck in the traditional mesh way of looking at things, and made some limiting decisions. You can still get what you want (almost), but you need to distribute the mesh with an overlap of 1. If you do, then you can get the transitive support of vertices. The problem is basically that many places in the code assume that the overlap always contains the closure of any point in it. It is difficult to be able to switch this assumption to the star (transitive support)so I have not done that yet. > Then I need a vertex-based DM, so I figure it's pretty much what's done in > CreateMassMatrix in TS ex11.c. > > call DMClone(dm, dmNodal, ierr) > call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) > call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) > call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, ierr); > call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); > CHKERRA(ierr) > call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) > call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); CHKERRA(ierr) > do inode = vstart, vend-1 > ! 3 dof per node, ndim = 3 > call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) > end do > call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) > call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) > call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) > > From what I understood, this new dmNodal basically has all the > characteristics of the ori dm, except it's primary layer is vertex-based > with 3 DOF per vertex. > Take out the coordinate stuff above. This is now handled internally in a better way. Other than that, it looks good. > Then I get a local vector and work on it : > > call DMGetLocalVector(dmNodal, nodalvec, ierr) > call VecGetArrayF90(nodalvec, nodalvecarray, ierr) > call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) > call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) > ! For each node ... > do inode = vstart, vend-1 > A = 0 ! matrix, ndim x ndim > b = 0 ! vector, ndim > call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, > vertexclosure, ierr); > ! For each face attached to that node > do iface = 1, size(vertexclosure) > if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= fend)) > cycle > A = A + ... > b = b + ... > end do > nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, b) > ... > end do > There are some subtleties here I think. You are looping over all vertices, but you only own some of them. What do you do when putting these values in the global vector? I think there are at least three choices: 1) Everyone has everything, so you compute all the vertex stuff redundantly, and then use INSERT_VALUES for the scatter 2) Only vertices are redundant, so the transitive supports of vertices are partial but non-overlapping, and you use ADD_VALUES for the scatter 3) Only cells are redundant, so you do the complete vertex computation only for local vertices > Then I want to visualize it : > > call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) > call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) > call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, > ierr) > call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); CHKERRA(ierr) > call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) > call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) > call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) > call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) > > In the toto.vtk file, I get vertex-based values as expected, and the whole > shebang works in sequential. Now when I run it in parallel, I get > vertex-based values inside the MPI blocks, but in the neighbourhood of the > join between the MPI blocks, all the vertex-based values are NaN (see > attachment). > If you have redundant cells, meaning you distributed with overlap = 1, did you remember to full up the values for those ghost cells using DMGlobalToLocal() or something like that? If not, they will start as NaNs. Thanks, Matt > All in all, I think I do not understand fully the consequences of the > DMPlexDistribute with useCone=true & useClosure=true ... but I cannot > figure out where the NaN are coming from : is it because the vertices do > not actually know all their closure through the MPI join, or is it some > Global / Local error I made ? > > Thank you very much in advance for your support !! > > Thibault > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jan 11 13:59:41 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 11 Jan 2021 11:59:41 -0800 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation Message-ID: Dear PETSc Dev Team, The documentation recommends calling both of the above preallocation routines for simplicity. Do we waste memory by calling both? Thanks, Sam MATAIJMATAIJ = "aij" - A matrix type to be used for sparse matrices. This matrix type is identical to MATSEQAIJ when constructed with a single process communicator, and MATMPIAIJ otherwise. As a result, for single process communicators, MatSeqAIJSetPreallocation is supported, and similarly MatMPIAIJSetPreallocation () is supported for communicators controlling multiple processes. It is recommended that you call both of the above preallocation routines for simplicity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jan 11 14:25:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 11 Jan 2021 15:25:27 -0500 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: > Dear PETSc Dev Team, > The documentation recommends calling both of the above preallocation > routines for simplicity. Do we waste memory by calling both? > No. Only one will function, depending on the matrix type. Thanks, Matt > Thanks, > Sam > MATAIJMATAIJ > = > "aij" - A matrix type to be used for sparse matrices. This matrix type is > identical to MATSEQAIJ > when > constructed with a single process communicator, and MATMPIAIJ > otherwise. > As a result, for single process communicators, MatSeqAIJSetPreallocation > is > supported, and similarly MatMPIAIJSetPreallocation > () > is supported for communicators controlling multiple processes. It is > recommended that you call both of the above preallocation routines for > simplicity. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jan 11 14:27:19 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 11 Jan 2021 12:27:19 -0800 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: Thanks! On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley wrote: > On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: > >> Dear PETSc Dev Team, >> The documentation recommends calling both of the above preallocation >> routines for simplicity. Do we waste memory by calling both? >> > > No. Only one will function, depending on the matrix type. > > Thanks, > > Matt > > >> Thanks, >> Sam >> MATAIJMATAIJ >> = >> "aij" - A matrix type to be used for sparse matrices. This matrix type is >> identical to MATSEQAIJ >> when >> constructed with a single process communicator, and MATMPIAIJ >> otherwise. >> As a result, for single process communicators, MatSeqAIJSetPreallocation >> is >> supported, and similarly MatMPIAIJSetPreallocation >> () >> is supported for communicators controlling multiple processes. It is >> recommended that you call both of the above preallocation routines for >> simplicity. >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jan 11 15:31:58 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 11 Jan 2021 13:31:58 -0800 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: A follow up question: if I call preallocation, is there any performance difference between MatSetValues (mat,1,&I,1,&J,&v,INSERT_VALUES ); // insert value one by one vs MatSetValues (Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) // insert multiple values My input is triple arrays vector r; vector c; vector a; where r/c are not sorted by rows/cols. I don't want to waste memory/time to create idxm/idxn unless there is a performance penalty. Thanks, Sam On Mon, Jan 11, 2021 at 12:27 PM Sam Guo wrote: > Thanks! > > On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley > wrote: > >> On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: >> >>> Dear PETSc Dev Team, >>> The documentation recommends calling both of the above preallocation >>> routines for simplicity. Do we waste memory by calling both? >>> >> >> No. Only one will function, depending on the matrix type. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Sam >>> MATAIJMATAIJ >>> = >>> "aij" - A matrix type to be used for sparse matrices. This matrix type is >>> identical to MATSEQAIJ >>> when >>> constructed with a single process communicator, and MATMPIAIJ >>> otherwise. >>> As a result, for single process communicators, MatSeqAIJSetPreallocation >>> is >>> supported, and similarly MatMPIAIJSetPreallocation >>> () >>> is supported for communicators controlling multiple processes. It is >>> recommended that you call both of the above preallocation routines for >>> simplicity. >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jan 11 15:38:39 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 11 Jan 2021 16:38:39 -0500 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: On Mon, Jan 11, 2021 at 4:32 PM Sam Guo wrote: > A follow up question: if I call preallocation, is there any > performance difference between > > MatSetValues (mat,1,&I,1,&J,&v,INSERT_VALUES ); // insert value one by one > > vs > > MatSetValues (Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) // insert multiple values > > My input is triple arrays > > vector r; > > vector c; > > vector a; > > where r/c are not sorted by rows/cols. I don't want to waste memory/time to create idxm/idxn unless there is a performance penalty. > > It is cheaper to insert many values than sequences of single values. However, allocating memory is much much more expensive. As with most performance questions, there is no substitute for experiments. Thanks, Matt > Thanks, > > Sam > > > > On Mon, Jan 11, 2021 at 12:27 PM Sam Guo wrote: > >> Thanks! >> >> On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley >> wrote: >> >>> On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: >>> >>>> Dear PETSc Dev Team, >>>> The documentation recommends calling both of the above >>>> preallocation routines for simplicity. Do we waste memory by calling both? >>>> >>> >>> No. Only one will function, depending on the matrix type. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Sam >>>> MATAIJMATAIJ >>>> = >>>> "aij" - A matrix type to be used for sparse matrices. This matrix type is >>>> identical to MATSEQAIJ >>>> when >>>> constructed with a single process communicator, and MATMPIAIJ >>>> otherwise. >>>> As a result, for single process communicators, >>>> MatSeqAIJSetPreallocation >>>> is >>>> supported, and similarly MatMPIAIJSetPreallocation >>>> () >>>> is supported for communicators controlling multiple processes. It is >>>> recommended that you call both of the above preallocation routines for >>>> simplicity. >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jan 11 15:45:41 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 11 Jan 2021 13:45:41 -0800 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: Ok, thanks. On Mon, Jan 11, 2021 at 1:38 PM Matthew Knepley wrote: > On Mon, Jan 11, 2021 at 4:32 PM Sam Guo wrote: > >> A follow up question: if I call preallocation, is there any >> performance difference between >> >> MatSetValues (mat,1,&I,1,&J,&v,INSERT_VALUES ); // insert value one by one >> >> vs >> >> MatSetValues (Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) // insert multiple values >> >> My input is triple arrays >> >> vector r; >> >> vector c; >> >> vector a; >> >> where r/c are not sorted by rows/cols. I don't want to waste memory/time to create idxm/idxn unless there is a performance penalty. >> >> > It is cheaper to insert many values than sequences of single values. > However, allocating memory is much much more expensive. As with most > performance questions, there is no substitute for experiments. > > Thanks, > > Matt > > >> Thanks, >> >> Sam >> >> >> >> On Mon, Jan 11, 2021 at 12:27 PM Sam Guo wrote: >> >>> Thanks! >>> >>> On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley >>> wrote: >>> >>>> On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: >>>> >>>>> Dear PETSc Dev Team, >>>>> The documentation recommends calling both of the above >>>>> preallocation routines for simplicity. Do we waste memory by calling both? >>>>> >>>> >>>> No. Only one will function, depending on the matrix type. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Sam >>>>> MATAIJMATAIJ >>>>> = >>>>> "aij" - A matrix type to be used for sparse matrices. This matrix type is >>>>> identical to MATSEQAIJ >>>>> when >>>>> constructed with a single process communicator, and MATMPIAIJ >>>>> otherwise. >>>>> As a result, for single process communicators, >>>>> MatSeqAIJSetPreallocation >>>>> is >>>>> supported, and similarly MatMPIAIJSetPreallocation >>>>> () >>>>> is supported for communicators controlling multiple processes. It is >>>>> recommended that you call both of the above preallocation routines for >>>>> simplicity. >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jan 11 16:40:10 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 11 Jan 2021 16:40:10 -0600 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: Sam, In the master development branch of PETSc Stefano has started to add support for directly providing matrix entries in "coordinate" form (COO) (the triple arrays you use). It supports MatSetPreallocationCOO() and MatSetValuesCOO(). Currently Stefano has optimized it for use with GPUs but it will also work for CPUs and we can add optimizations in the future. You might consider using these in your code instead of managing the setting of each value individually. Barry > On Jan 11, 2021, at 3:45 PM, Sam Guo wrote: > > Ok, thanks. > > On Mon, Jan 11, 2021 at 1:38 PM Matthew Knepley > wrote: > On Mon, Jan 11, 2021 at 4:32 PM Sam Guo > wrote: > A follow up question: if I call preallocation, is there any performance difference between > MatSetValues (mat,1,&I,1,&J,&v,INSERT_VALUES ); // insert value one by one > vs > MatSetValues (Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) // insert multiple values > My input is triple arrays > vector r; > vector c; > vector a; > where r/c are not sorted by rows/cols. I don't want to waste memory/time to create idxm/idxn unless there is a performance penalty. > > It is cheaper to insert many values than sequences of single values. However, allocating memory is much much more expensive. As with most performance questions, there is no substitute for experiments. > > Thanks, > > Matt > > Thanks, > Sam > > > > > > > On Mon, Jan 11, 2021 at 12:27 PM Sam Guo > wrote: > Thanks! > > On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley > wrote: > On Mon, Jan 11, 2021 at 3:00 PM Sam Guo > wrote: > Dear PETSc Dev Team, > The documentation recommends calling both of the above preallocation routines for simplicity. Do we waste memory by calling both? > > No. Only one will function, depending on the matrix type. > > Thanks, > > Matt > > Thanks, > Sam > MATAIJ > > <>MATAIJ = "aij" - A matrix type to be used for sparse matrices. This matrix type is identical to MATSEQAIJ when constructed with a single process communicator, and MATMPIAIJ otherwise. As a result, for single process communicators, MatSeqAIJSetPreallocation is supported, and similarly MatMPIAIJSetPreallocation () is supported for communicators controlling multiple processes. It is recommended that you call both of the above preallocation routines for simplicity. > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shrirang.abhyankar at pnnl.gov Mon Jan 11 16:52:56 2021 From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G) Date: Mon, 11 Jan 2021 22:52:56 +0000 Subject: [petsc-users] Not displaying petsc options with -help Message-ID: I have an application where I set a bunch of application-specific options in PetscOptionsBegin/End and it uses the default PetscOptions database. When running the application with ?-help? option all the application and PETSc options get displayed. Is there a way to suppress displaying PETSc options, but only display the application options. Thanks, Shri -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jan 11 17:36:23 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 11 Jan 2021 18:36:23 -0500 Subject: [petsc-users] Not displaying petsc options with -help In-Reply-To: References: Message-ID: I don't think so. I use grep. On Mon, Jan 11, 2021 at 5:53 PM Abhyankar, Shrirang G via petsc-users < petsc-users at mcs.anl.gov> wrote: > I have an application where I set a bunch of application-specific options > in PetscOptionsBegin/End and it uses the default PetscOptions database. > When running the application with ?-help? option all the application and > PETSc options get displayed. Is there a way to suppress displaying PETSc > options, but only display the application options. > > > > Thanks, > > Shri > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Mon Jan 11 17:58:00 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Mon, 11 Jan 2021 15:58:00 -0800 Subject: [petsc-users] calling both MatSeqAIJSetPreallocation and MatMPIAIJSetPreallocation In-Reply-To: References: Message-ID: Thanks! On Mon, Jan 11, 2021 at 2:40 PM Barry Smith wrote: > > Sam, > > In the master development branch of PETSc Stefano has started to add > support for directly providing matrix entries in "coordinate" form (COO) > (the triple arrays you use). > > It supports MatSetPreallocationCOO() and MatSetValuesCOO(). > > Currently Stefano has optimized it for use with GPUs but it will also > work for CPUs and we can add optimizations in the future. > > You might consider using these in your code instead of managing the > setting of each value individually. > > Barry > > > On Jan 11, 2021, at 3:45 PM, Sam Guo wrote: > > Ok, thanks. > > On Mon, Jan 11, 2021 at 1:38 PM Matthew Knepley wrote: > >> On Mon, Jan 11, 2021 at 4:32 PM Sam Guo wrote: >> >>> A follow up question: if I call preallocation, is there any >>> performance difference between >>> >>> MatSetValues (mat,1,&I,1,&J,&v,INSERT_VALUES ); // insert value one by one >>> >>> vs >>> >>> MatSetValues (Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) // insert multiple values >>> >>> My input is triple arrays >>> >>> vector r; >>> >>> vector c; >>> >>> vector a; >>> >>> where r/c are not sorted by rows/cols. I don't want to waste memory/time to create idxm/idxn unless there is a performance penalty. >>> >>> >> It is cheaper to insert many values than sequences of single values. >> However, allocating memory is much much more expensive. As with most >> performance questions, there is no substitute for experiments. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Sam >>> >>> >>> >>> On Mon, Jan 11, 2021 at 12:27 PM Sam Guo wrote: >>> >>>> Thanks! >>>> >>>> On Mon, Jan 11, 2021 at 12:25 PM Matthew Knepley >>>> wrote: >>>> >>>>> On Mon, Jan 11, 2021 at 3:00 PM Sam Guo wrote: >>>>> >>>>>> Dear PETSc Dev Team, >>>>>> The documentation recommends calling both of the above >>>>>> preallocation routines for simplicity. Do we waste memory by calling both? >>>>>> >>>>> >>>>> No. Only one will function, depending on the matrix type. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Sam >>>>>> MATAIJMATAIJ >>>>>> = >>>>>> "aij" - A matrix type to be used for sparse matrices. This matrix type is >>>>>> identical to MATSEQAIJ >>>>>> when >>>>>> constructed with a single process communicator, and MATMPIAIJ >>>>>> otherwise. >>>>>> As a result, for single process communicators, >>>>>> MatSeqAIJSetPreallocation >>>>>> is >>>>>> supported, and similarly MatMPIAIJSetPreallocation >>>>>> () >>>>>> is supported for communicators controlling multiple processes. It is >>>>>> recommended that you call both of the above preallocation routines for >>>>>> simplicity. >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob.fai at gmail.com Mon Jan 11 18:02:24 2021 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Mon, 11 Jan 2021 18:02:24 -0600 Subject: [petsc-users] Not displaying petsc options with -help In-Reply-To: References: Message-ID: Not directly, but you can use PetscOptionsHasHelp() to query whether the user runs with ?-help?, print all of your available options and exit early. So you might do: PetscOptionsBegin(); < your options list here > PetscOptionsEnd(); PetscOptionsHasHelp(NULL, &set); If (set) PetscFinalize(); You?ll still get the header with the basic PetscOptions but it shouldn?t print all the other options. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: +1 (312) 694-3391 > On Jan 11, 2021, at 17:36, Mark Adams wrote: > > ? > I don't think so. I use grep. > >> On Mon, Jan 11, 2021 at 5:53 PM Abhyankar, Shrirang G via petsc-users wrote: >> I have an application where I set a bunch of application-specific options in PetscOptionsBegin/End and it uses the default PetscOptions database. When running the application with ?-help? option all the application and PETSc options get displayed. Is there a way to suppress displaying PETSc options, but only display the application options. >> >> >> >> Thanks, >> >> Shri >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jan 11 19:13:19 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 11 Jan 2021 19:13:19 -0600 Subject: [petsc-users] Not displaying petsc options with -help In-Reply-To: References: Message-ID: <09681912-7B24-4C6C-8B51-0DE5627EFAD7@petsc.dev> Change the value of PetscHelpPrintf() to do nothing before PetscInitialize() and then change it PetscHelpPrintfDefault() just around your options. I have fixed up the manual page in https://gitlab.com/petsc/petsc/-/merge_requests/3553 But are you sure you want to do this? Even -ksp_monitor etc will no longer produce help messages which users may want. Barry > On Jan 11, 2021, at 4:52 PM, Abhyankar, Shrirang G via petsc-users wrote: > > I have an application where I set a bunch of application-specific options in PetscOptionsBegin/End and it uses the default PetscOptions database. When running the application with ?-help? option all the application and PETSc options get displayed. Is there a way to suppress displaying PETSc options, but only display the application options. > > Thanks, > Shri -------------- next part -------------- An HTML attachment was scrubbed... URL: From shrirang.abhyankar at pnnl.gov Mon Jan 11 22:48:09 2021 From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G) Date: Tue, 12 Jan 2021 04:48:09 +0000 Subject: [petsc-users] Not displaying petsc options with -help In-Reply-To: <09681912-7B24-4C6C-8B51-0DE5627EFAD7@petsc.dev> References: <09681912-7B24-4C6C-8B51-0DE5627EFAD7@petsc.dev> Message-ID: <231D1956-6A82-4334-AE3F-AFC476639D58@pnnl.gov> I have the application options prefixed so I can grep for them, as Mark suggested. But, we were looking for ways to only display the application options with ?-help? without using grep. But are you sure you want to do this? Even -ksp_monitor etc will no longer produce help messages which users may want. Hmm..for the power grid application users, they don?t really care much about the solver details so not displaying -xxx_monitor would be fine. But, as developers we?d like to know what the solver is doing so we?d definitely want them to be displayed. I?ll try your suggestions. Thank you for your suggestions - Mark, Jacob, and Barry. Thanks, Shri Another possibility, perhaps the easiest, would be to just create a different help option name for the application and not fiddle with changing the behavior of ?-help?. From: Barry Smith Date: Monday, January 11, 2021 at 7:13 PM To: "Abhyankar, Shrirang G" Cc: PETSc Users , "Mancinelli, Asher J" Subject: Re: [petsc-users] Not displaying petsc options with -help Check twice before you click! This email originated from outside PNNL. Change the value of PetscHelpPrintf() to do nothing before PetscInitialize() and then change it PetscHelpPrintfDefault() just around your options. I have fixed up the manual page in https://gitlab.com/petsc/petsc/-/merge_requests/3553 But are you sure you want to do this? Even -ksp_monitor etc will no longer produce help messages which users may want. Barry On Jan 11, 2021, at 4:52 PM, Abhyankar, Shrirang G via petsc-users > wrote: I have an application where I set a bunch of application-specific options in PetscOptionsBegin/End and it uses the default PetscOptions database. When running the application with ?-help? option all the application and PETSc options get displayed. Is there a way to suppress displaying PETSc options, but only display the application options. Thanks, Shri -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Tue Jan 12 10:41:57 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 12 Jan 2021 09:41:57 -0700 Subject: [petsc-users] counter->tag = *maxval - 128 Message-ID: Hi All, I am curious about why we subtract 128 from the max value of tag? Can we directly use the max tag value? Thanks, Fande, PetscErrorCode PetscCommGetNewTag(MPI_Comm comm,PetscMPIInt *tag) { PetscErrorCode ierr; PetscCommCounter *counter; PetscMPIInt *maxval,flg; MPI_Comm_get_attr(comm,Petsc_Counter_keyval,&counter,&flg); if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI communicator supplied; must be a PETSc communicator"); if (counter->tag < 1) { PetscInfo1(NULL,"Out of tags for object, starting to recycle. Comm reference count %d\n",counter->refcount); MPI_Comm_get_attr(MPI_COMM_WORLD,MPI_TAG_UB,&maxval,&flg); if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"MPI error: MPI_Comm_get_attr() is not returning a MPI_TAG_UB"); counter->tag = *maxval - 128; /* hope that any still active tags were issued right at the beginning of the run */ } *tag = counter->tag--; if (PetscDefined(USE_DEBUG)) { /* Hanging here means that some processes have called PetscCommGetNewTag() and others have not. */ MPI_Barrier(comm); } return(0); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Tue Jan 12 15:12:42 2021 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Tue, 12 Jan 2021 22:12:42 +0100 Subject: [petsc-users] DMPlex with dof in cells AND at nodes In-Reply-To: References: Message-ID: Good evening Matthew, Thank you for your answer, and sorry for the delay in mine ! I am trying to figure things out but when one gets in the distribution of a DMPlex, it becomes quite complex! Le lun. 11 janv. 2021 ? 16:24, Matthew Knepley a ?crit : > On Mon, Jan 11, 2021 at 6:41 AM Thibault Bridel-Bertomeu < > thibault.bridelbertomeu at gmail.com> wrote: > >> Dear all, >> >> I haven't been on this mailing list since december, so first of all, >> happy new year to everyone ! >> >> I am facing another challenge with the DMPlex structure : although I am >> doing cell-centered finite volume (using PetscFV), I also need to do some >> manipulations and store information that are vertex-based. >> The problem I have is when I run the code in parallel : when I visualize >> the nodal data using a vtk viewer, it shows NaN at all the vertices in >> close proximity to the MPI partition line. >> Here is how I proceed to work on the vertex-based data : >> >> First comes the creation of the DM that I'll be using for the >> cell-centered FVM >> >> call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) >> call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) >> call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) >> ... >> dm = dmDist >> ... >> call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) >> ... >> dm = dmGhost >> ... >> >> I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in MPI >> the neighbouring relationships are conserved for all strata. Is that true ? >> Like, does doing this actually enable to get all the transitive closure of >> a given vertex, even if part of it is in another MPI block ? >> > > What you say should be true, but it is not. When I designed everything, I > was very much stuck in the traditional mesh way of looking at things, and > made some limiting decisions. You can still get what you want (almost), but > you need to distribute the mesh with an overlap of 1. If you do, then you > can get the transitive support of vertices. > I actually already distribute the mesh with an overlap of 1, but the result is the same. Do I need to re-distribute the clone "dmNodal" somehow ? Is the cloning process maybe not preserving the MPI support of the different strata ? > > The problem is basically that many places in the code assume that the > overlap always contains the closure of any point in it. It is difficult to > be able to switch this assumption to the star (transitive support)so I have > not done that yet. > > >> Then I need a vertex-based DM, so I figure it's pretty much what's done >> in CreateMassMatrix in TS ex11.c. >> >> call DMClone(dm, dmNodal, ierr) >> call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) >> call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) >> call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, ierr); >> call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); >> CHKERRA(ierr) >> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) >> call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); CHKERRA(ierr) >> do inode = vstart, vend-1 >> ! 3 dof per node, ndim = 3 >> call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) >> end do >> call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) >> call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) >> call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) >> >> From what I understood, this new dmNodal basically has all the >> characteristics of the ori dm, except it's primary layer is vertex-based >> with 3 DOF per vertex. >> > > Take out the coordinate stuff above. This is now handled internally in a > better way. Other than that, it looks good. > You mean the SetCoordinateSection is already handled by the cloning process ? > > >> Then I get a local vector and work on it : >> >> call DMGetLocalVector(dmNodal, nodalvec, ierr) >> call VecGetArrayF90(nodalvec, nodalvecarray, ierr) >> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) >> call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) >> ! For each node ... >> do inode = vstart, vend-1 >> A = 0 ! matrix, ndim x ndim >> b = 0 ! vector, ndim >> call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, >> vertexclosure, ierr); >> ! For each face attached to that node >> do iface = 1, size(vertexclosure) >> if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= fend)) >> cycle >> A = A + ... >> b = b + ... >> end do >> nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, b) >> ... >> end do >> > > There are some subtleties here I think. You are looping over all vertices, > but you only own some of them. > Here I am quite surprised by your remark : from the random print I did, the DMPlexGetDepthStratum returns different vstart and vend for different MPI processes, hence I assumed it was a local numbering. And then I assumed that, with the DMPlexDistribute, it ensured that the "vertexclosure" contained (among others) all the faces connected to each "inode", even if the closure is spread among multiple MPI processes. So you mean that all these assumptions are actually wrong to some degree ? > What do you do when putting these values in the global vector? I think > there are at least three choices: > > 1) Everyone has everything, so you compute all the vertex stuff > redundantly, and then use INSERT_VALUES for the scatter > > 2) Only vertices are redundant, so the transitive supports of vertices are > partial but non-overlapping, and you use ADD_VALUES for the scatter > > 3) Only cells are redundant, so you do the complete vertex computation > only for local vertices > I distributed with overlap = 1, so I expect I have a halo of redundant cells in each MPI block (option no. 3). With the (..., PETSC_TRUE, PETSC_TRUE, ...) setting of the distribute however, I also expected to have a halo of vertices and a halo of edges and a halo of faces. Therefore, completing the vertex computation only for local vertices should have been fine since anyways even local vertices know their complete closure in terms of faces, but I am missing something because it does not turn out right ... > > >> Then I want to visualize it : >> >> call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) >> call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) >> call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, >> ierr) >> call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); CHKERRA(ierr) >> call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) >> call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) >> call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) >> call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) >> >> In the toto.vtk file, I get vertex-based values as expected, and the >> whole shebang works in sequential. Now when I run it in parallel, I get >> vertex-based values inside the MPI blocks, but in the neighbourhood of the >> join between the MPI blocks, all the vertex-based values are NaN (see >> attachment). >> > > If you have redundant cells, meaning you distributed with overlap = 1, did > you remember to full up the values for those ghost cells using > DMGlobalToLocal() or something like that? If not, they will start as NaNs. > Indeed, in the vertex computation, at some point I use the values in the cells. But I got the solution vector from TSGetSolution, then I do a GlobalToLocal, then I even add a InsertBoundaryValues to fill the ghost cells correctly. Even by doing this, when I print the nodalvecarray on each process, they somehow contain some NaNs. This means, I think, that at some point, a given vertex tries to get the information of a face that is supposed to be in its closure but is actually not .. (?) Thank you for your help and feedback ! Cheers, Thibault > > Thanks, > > Matt > > >> All in all, I think I do not understand fully the consequences of the >> DMPlexDistribute with useCone=true & useClosure=true ... but I cannot >> figure out where the NaN are coming from : is it because the vertices do >> not actually know all their closure through the MPI join, or is it some >> Global / Local error I made ? >> >> Thank you very much in advance for your support !! >> >> Thibault >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jan 12 15:51:07 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 12 Jan 2021 15:51:07 -0600 Subject: [petsc-users] counter->tag = *maxval - 128 In-Reply-To: References: Message-ID: I think you can, if you always use PetscCommGetNewTag() to get a tag for use with a PetscComm. The -128 is to avoid wrong usages that someone provided their own tag from the maximal. --Junchao Zhang On Tue, Jan 12, 2021 at 10:42 AM Fande Kong wrote: > Hi All, > > I am curious about why we subtract 128 from the max value of tag? Can we > directly use the max tag value? > > Thanks, > > Fande, > > > PetscErrorCode PetscCommGetNewTag(MPI_Comm comm,PetscMPIInt *tag) > { > PetscErrorCode ierr; > PetscCommCounter *counter; > PetscMPIInt *maxval,flg; > > > MPI_Comm_get_attr(comm,Petsc_Counter_keyval,&counter,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI > communicator supplied; must be a PETSc communicator"); > > if (counter->tag < 1) { > PetscInfo1(NULL,"Out of tags for object, starting to recycle. Comm > reference count %d\n",counter->refcount); > MPI_Comm_get_attr(MPI_COMM_WORLD,MPI_TAG_UB,&maxval,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"MPI error: > MPI_Comm_get_attr() is not returning a MPI_TAG_UB"); > counter->tag = *maxval - 128; /* hope that any still active tags were > issued right at the beginning of the run */ > } > > *tag = counter->tag--; > if (PetscDefined(USE_DEBUG)) { > /* > Hanging here means that some processes have called > PetscCommGetNewTag() and others have not. > */ > MPI_Barrier(comm); > } > return(0); > } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jan 12 19:49:01 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 12 Jan 2021 19:49:01 -0600 Subject: [petsc-users] counter->tag = *maxval - 128 In-Reply-To: References: Message-ID: <7AE622D5-1B5B-4DBA-A791-688E925DE4BE@petsc.dev> Fande, /* hope that any still active tags were issued right at the beginning of the run */ PETSc actually starts with *maxval (see line 130). It is only when it runs out that it does this silly thing for the reason indicated in the comment. PETSc should actually keep track which of which tags have been "returned" and if the counter gets to zero use those returned tags instead of starting again at the top which could clash with the same value used for another reason. In other words the current code is buggy, but it has always been "good enough". Barry > On Jan 12, 2021, at 10:41 AM, Fande Kong wrote: > > Hi All, > > I am curious about why we subtract 128 from the max value of tag? Can we directly use the max tag value? > > Thanks, > > Fande, > > > PetscErrorCode PetscCommGetNewTag(MPI_Comm comm,PetscMPIInt *tag) > { > PetscErrorCode ierr; > PetscCommCounter *counter; > PetscMPIInt *maxval,flg; > > > MPI_Comm_get_attr(comm,Petsc_Counter_keyval,&counter,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI communicator supplied; must be a PETSc communicator"); > > if (counter->tag < 1) { > PetscInfo1(NULL,"Out of tags for object, starting to recycle. Comm reference count %d\n",counter->refcount); > MPI_Comm_get_attr(MPI_COMM_WORLD,MPI_TAG_UB,&maxval,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"MPI error: MPI_Comm_get_attr() is not returning a MPI_TAG_UB"); > counter->tag = *maxval - 128; /* hope that any still active tags were issued right at the beginning of the run */ > } > > *tag = counter->tag--; > if (PetscDefined(USE_DEBUG)) { > /* > Hanging here means that some processes have called PetscCommGetNewTag() and others have not. > */ > MPI_Barrier(comm); > } > return(0); > } -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Wed Jan 13 06:39:56 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Wed, 13 Jan 2021 13:39:56 +0100 Subject: [petsc-users] Understanding global and local rows in a distributed dense matrix Message-ID: Hei, I am currently struggling a bit with my understanding of local and global rows/columns in distributed dense matrices in PETSc. The attached short program is creating a local matrix in every thread with [matrix_size_rows, matrix_size_cols] as global size. Afterwards, I flatten the matrix into a vector, and split this vector using the boost-function split() when running using several MPI threads. In addition I create a dense MPI-distributed matrix based on PETSc, using the same input sizes, and set all values to zero (after I'm not interested in the content itself, just the shape). Finally, I retrieve the global and local values for the number of rows and columns using MatGetLocalSize, and the number of the first and last row in each matrix using MatGetOwnershipRange() and print them. According to the documentation I would expect that the PETSc-based matrix is split up Now, for an input size of [8, 8] and a single thread I get /Rank 0 has a total size of 8, 8 Rank 0 has an initial size of 64 Rank 0 has a local matrix size of 8, 8 Rank 0 has a global matrix size of 8, 8 Rank 0 has a matrix spanning from row 0 to 8 Rank 0 has a vector size of 64/// which is what I expect. For two threads, though, I get /Rank 0 has a total size of 8, 8// //Rank 0 has an initial size of 64// //Rank 0 has a local matrix size of 4, 4// //Rank 0 has a global matrix size of 8, 8// //Rank 0 has a matrix spanning from row 0 to 4// //Rank 0 has a vector size of 32// //Rank 1 has a total size of 8, 8// //Rank 1 has an initial size of 64// //Rank 1 has a local matrix size of 4, 4// //Rank 1 has a global matrix size of 8, 8// //Rank 1 has a matrix spanning from row 4 to 8// Rank 1 has a vector size of 32/ Here, most entries make sense, except the size of the local matrices. Why do I get a size of [4, 4], and not a size of [4, 8]? Each row should be contiguous in the local process, and therefore each row should contain all columns, not only a part of it. Is there a misunderstanding about how to use MatGetLocalSize(), or something else? Thanks! Regards, Roland Richter / / -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: main_mpi_short.cpp Type: text/x-c++src Size: 2579 bytes Desc: not available URL: From stefano.zampini at gmail.com Wed Jan 13 07:26:03 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Wed, 13 Jan 2021 16:26:03 +0300 Subject: [petsc-users] Understanding global and local rows in a distributed dense matrix In-Reply-To: References: Message-ID: MATMPIDENSE does not implement any cyclic distribution. In parallel, a dense matrix is split by rows. Each process owns localrows*globalcols entries. Local sizes are to be intended as the size of the right and left vectors used in matvec operations, and are not strictly related with storage considerations. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSize.html Il Mer 13 Gen 2021, 15:40 Roland Richter ha scritto: > Hei, > > I am currently struggling a bit with my understanding of local and global > rows/columns in distributed dense matrices in PETSc. > > The attached short program is creating a local matrix in every thread with > [matrix_size_rows, matrix_size_cols] as global size. Afterwards, I flatten > the matrix into a vector, and split this vector using the boost-function > split() when running using several MPI threads. In addition I create a > dense MPI-distributed matrix based on PETSc, using the same input sizes, > and set all values to zero (after I'm not interested in the content itself, > just the shape). Finally, I retrieve the global and local values for the > number of rows and columns using MatGetLocalSize, and the number of the > first and last row in each matrix using MatGetOwnershipRange() and print > them. According to the documentation I would expect that the PETSc-based > matrix is split up > > Now, for an input size of [8, 8] and a single thread I get > > > > > > > *Rank 0 has a total size of 8, 8 Rank 0 has an initial size of 64 Rank 0 > has a local matrix size of 8, 8 Rank 0 has a global matrix size of 8, 8 > Rank 0 has a matrix spanning from row 0 to 8 Rank 0 has a vector size of 64* > > which is what I expect. For two threads, though, I get > *Rank 0 has a total size of 8, 8* > *Rank 0 has an initial size of 64* > *Rank 0 has a local matrix size of 4, 4* > *Rank 0 has a global matrix size of 8, 8* > *Rank 0 has a matrix spanning from row 0 to 4* > *Rank 0 has a vector size of 32* > *Rank 1 has a total size of 8, 8* > *Rank 1 has an initial size of 64* > *Rank 1 has a local matrix size of 4, 4* > *Rank 1 has a global matrix size of 8, 8* > *Rank 1 has a matrix spanning from row 4 to 8* > * Rank 1 has a vector size of 32* > > Here, most entries make sense, except the size of the local matrices. Why > do I get a size of [4, 4], and not a size of [4, 8]? Each row should be > contiguous in the local process, and therefore each row should contain all > columns, not only a part of it. > > Is there a misunderstanding about how to use MatGetLocalSize(), or > something else? > > Thanks! > > Regards, > > Roland Richter > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 13 07:36:07 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jan 2021 08:36:07 -0500 Subject: [petsc-users] Understanding global and local rows in a distributed dense matrix In-Reply-To: References: Message-ID: On Wed, Jan 13, 2021 at 8:26 AM Stefano Zampini wrote: > MATMPIDENSE does not implement any cyclic distribution. In parallel, a > dense matrix is split by rows. Each process owns localrows*globalcols > entries. Local sizes are to be intended as the size of the right and left > vectors used in matvec operations, and are not strictly related with > storage considerations. > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSize.html > To expand on Stefano's point. MATDENSE is intended for very simple matrix handling, and the optimizations are most appropriate for blocks of vectors. Sophisticated desne matrix operations are better handled by the MATELEMENTAL type. What you give up here is the ability to directly manipulate the storage since it is now distributed in a crazy way. Thus most users actually want MATDENSE. Thanks, Matt > Il Mer 13 Gen 2021, 15:40 Roland Richter ha > scritto: > >> Hei, >> >> I am currently struggling a bit with my understanding of local and global >> rows/columns in distributed dense matrices in PETSc. >> >> The attached short program is creating a local matrix in every thread >> with [matrix_size_rows, matrix_size_cols] as global size. Afterwards, I >> flatten the matrix into a vector, and split this vector using the >> boost-function split() when running using several MPI threads. In addition >> I create a dense MPI-distributed matrix based on PETSc, using the same >> input sizes, and set all values to zero (after I'm not interested in the >> content itself, just the shape). Finally, I retrieve the global and local >> values for the number of rows and columns using MatGetLocalSize, and the >> number of the first and last row in each matrix using >> MatGetOwnershipRange() and print them. According to the documentation I >> would expect that the PETSc-based matrix is split up >> >> Now, for an input size of [8, 8] and a single thread I get >> >> >> >> >> >> >> *Rank 0 has a total size of 8, 8 Rank 0 has an initial size of 64 Rank 0 >> has a local matrix size of 8, 8 Rank 0 has a global matrix size of 8, 8 >> Rank 0 has a matrix spanning from row 0 to 8 Rank 0 has a vector size of 64* >> >> which is what I expect. For two threads, though, I get >> *Rank 0 has a total size of 8, 8* >> *Rank 0 has an initial size of 64* >> *Rank 0 has a local matrix size of 4, 4* >> *Rank 0 has a global matrix size of 8, 8* >> *Rank 0 has a matrix spanning from row 0 to 4* >> *Rank 0 has a vector size of 32* >> *Rank 1 has a total size of 8, 8* >> *Rank 1 has an initial size of 64* >> *Rank 1 has a local matrix size of 4, 4* >> *Rank 1 has a global matrix size of 8, 8* >> *Rank 1 has a matrix spanning from row 4 to 8* >> * Rank 1 has a vector size of 32* >> >> Here, most entries make sense, except the size of the local matrices. Why >> do I get a size of [4, 4], and not a size of [4, 8]? Each row should be >> contiguous in the local process, and therefore each row should contain all >> columns, not only a part of it. >> >> Is there a misunderstanding about how to use MatGetLocalSize(), or >> something else? >> >> Thanks! >> >> Regards, >> >> Roland Richter >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 13 08:00:17 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jan 2021 09:00:17 -0500 Subject: [petsc-users] DMPlex with dof in cells AND at nodes In-Reply-To: References: Message-ID: On Tue, Jan 12, 2021 at 4:12 PM Thibault Bridel-Bertomeu < thibault.bridelbertomeu at gmail.com> wrote: > Good evening Matthew, > > Thank you for your answer, and sorry for the delay in mine ! I am trying > to figure things out but when one gets in the distribution of a DMPlex, it > becomes quite complex! > > Le lun. 11 janv. 2021 ? 16:24, Matthew Knepley a > ?crit : > >> On Mon, Jan 11, 2021 at 6:41 AM Thibault Bridel-Bertomeu < >> thibault.bridelbertomeu at gmail.com> wrote: >> >>> Dear all, >>> >>> I haven't been on this mailing list since december, so first of all, >>> happy new year to everyone ! >>> >>> I am facing another challenge with the DMPlex structure : although I am >>> doing cell-centered finite volume (using PetscFV), I also need to do some >>> manipulations and store information that are vertex-based. >>> The problem I have is when I run the code in parallel : when I visualize >>> the nodal data using a vtk viewer, it shows NaN at all the vertices in >>> close proximity to the MPI partition line. >>> Here is how I proceed to work on the vertex-based data : >>> >>> First comes the creation of the DM that I'll be using for the >>> cell-centered FVM >>> >>> call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) >>> call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) >>> call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) >>> ... >>> dm = dmDist >>> ... >>> call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) >>> ... >>> dm = dmGhost >>> ... >>> >>> I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in MPI >>> the neighbouring relationships are conserved for all strata. Is that true ? >>> Like, does doing this actually enable to get all the transitive closure of >>> a given vertex, even if part of it is in another MPI block ? >>> >> >> What you say should be true, but it is not. When I designed everything, I >> was very much stuck in the traditional mesh way of looking at things, and >> made some limiting decisions. You can still get what you want (almost), but >> you need to distribute the mesh with an overlap of 1. If you do, then you >> can get the transitive support of vertices. >> > > I actually already distribute the mesh with an overlap of 1, but the > result is the same. Do I need to re-distribute the clone "dmNodal" somehow > ? Is the cloning process maybe not preserving the MPI support of the > different strata ? > No. However, it is easy to check what you have using -dm_view. I would encourage you to put DMViewFromOptions(dm, NULL, "-dm_view"); DMViewFromOptions(dmNodal, NULL, "-nodal_dm_view"); so that you can give those options and send the output (for a very simple mesh) with the emails. What you should see is that for both meshes, the number of local cells adds up to more than the number of global cells. > The problem is basically that many places in the code assume that the >> overlap always contains the closure of any point in it. It is difficult to >> be able to switch this assumption to the star (transitive support)so I have >> not done that yet. >> > >> >>> Then I need a vertex-based DM, so I figure it's pretty much what's done >>> in CreateMassMatrix in TS ex11.c. >>> >>> call DMClone(dm, dmNodal, ierr) >>> call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) >>> call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) >>> call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, >>> ierr); >>> call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); >>> CHKERRA(ierr) >>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) >>> call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); >>> CHKERRA(ierr) >>> do inode = vstart, vend-1 >>> ! 3 dof per node, ndim = 3 >>> call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) >>> end do >>> call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) >>> call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) >>> call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) >>> >>> From what I understood, this new dmNodal basically has all the >>> characteristics of the ori dm, except it's primary layer is vertex-based >>> with 3 DOF per vertex. >>> >> >> Take out the coordinate stuff above. This is now handled internally in a >> better way. Other than that, it looks good. >> > > You mean the SetCoordinateSection is already handled by the cloning > process ? > Yes. > >> >>> Then I get a local vector and work on it : >>> >>> call DMGetLocalVector(dmNodal, nodalvec, ierr) >>> call VecGetArrayF90(nodalvec, nodalvecarray, ierr) >>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) >>> call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) >>> ! For each node ... >>> do inode = vstart, vend-1 >>> A = 0 ! matrix, ndim x ndim >>> b = 0 ! vector, ndim >>> call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, >>> vertexclosure, ierr); >>> ! For each face attached to that node >>> do iface = 1, size(vertexclosure) >>> if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= fend)) >>> cycle >>> A = A + ... >>> b = b + ... >>> end do >>> nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, b) >>> ... >>> end do >>> >> >> There are some subtleties here I think. You are looping over all >> vertices, but you only own some of them. >> > > Here I am quite surprised by your remark : from the random print I did, > the DMPlexGetDepthStratum returns different vstart and vend for different > MPI processes, hence I assumed it was a local numbering. > It is a local numbering. If you have a mesh partition, it is usually to have vertices present in multiple local meshes. We think of one process as "owning" the vertex. That is what I meant here. > And then I assumed that, with the DMPlexDistribute, it ensured that the > "vertexclosure" > This is the wrong terminology. Closures mean including the boundary of something. Vertices have no boundary. You want the dual notion "support", or its transitive closure "star". So is the vertex star included when distributing? No, only the point closure is guaranteed when distributing. This was the point of my explanation. However, if you request overlap = 1, then you get everything in the star of the original boundary. > contained (among others) all the faces connected to each "inode", even if > the closure is spread among multiple MPI processes. So you mean that all > these assumptions are actually wrong to some degree ? > > >> What do you do when putting these values in the global vector? I think >> there are at least three choices: >> >> 1) Everyone has everything, so you compute all the vertex stuff >> redundantly, and then use INSERT_VALUES for the scatter >> >> 2) Only vertices are redundant, so the transitive supports of vertices >> are partial but non-overlapping, and you use ADD_VALUES for the scatter >> >> 3) Only cells are redundant, so you do the complete vertex computation >> only for local vertices >> > > I distributed with overlap = 1, so I expect I have a halo of redundant > cells in each MPI block (option no. 3). > Yes. > With the (..., PETSC_TRUE, PETSC_TRUE, ...) setting of the distribute > however, I also expected to have a halo of vertices and a halo of edges and > a halo of faces. Therefore, completing the vertex computation only for > local vertices should have been fine since anyways even local vertices know > their complete closure in terms of faces, > Yes, this should work. > but I am missing something because it does not turn out right ... > Start with a very simple mesh, say 2 cells or 4 cells. Then you can print everything out. Thanks, Matt > >> >>> Then I want to visualize it : >>> >>> call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) >>> call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) >>> call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, >>> ierr) >>> call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); CHKERRA(ierr) >>> call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) >>> call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) >>> call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) >>> call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) >>> >>> In the toto.vtk file, I get vertex-based values as expected, and the >>> whole shebang works in sequential. Now when I run it in parallel, I get >>> vertex-based values inside the MPI blocks, but in the neighbourhood of the >>> join between the MPI blocks, all the vertex-based values are NaN (see >>> attachment). >>> >> >> If you have redundant cells, meaning you distributed with overlap = 1, >> did you remember to full up the values for those ghost cells using >> DMGlobalToLocal() or something like that? If not, they will start as NaNs. >> > > Indeed, in the vertex computation, at some point I use the values in the > cells. But I got the solution vector from TSGetSolution, then I do a > GlobalToLocal, then I even add a InsertBoundaryValues to fill the ghost > cells correctly. > Even by doing this, when I print the nodalvecarray on each process, they > somehow contain some NaNs. This means, I think, that at some point, a given > vertex tries to get the information of a face that is supposed to be in its > closure but is actually not .. (?) > > Thank you for your help and feedback ! > > Cheers, > > Thibault > > >> >> Thanks, >> >> Matt >> >> >>> All in all, I think I do not understand fully the consequences of the >>> DMPlexDistribute with useCone=true & useClosure=true ... but I cannot >>> figure out where the NaN are coming from : is it because the vertices do >>> not actually know all their closure through the MPI join, or is it some >>> Global / Local error I made ? >>> >>> Thank you very much in advance for your support !! >>> >>> Thibault >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From griesser.jan at googlemail.com Wed Jan 13 10:28:56 2021 From: griesser.jan at googlemail.com (=?utf-8?Q?Jan_Grie=C3=9Fer?=) Date: Wed, 13 Jan 2021 17:28:56 +0100 Subject: [petsc-users] Running PETSc/SLEPc tests inside/outside a singularity container Message-ID: <83698D4D-E953-4205-8071-3675A6F43820@googlemail.com> Hello all, I have a question about the tests for PETSc/PETSc4Py and SLEPc4Py when these libraries are compiled inside a Singularity container (The .def file for compilation is attached). The compilation of the libraries is no problem, only if you call "make all check" for PETSc and "make check" for SLEPC at the end of the compilation, error messages appear for PETSc or the Singularity build aborts for SLEPc (see output.txt). I guess the problems are only related to the "sudo" rights, which are needed to build the containers. With slepc I don't even have access to the makelog because the container can't be built at all. For the 4Py versions I could navigate via singularity shell into the corresponding directories and execute the runtests. These returned an ok for all tests. Is there a clever way to start tests for the c-Versions of PETSc/SLEPc afterwards to check for the correct build of the Libaries? Using the singularity shell to navigate into petsc and using gmakefile test throws an ?Read-only file system? error, as expected. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output.txt URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_slepc.def Type: application/octet-stream Size: 3324 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Jan 13 11:17:04 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 13 Jan 2021 11:17:04 -0600 Subject: [petsc-users] Running PETSc/SLEPc tests inside/outside a singularity container In-Reply-To: <83698D4D-E953-4205-8071-3675A6F43820@googlemail.com> References: <83698D4D-E953-4205-8071-3675A6F43820@googlemail.com> Message-ID: On Wed, 13 Jan 2021, Jan Grie?er via petsc-users wrote: > Hello all, > I have a question about the tests for PETSc/PETSc4Py and SLEPc4Py when these libraries are compiled inside a Singularity container (The .def file for compilation is attached). The compilation of the libraries is no problem, only if you call "make all check" for PETSc and "make check" for SLEPC at the end of the compilation, error messages appear for PETSc or the Singularity build aborts for SLEPc (see output.txt). I guess the problems are only related to the "sudo" rights, which are needed to build the containers. ooks like the container build [and the test] is done as root. and OpenMPI suggests: >>> You can override this protection by adding the --allow-run-as-root option to the cmd line or by setting two environment variables in the following way: the variable OMPI_ALLOW_RUN_AS_ROOT=1 to indicate the desire to override this protection, and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 to confirm the choice and add one more layer of certainty that you want to do so. We reiterate our advice against doing so - please proceed at your own risk. <<<< So perhaps there is a way to do this whole process as a regular user [and not root?] - or set the above 2 env variables? > With slepc I don't even have access to the makelog because the container can't be built at all. I'm not sure what you mean here - and why the make.log would exist in a regular build - and not in a container build. > For the 4Py versions I could navigate via singularity shell into the corresponding directories and execute the runtests. These returned an ok for all tests. > Is there a clever way to start tests for the c-Versions of PETSc/SLEPc afterwards to check for the correct build of the Libaries? Using the singularity shell to navigate into petsc and using gmakefile test throws an ?Read-only file system? error, as expected. > Hm - you should be able to copy a petsc example, corresponding makefile to a read/write location, and compile/run it. Satish From jroman at dsic.upv.es Wed Jan 13 11:29:59 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 13 Jan 2021 18:29:59 +0100 Subject: [petsc-users] Running PETSc/SLEPc tests inside/outside a singularity container In-Reply-To: References: <83698D4D-E953-4205-8071-3675A6F43820@googlemail.com> Message-ID: <400F6138-CD7D-45B2-9792-D19CACFE904F@dsic.upv.es> > El 13 ene 2021, a las 18:17, Satish Balay via petsc-users escribi?: > > On Wed, 13 Jan 2021, Jan Grie?er via petsc-users wrote: > >> Hello all, >> I have a question about the tests for PETSc/PETSc4Py and SLEPc4Py when these libraries are compiled inside a Singularity container (The .def file for compilation is attached). The compilation of the libraries is no problem, only if you call "make all check" for PETSc and "make check" for SLEPC at the end of the compilation, error messages appear for PETSc or the Singularity build aborts for SLEPc (see output.txt). I guess the problems are only related to the "sudo" rights, which are needed to build the containers. > > ooks like the container build [and the test] is done as root. > > and OpenMPI suggests: > >>>> > You can override this protection by adding the --allow-run-as-root option > to the cmd line or by setting two environment variables in the following way: > the variable OMPI_ALLOW_RUN_AS_ROOT=1 to indicate the desire to override this > protection, and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 to confirm the choice and > add one more layer of certainty that you want to do so. > We reiterate our advice against doing so - please proceed at your own risk. > <<<< > > So perhaps there is a way to do this whole process as a regular user [and not root?] - or set the above 2 env variables? > >> With slepc I don't even have access to the makelog because the container can't be built at all. > > I'm not sure what you mean here - and why the make.log would exist in a regular build - and not in a container build. I recently changed SLEPc to return a nonzero error code if 'make check' fails. So this is now different from PETSc. I guess that is why the container building is not completed. Jose > >> For the 4Py versions I could navigate via singularity shell into the corresponding directories and execute the runtests. These returned an ok for all tests. >> Is there a clever way to start tests for the c-Versions of PETSc/SLEPc afterwards to check for the correct build of the Libaries? Using the singularity shell to navigate into petsc and using gmakefile test throws an ?Read-only file system? error, as expected. >> > > Hm - you should be able to copy a petsc example, corresponding makefile to a read/write location, and compile/run it. > > Satish From thibault.bridelbertomeu at gmail.com Wed Jan 13 11:36:32 2021 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Wed, 13 Jan 2021 18:36:32 +0100 Subject: [petsc-users] DMPlex with dof in cells AND at nodes In-Reply-To: References: Message-ID: Hello Matt, Thank you for the follow up. Here are the print out for the two DMs as you suggested (the smallest tetrahedral mesh possible in a cube). :: [DEBUG] Visualizing DM in console :: DM Object: 2 MPI processes type: plex DM_0x7faa62d416f0_1 in 3 dimensions: 0-cells: 14 14 1-cells: 49 49 2-cells: 60 60 3-cells: 36 (12) [12] 36 (12) [12] Labels: depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (36)) celltype: 5 strata with value/size (0 (14), 1 (49), 3 (60), 6 (24), 12 (12)) Cell Sets: 1 strata with value/size (7 (24)) Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), 6 (4)) vtk: 1 strata with value/size (1 (12)) ghost: 2 strata with value/size (2 (12), 1 (27)) :: [DEBUG] Visualizing DM_nodal in console :: DM Object: 2 MPI processes type: plex DM_0x7faa62d416f0_3 in 3 dimensions: 0-cells: 14 14 1-cells: 49 49 2-cells: 60 60 3-cells: 36 (12) [12] 36 (12) [12] Labels: depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (36)) celltype: 5 strata with value/size (0 (14), 1 (49), 3 (60), 6 (24), 12 (12)) Cell Sets: 1 strata with value/size (7 (24)) Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), 6 (4)) vtk: 1 strata with value/size (1 (12)) ghost: 2 strata with value/size (2 (12), 1 (27)) As you predicted, the sum of the numbers of local cells is greater than the total number of cells. I was doing some more digging, and it appears that I have issues because the normals of some faces attached to the vertices are not populated. In other words, "call DMPlexGetGeometryFVM(dm, facegeom, cellgeom, minradius, ierr); CHKERRA(ierr)" yields a facegeom Vec that is a local Vec. For the mesh that are described above, that Vec has a global size of 720 (12 * 60 faces), and also a local size of 720 (12 * 60) because (I think .. ?) the mesh is so small, with the distribution chosen, both processes know the existence of all the entities (vertices, edges, faces & cells). However, for, say MPI proc no.1, a lot of places in this Vec are filled with zeros. As I use the normal normalized by the face area, I get NaNs because I am trying to divide by zero. Now the question is, a vertex on a given MPI proc knows that it has, say, 6 faces attached, and I can get that number along with the indices of those faces with "call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, vertexclosure, ierr); CHKERRA(ierr)", but then when I fetch the face characteristics in the facegeom Vec, some are not populated (full of zeros). Is this the consequence of what you were explaining, "only the point closure is guaranteed when distributing" ? Or is there a way to do some equivalent to GlobalToLocal for that facegeom Vec ? Thank you very much for your support, Thibault Le mer. 13 janv. 2021 ? 15:00, Matthew Knepley a ?crit : > On Tue, Jan 12, 2021 at 4:12 PM Thibault Bridel-Bertomeu < > thibault.bridelbertomeu at gmail.com> wrote: > >> Good evening Matthew, >> >> Thank you for your answer, and sorry for the delay in mine ! I am trying >> to figure things out but when one gets in the distribution of a DMPlex, it >> becomes quite complex! >> >> Le lun. 11 janv. 2021 ? 16:24, Matthew Knepley a >> ?crit : >> >>> On Mon, Jan 11, 2021 at 6:41 AM Thibault Bridel-Bertomeu < >>> thibault.bridelbertomeu at gmail.com> wrote: >>> >>>> Dear all, >>>> >>>> I haven't been on this mailing list since december, so first of all, >>>> happy new year to everyone ! >>>> >>>> I am facing another challenge with the DMPlex structure : although I am >>>> doing cell-centered finite volume (using PetscFV), I also need to do some >>>> manipulations and store information that are vertex-based. >>>> The problem I have is when I run the code in parallel : when I >>>> visualize the nodal data using a vtk viewer, it shows NaN at all the >>>> vertices in close proximity to the MPI partition line. >>>> Here is how I proceed to work on the vertex-based data : >>>> >>>> First comes the creation of the DM that I'll be using for the >>>> cell-centered FVM >>>> >>>> call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) >>>> call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) >>>> call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) >>>> ... >>>> dm = dmDist >>>> ... >>>> call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) >>>> ... >>>> dm = dmGhost >>>> ... >>>> >>>> I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in >>>> MPI the neighbouring relationships are conserved for all strata. Is that >>>> true ? Like, does doing this actually enable to get all the transitive >>>> closure of a given vertex, even if part of it is in another MPI block ? >>>> >>> >>> What you say should be true, but it is not. When I designed everything, >>> I was very much stuck in the traditional mesh way of looking at things, and >>> made some limiting decisions. You can still get what you want (almost), but >>> you need to distribute the mesh with an overlap of 1. If you do, then you >>> can get the transitive support of vertices. >>> >> >> I actually already distribute the mesh with an overlap of 1, but the >> result is the same. Do I need to re-distribute the clone "dmNodal" somehow >> ? Is the cloning process maybe not preserving the MPI support of the >> different strata ? >> > > No. However, it is easy to check what you have using -dm_view. I would > encourage you to put > > DMViewFromOptions(dm, NULL, "-dm_view"); > DMViewFromOptions(dmNodal, NULL, "-nodal_dm_view"); > > so that you can give those options and send the output (for a very simple > mesh) with the emails. What you should > see is that for both meshes, the number of local cells adds up to more > than the number of global cells. > > >> The problem is basically that many places in the code assume that the >>> overlap always contains the closure of any point in it. It is difficult to >>> be able to switch this assumption to the star (transitive support)so I have >>> not done that yet. >>> >> >>> >>>> Then I need a vertex-based DM, so I figure it's pretty much what's done >>>> in CreateMassMatrix in TS ex11.c. >>>> >>>> call DMClone(dm, dmNodal, ierr) >>>> call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) >>>> call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) >>>> call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, >>>> ierr); >>>> call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); >>>> CHKERRA(ierr) >>>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) >>>> call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); >>>> CHKERRA(ierr) >>>> do inode = vstart, vend-1 >>>> ! 3 dof per node, ndim = 3 >>>> call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) >>>> end do >>>> call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) >>>> call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) >>>> call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) >>>> >>>> From what I understood, this new dmNodal basically has all the >>>> characteristics of the ori dm, except it's primary layer is vertex-based >>>> with 3 DOF per vertex. >>>> >>> >>> Take out the coordinate stuff above. This is now handled internally in a >>> better way. Other than that, it looks good. >>> >> >> You mean the SetCoordinateSection is already handled by the cloning >> process ? >> > > Yes. > > >> >>> >>>> Then I get a local vector and work on it : >>>> >>>> call DMGetLocalVector(dmNodal, nodalvec, ierr) >>>> call VecGetArrayF90(nodalvec, nodalvecarray, ierr) >>>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) >>>> call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) >>>> ! For each node ... >>>> do inode = vstart, vend-1 >>>> A = 0 ! matrix, ndim x ndim >>>> b = 0 ! vector, ndim >>>> call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, >>>> vertexclosure, ierr); >>>> ! For each face attached to that node >>>> do iface = 1, size(vertexclosure) >>>> if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= >>>> fend)) cycle >>>> A = A + ... >>>> b = b + ... >>>> end do >>>> nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, >>>> b) >>>> ... >>>> end do >>>> >>> >>> There are some subtleties here I think. You are looping over all >>> vertices, but you only own some of them. >>> >> >> Here I am quite surprised by your remark : from the random print I did, >> the DMPlexGetDepthStratum returns different vstart and vend for different >> MPI processes, hence I assumed it was a local numbering. >> > > It is a local numbering. If you have a mesh partition, it is usually to > have vertices present in multiple local meshes. We think of one process as > "owning" the vertex. That is what I meant here. > > >> And then I assumed that, with the DMPlexDistribute, it ensured that the >> "vertexclosure" >> > > This is the wrong terminology. Closures mean including the boundary of > something. Vertices have no boundary. You want the dual notion "support", > or its transitive closure "star". So is the vertex star included when > distributing? No, only the point closure is guaranteed when distributing. > This was the point of my explanation. However, if you request overlap = 1, > then you get everything in the star of the original boundary. > > >> contained (among others) all the faces connected to each "inode", even if >> the closure is spread among multiple MPI processes. So you mean that all >> these assumptions are actually wrong to some degree ? >> >> >>> What do you do when putting these values in the global vector? I think >>> there are at least three choices: >>> >>> 1) Everyone has everything, so you compute all the vertex stuff >>> redundantly, and then use INSERT_VALUES for the scatter >>> >>> 2) Only vertices are redundant, so the transitive supports of vertices >>> are partial but non-overlapping, and you use ADD_VALUES for the scatter >>> >>> 3) Only cells are redundant, so you do the complete vertex computation >>> only for local vertices >>> >> >> I distributed with overlap = 1, so I expect I have a halo of redundant >> cells in each MPI block (option no. 3). >> > > Yes. > > >> With the (..., PETSC_TRUE, PETSC_TRUE, ...) setting of the distribute >> however, I also expected to have a halo of vertices and a halo of edges and >> a halo of faces. Therefore, completing the vertex computation only for >> local vertices should have been fine since anyways even local vertices know >> their complete closure in terms of faces, >> > > Yes, this should work. > > >> but I am missing something because it does not turn out right ... >> > > Start with a very simple mesh, say 2 cells or 4 cells. Then you can print > everything out. > > Thanks, > > Matt > > >> >>> >>>> Then I want to visualize it : >>>> >>>> call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) >>>> call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) >>>> call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, >>>> ierr) >>>> call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); CHKERRA(ierr) >>>> call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) >>>> call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) >>>> call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) >>>> call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) >>>> >>>> In the toto.vtk file, I get vertex-based values as expected, and the >>>> whole shebang works in sequential. Now when I run it in parallel, I get >>>> vertex-based values inside the MPI blocks, but in the neighbourhood of the >>>> join between the MPI blocks, all the vertex-based values are NaN (see >>>> attachment). >>>> >>> >>> If you have redundant cells, meaning you distributed with overlap = 1, >>> did you remember to full up the values for those ghost cells using >>> DMGlobalToLocal() or something like that? If not, they will start as >>> NaNs. >>> >> >> Indeed, in the vertex computation, at some point I use the values in the >> cells. But I got the solution vector from TSGetSolution, then I do a >> GlobalToLocal, then I even add a InsertBoundaryValues to fill the ghost >> cells correctly. >> Even by doing this, when I print the nodalvecarray on each process, they >> somehow contain some NaNs. This means, I think, that at some point, a given >> vertex tries to get the information of a face that is supposed to be in its >> closure but is actually not .. (?) >> >> Thank you for your help and feedback ! >> >> Cheers, >> >> Thibault >> >> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> All in all, I think I do not understand fully the consequences of the >>>> DMPlexDistribute with useCone=true & useClosure=true ... but I cannot >>>> figure out where the NaN are coming from : is it because the vertices do >>>> not actually know all their closure through the MPI join, or is it some >>>> Global / Local error I made ? >>>> >>>> Thank you very much in advance for your support !! >>>> >>>> Thibault >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Jan 13 12:04:58 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 13 Jan 2021 11:04:58 -0700 Subject: [petsc-users] counter->tag = *maxval - 128 In-Reply-To: <7AE622D5-1B5B-4DBA-A791-688E925DE4BE@petsc.dev> References: <7AE622D5-1B5B-4DBA-A791-688E925DE4BE@petsc.dev> Message-ID: On Tue, Jan 12, 2021 at 6:49 PM Barry Smith wrote: > > Fande, > > /* hope that any still active tags were issued right at the beginning > of the run */ > > PETSc actually starts with *maxval (see line 130). It is only when it > runs out that it does this silly thing for the reason indicated in the > comment. > > PETSc should actually keep track which of which tags have been > "returned" and if the counter gets to zero use those returned tags instead > of starting again at the top which could clash with the same value used for > another reason. In other words the current code is buggy, but it has > always been "good enough". > I agreed that it is "good enough" for most people for most cases. However, I was worried that there is almost no way to debug once we reuse active tags. At least it is not easy to debug. Thanks, Fande > > Barry > > > > On Jan 12, 2021, at 10:41 AM, Fande Kong wrote: > > Hi All, > > I am curious about why we subtract 128 from the max value of tag? Can we > directly use the max tag value? > > Thanks, > > Fande, > > > PetscErrorCode PetscCommGetNewTag(MPI_Comm comm,PetscMPIInt *tag) > { > PetscErrorCode ierr; > PetscCommCounter *counter; > PetscMPIInt *maxval,flg; > > > MPI_Comm_get_attr(comm,Petsc_Counter_keyval,&counter,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI > communicator supplied; must be a PETSc communicator"); > > if (counter->tag < 1) { > PetscInfo1(NULL,"Out of tags for object, starting to recycle. Comm > reference count %d\n",counter->refcount); > MPI_Comm_get_attr(MPI_COMM_WORLD,MPI_TAG_UB,&maxval,&flg); > if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"MPI error: > MPI_Comm_get_attr() is not returning a MPI_TAG_UB"); > counter->tag = *maxval - 128; /* hope that any still active tags were > issued right at the beginning of the run */ > } > > *tag = counter->tag--; > if (PetscDefined(USE_DEBUG)) { > /* > Hanging here means that some processes have called > PetscCommGetNewTag() and others have not. > */ > MPI_Barrier(comm); > } > return(0); > } > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Jan 13 12:32:43 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 13 Jan 2021 11:32:43 -0700 Subject: [petsc-users] PetscAllreduceBarrierCheck is valgrind clean? Message-ID: Hi All, I ran valgrind with mvapich-2.3.5 for a moose simulation. The motivation was that we have a few non-deterministic parallel simulations in moose. I want to check if we have any memory issues. I got some complaints from PetscAllreduceBarrierCheck Thanks, Fande ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 31 of 54 ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) ==98001== by 0x99F70BE: PetscSplitOwnership (psplit.c:84) ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) ==98001== by 0xA08C66B: MatMPIAdjSetPreallocation_MPIAdj (mpiadj.c:630) ==98001== by 0xA08EB9A: MatMPIAdjSetPreallocation (mpiadj.c:856) ==98001== by 0xA08F6D3: MatCreateMPIAdj (mpiadj.c:904) ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 32 of 54 ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) ==98001== by 0xD59A9A4: mv2_increment_allgather_coll_counter (ch3_shmem_coll.c:2218) ==98001== by 0xD4E4CE4: PMPI_Allreduce (allreduce.c:917) ==98001== by 0xCD9D74D: libparmetis__gkMPI_Allreduce (gkmpi.c:103) ==98001== by 0xCDBB663: libparmetis__ComputeParallelBalance (stat.c:87) ==98001== by 0xCDA4FE0: libparmetis__KWayFM (kwayrefine.c:352) ==98001== by 0xCDA21ED: libparmetis__Global_Partition (kmetis.c:222) ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) ==98001== by 0xCDA2748: ParMETIS_V3_PartKway (kmetis.c:94) ==98001== by 0xA2D6B39: MatPartitioningApply_Parmetis_Private (pmetis.c:145) ==98001== by 0xA2D77D9: MatPartitioningApply_Parmetis (pmetis.c:219) ==98001== by 0xA2CD46A: MatPartitioningApply (partition.c:332) ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 33 of 54 ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) ==98001== by 0x99F733E: PetscSplitOwnership (psplit.c:91) ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) ==98001== by 0x9C5DB0D: PetscLayoutCreateFromSizes (pmap.c:112) ==98001== by 0x9D9A018: ISGeneralSetIndices_General (general.c:568) ==98001== by 0x9D9AB44: ISGeneralSetIndices (general.c:554) ==98001== by 0x9D9ADC4: ISCreateGeneral (general.c:529) ==98001== by 0x9B431E6: VecCreateGhostWithArray (pbvec.c:692) ==98001== by 0x9B43A33: VecCreateGhost (pbvec.c:748) ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 34 of 54 ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) ==98001== by 0x9B0B5F3: VecSetSizes (vector.c:1318) ==98001== by 0x9B42DDC: VecCreateMPIWithArray (pbvec.c:625) ==98001== by 0xA7CF280: PCSetUp_Redundant (redundant.c:125) ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) ==98001== by 0xA92C490: PCSetUp_MG (mg.c:907) ==98001== by 0xA93CAE9: PCSetUp_HMG (hmg.c:220) ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) ==98001== by 0xABD95AA: SNESSolve (snes.c:4569) ==98001== 176 (48 direct, 128 indirect) bytes in 2 blocks are definitely lost in loss record 39 of 54 ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) ==98001== by 0xBAB3848: hypre_MPI_Allreduce (mpistubs.c:1180) ==98001== by 0xB96093E: hypre_ParCSRMatrixSetNumNonzeros_core (par_csr_matrix.c:383) ==98001== by 0xB960A11: hypre_ParCSRMatrixSetDNumNonzeros (par_csr_matrix.c:413) ==98001== by 0xB8B32DF: hypre_BoomerAMGSetup (par_amg_setup.c:2784) ==98001== by 0xB8A0D26: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:47) ==98001== by 0xA9EDB17: PCSetUp_HYPRE (hypre.c:408) ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) ==98001== by 0xA93B2DB: PCSetUp_HMG (hmg.c:161) ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jan 13 12:34:30 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 13 Jan 2021 12:34:30 -0600 Subject: [petsc-users] counter->tag = *maxval - 128 In-Reply-To: References: <7AE622D5-1B5B-4DBA-A791-688E925DE4BE@petsc.dev> Message-ID: <369385E4-C852-423E-82F3-07692FC2AC5B@petsc.dev> Yes, definitely not easy to debug. If you have concerns in your code about running out and then producing bugs you can fix it by just keeping an array of returned tags and use these tags when needed. You'll also have to add something like PetscObject/CommRestoreTag() to return ones no longer needed. Barry > On Jan 13, 2021, at 12:04 PM, Fande Kong wrote: > > > > On Tue, Jan 12, 2021 at 6:49 PM Barry Smith > wrote: > > Fande, > > /* hope that any still active tags were issued right at the beginning of the run */ > > PETSc actually starts with *maxval (see line 130). It is only when it runs out that it does this silly thing for the reason indicated in the comment. > > PETSc should actually keep track which of which tags have been "returned" and if the counter gets to zero use those returned tags instead of starting again at the top which could clash with the same value used for another reason. In other words the current code is buggy, but it has always been "good enough". > > I agreed that it is "good enough" for most people for most cases. However, I was worried that there is almost no way to debug once we reuse active tags. At least it is not easy to debug. > > Thanks, > > Fande > > > Barry > > > >> On Jan 12, 2021, at 10:41 AM, Fande Kong > wrote: >> >> Hi All, >> >> I am curious about why we subtract 128 from the max value of tag? Can we directly use the max tag value? >> >> Thanks, >> >> Fande, >> >> >> PetscErrorCode PetscCommGetNewTag(MPI_Comm comm,PetscMPIInt *tag) >> { >> PetscErrorCode ierr; >> PetscCommCounter *counter; >> PetscMPIInt *maxval,flg; >> >> >> MPI_Comm_get_attr(comm,Petsc_Counter_keyval,&counter,&flg); >> if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI communicator supplied; must be a PETSc communicator"); >> >> if (counter->tag < 1) { >> PetscInfo1(NULL,"Out of tags for object, starting to recycle. Comm reference count %d\n",counter->refcount); >> MPI_Comm_get_attr(MPI_COMM_WORLD,MPI_TAG_UB,&maxval,&flg); >> if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"MPI error: MPI_Comm_get_attr() is not returning a MPI_TAG_UB"); >> counter->tag = *maxval - 128; /* hope that any still active tags were issued right at the beginning of the run */ >> } >> >> *tag = counter->tag--; >> if (PetscDefined(USE_DEBUG)) { >> /* >> Hanging here means that some processes have called PetscCommGetNewTag() and others have not. >> */ >> MPI_Barrier(comm); >> } >> return(0); >> } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jan 13 12:49:47 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 13 Jan 2021 12:49:47 -0600 Subject: [petsc-users] PetscAllreduceBarrierCheck is valgrind clean? In-Reply-To: References: Message-ID: Fande, Look at https://scm.mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/trunk/src/mpid/ch3/channels/common/src/detect/arch/mv2_arch_detect.c cpubind_set = hwloc_bitmap_alloc(); but I don't find a corresponding hwloc_bitmap_free(cpubind_set ); in get_socket_bound_info(). Barry > > On Jan 13, 2021, at 12:32 PM, Fande Kong wrote: > > Hi All, > > I ran valgrind with mvapich-2.3.5 for a moose simulation. The motivation was that we have a few non-deterministic parallel simulations in moose. I want to check if we have any memory issues. I got some complaints from PetscAllreduceBarrierCheck > > Thanks, > > > Fande > > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 31 of 54 > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) > ==98001== by 0x99F70BE: PetscSplitOwnership (psplit.c:84) > ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) > ==98001== by 0xA08C66B: MatMPIAdjSetPreallocation_MPIAdj (mpiadj.c:630) > ==98001== by 0xA08EB9A: MatMPIAdjSetPreallocation (mpiadj.c:856) > ==98001== by 0xA08F6D3: MatCreateMPIAdj (mpiadj.c:904) > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 32 of 54 > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > ==98001== by 0xD59A9A4: mv2_increment_allgather_coll_counter (ch3_shmem_coll.c:2218) > ==98001== by 0xD4E4CE4: PMPI_Allreduce (allreduce.c:917) > ==98001== by 0xCD9D74D: libparmetis__gkMPI_Allreduce (gkmpi.c:103) > ==98001== by 0xCDBB663: libparmetis__ComputeParallelBalance (stat.c:87) > ==98001== by 0xCDA4FE0: libparmetis__KWayFM (kwayrefine.c:352) > ==98001== by 0xCDA21ED: libparmetis__Global_Partition (kmetis.c:222) > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > ==98001== by 0xCDA2748: ParMETIS_V3_PartKway (kmetis.c:94) > ==98001== by 0xA2D6B39: MatPartitioningApply_Parmetis_Private (pmetis.c:145) > ==98001== by 0xA2D77D9: MatPartitioningApply_Parmetis (pmetis.c:219) > ==98001== by 0xA2CD46A: MatPartitioningApply (partition.c:332) > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 33 of 54 > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) > ==98001== by 0x99F733E: PetscSplitOwnership (psplit.c:91) > ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) > ==98001== by 0x9C5DB0D: PetscLayoutCreateFromSizes (pmap.c:112) > ==98001== by 0x9D9A018: ISGeneralSetIndices_General (general.c:568) > ==98001== by 0x9D9AB44: ISGeneralSetIndices (general.c:554) > ==98001== by 0x9D9ADC4: ISCreateGeneral (general.c:529) > ==98001== by 0x9B431E6: VecCreateGhostWithArray (pbvec.c:692) > ==98001== by 0x9B43A33: VecCreateGhost (pbvec.c:748) > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely lost in loss record 34 of 54 > =98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > ==98001== by 0xDA7523F: get_socket_ > = > bound_info (mv2_arch_detect.c:898) > ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > ==98001== by 0x9B0B5F3: VecSetSizes (vector.c:1318) > ==98001== by 0x9B42DDC: VecCreateMPIWithArray (pbvec.c:625) > ==98001== by 0xA7CF280: PCSetUp_Redundant (redundant.c:125) > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > ==98001== by 0xA92C490: PCSetUp_MG (mg.c:907) > ==98001== by 0xA93CAE9: PCSetUp_HMG (hmg.c:220) > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) > ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) > ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) > ==98001== by 0xABD95AA: SNESSolve (snes.c:4569) > > > ==98001== 176 (48 direct, 128 indirect) bytes in 2 blocks are definitely lost in loss record 39 of 54 > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > ==98001== by 0xD93C87A: create_intra_sock_comm (create_2level_comm.c:593) > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter (ch3_shmem_coll.c:2183) > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > ==98001== by 0xBAB3848: hypre_MPI_Allreduce (mpistubs.c:1180) > ==98001== by 0xB96093E: hypre_ParCSRMatrixSetNumNonzeros_core (par_csr_matrix.c:383) > ==98001== by 0xB960A11: hypre_ParCSRMatrixSetDNumNonzeros (par_csr_matrix.c:413) > ==98001== by 0xB8B32DF: hypre_BoomerAMGSetup (par_amg_setup.c:2784) > ==98001== by 0xB8A0D26: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:47) > ==98001== by 0xA9EDB17: PCSetUp_HYPRE (hypre.c:408) > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > ==98001== by 0xA93B2DB: PCSetUp_HMG (hmg.c:161) > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) > ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) > ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) > > > > From fdkong.jd at gmail.com Wed Jan 13 14:18:05 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 13 Jan 2021 13:18:05 -0700 Subject: [petsc-users] PetscAllreduceBarrierCheck is valgrind clean? In-Reply-To: References: Message-ID: On Wed, Jan 13, 2021 at 11:49 AM Barry Smith wrote: > > Fande, > > Look at > https://scm.mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/trunk/src/mpid/ch3/channels/common/src/detect/arch/mv2_arch_detect.c > > cpubind_set = hwloc_bitmap_alloc(); > > but I don't find a corresponding hwloc_bitmap_free(cpubind_set ); in > get_socket_bound_info(). > Thanks. I added hwloc_bitmap_free(cpubind_set ) to the end of get_socket_bound_info(). And then these valgrind messages disappeared. Will ask mvapich developers to fix this. Thanks, Fande, > > > Barry > > > > > > > On Jan 13, 2021, at 12:32 PM, Fande Kong wrote: > > > > Hi All, > > > > I ran valgrind with mvapich-2.3.5 for a moose simulation. The > motivation was that we have a few non-deterministic parallel simulations in > moose. I want to check if we have any memory issues. I got some complaints > from PetscAllreduceBarrierCheck > > > > Thanks, > > > > > > Fande > > > > > > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely > lost in loss record 31 of 54 > > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > > ==98001== by 0xD93C87A: create_intra_sock_comm > (create_2level_comm.c:593) > > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter > (ch3_shmem_coll.c:2183) > > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > > ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) > > ==98001== by 0x99F70BE: PetscSplitOwnership (psplit.c:84) > > ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) > > ==98001== by 0xA08C66B: MatMPIAdjSetPreallocation_MPIAdj > (mpiadj.c:630) > > ==98001== by 0xA08EB9A: MatMPIAdjSetPreallocation (mpiadj.c:856) > > ==98001== by 0xA08F6D3: MatCreateMPIAdj (mpiadj.c:904) > > > > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely > lost in loss record 32 of 54 > > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > > ==98001== by 0xD93C87A: create_intra_sock_comm > (create_2level_comm.c:593) > > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > > ==98001== by 0xD59A9A4: mv2_increment_allgather_coll_counter > (ch3_shmem_coll.c:2218) > > ==98001== by 0xD4E4CE4: PMPI_Allreduce (allreduce.c:917) > > ==98001== by 0xCD9D74D: libparmetis__gkMPI_Allreduce (gkmpi.c:103) > > ==98001== by 0xCDBB663: libparmetis__ComputeParallelBalance > (stat.c:87) > > ==98001== by 0xCDA4FE0: libparmetis__KWayFM (kwayrefine.c:352) > > ==98001== by 0xCDA21ED: libparmetis__Global_Partition (kmetis.c:222) > > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > > ==98001== by 0xCDA20B2: libparmetis__Global_Partition (kmetis.c:191) > > ==98001== by 0xCDA2748: ParMETIS_V3_PartKway (kmetis.c:94) > > ==98001== by 0xA2D6B39: MatPartitioningApply_Parmetis_Private > (pmetis.c:145) > > ==98001== by 0xA2D77D9: MatPartitioningApply_Parmetis (pmetis.c:219) > > ==98001== by 0xA2CD46A: MatPartitioningApply (partition.c:332) > > > > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely > lost in loss record 33 of 54 > > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > > ==98001== by 0xD93C87A: create_intra_sock_comm > (create_2level_comm.c:593) > > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter > (ch3_shmem_coll.c:2183) > > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > > ==98001== by 0x99F1766: PetscAllreduceBarrierCheck (pbarrier.c:26) > > ==98001== by 0x99F733E: PetscSplitOwnership (psplit.c:91) > > ==98001== by 0x9C5C26B: PetscLayoutSetUp (pmap.c:262) > > ==98001== by 0x9C5DB0D: PetscLayoutCreateFromSizes (pmap.c:112) > > ==98001== by 0x9D9A018: ISGeneralSetIndices_General (general.c:568) > > ==98001== by 0x9D9AB44: ISGeneralSetIndices (general.c:554) > > ==98001== by 0x9D9ADC4: ISCreateGeneral (general.c:529) > > ==98001== by 0x9B431E6: VecCreateGhostWithArray (pbvec.c:692) > > ==98001== by 0x9B43A33: VecCreateGhost (pbvec.c:748) > > > > > > ==98001== 88 (24 direct, 64 indirect) bytes in 1 blocks are definitely > lost in loss record 34 of 54 > > =98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > > ==98001== by 0xDA7523F: get_socket_ > > = > > bound_info (mv2_arch_detect.c:898) > > ==98001== by 0xD93C87A: create_intra_sock_comm > (create_2level_comm.c:593) > > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter > (ch3_shmem_coll.c:2183) > > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > > ==98001== by 0x9B0B5F3: VecSetSizes (vector.c:1318) > > ==98001== by 0x9B42DDC: VecCreateMPIWithArray (pbvec.c:625) > > ==98001== by 0xA7CF280: PCSetUp_Redundant (redundant.c:125) > > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > > ==98001== by 0xA92C490: PCSetUp_MG (mg.c:907) > > ==98001== by 0xA93CAE9: PCSetUp_HMG (hmg.c:220) > > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > > ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) > > ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) > > ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) > > ==98001== by 0xABD95AA: SNESSolve (snes.c:4569) > > > > > > ==98001== 176 (48 direct, 128 indirect) bytes in 2 blocks are definitely > lost in loss record 39 of 54 > > ==98001== at 0x4C29F73: malloc (vg_replace_malloc.c:307) > > ==98001== by 0xDAE1D5E: hwloc_bitmap_alloc (bitmap.c:74) > > ==98001== by 0xDA7523F: get_socket_bound_info (mv2_arch_detect.c:898) > > ==98001== by 0xD93C87A: create_intra_sock_comm > (create_2level_comm.c:593) > > ==98001== by 0xD93BEBA: create_2level_comm (create_2level_comm.c:1762) > > ==98001== by 0xD59A894: mv2_increment_shmem_coll_counter > (ch3_shmem_coll.c:2183) > > ==98001== by 0xD4E4CBB: PMPI_Allreduce (allreduce.c:912) > > ==98001== by 0xBAB3848: hypre_MPI_Allreduce (mpistubs.c:1180) > > ==98001== by 0xB96093E: hypre_ParCSRMatrixSetNumNonzeros_core > (par_csr_matrix.c:383) > > ==98001== by 0xB960A11: hypre_ParCSRMatrixSetDNumNonzeros > (par_csr_matrix.c:413) > > ==98001== by 0xB8B32DF: hypre_BoomerAMGSetup (par_amg_setup.c:2784) > > ==98001== by 0xB8A0D26: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:47) > > ==98001== by 0xA9EDB17: PCSetUp_HYPRE (hypre.c:408) > > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > > ==98001== by 0xA93B2DB: PCSetUp_HMG (hmg.c:161) > > ==98001== by 0xA7BB0CE: PCSetUp (precon.c:1009) > > ==98001== by 0xAA2A9B9: KSPSetUp (itfunc.c:406) > > ==98001== by 0xAA2B2E9: KSPSolve_Private (itfunc.c:658) > > ==98001== by 0xAA2E8B7: KSPSolve (itfunc.c:889) > > ==98001== by 0xAC33950: SNESSolve_NEWTONLS (ls.c:225) > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Thu Jan 14 10:58:31 2021 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Thu, 14 Jan 2021 16:58:31 +0000 (UTC) Subject: [petsc-users] Error of MatMPIAIJSetPreallocation References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> Message-ID: <559521720.1160167.1610643511909@mail.yahoo.com> Hello, My program got error 55 from calling?MatMPIAIJSetPreallocation with Petsc version 3.12.5, what does this error mean? Interestingly, the same program using an older version 3.4.2 works fine. I did make necessary interface change in my program when I upgraded to 3.12.5.? Thanks in advance for your help.? Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jan 14 14:30:04 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jan 2021 15:30:04 -0500 Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: <559521720.1160167.1610643511909@mail.yahoo.com> References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> Message-ID: On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > My program got error 55 from calling MatMPIAIJSetPreallocation with Petsc > version 3.12.5, what does this error mean? Interestingly, the same program > using an older version 3.4.2 works fine. I did make necessary interface > change in my program when I upgraded to 3.12.5. > 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will get the full error message and stack to send in. 2) Your error code is here: https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 3) It is a memory error. You asked for more memory than your machine could allocate. Thanks, Matt > Thanks in advance for your help. > > Regards, > Qin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Thu Jan 14 15:38:48 2021 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Thu, 14 Jan 2021 21:38:48 +0000 (UTC) Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> Message-ID: <527768665.1223531.1610660328028@mail.yahoo.com> Hi Matthew, Thanks for your response! Does CHKERRQ work for FORTRAN-90? I linked my program with debug version of Petsc lib and got more error messages as attached below. I only used 8 processes, but the message said a big number, as well as a huge memory size, which must be wrong since the problem size is not very big. Does this mean the non-zero sizes passed to the subroutine are not correct? Thanks,Qin ==== error messages ===== [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Out of memory. This could be due to allocating [2]PETSC ERROR: too large an object or bleeding by not properly [2]PETSC ERROR: destroying unneeded objects. [2]PETSC ERROR: Memory allocated 0 Memory used by process?2049617920 [2]PETSC ERROR: Try running with -malloc_dump or -malloc_view for info. [2]PETSC ERROR: Memory requested 18446744073458206720 [2]PETSC ERROR: See?https://www.mcs.anl.gov/petsc/documentation/faq.html?for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 Sent from Yahoo Mail for iPhone On Thursday, January 14, 2021, 2:30 PM, Matthew Knepley wrote: On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users wrote: Hello, My program got error 55 from calling?MatMPIAIJSetPreallocation with Petsc version 3.12.5, what does this error mean? Interestingly, the same program using an older version 3.4.2 works fine. I did make necessary interface change in my program when I upgraded to 3.12.5.? 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will get the full error message and stack to send in. 2) Your error code is here: ??https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 3) It is a memory error. You asked for more memory than your machine could allocate. ? Thanks, ? ? ?Matt? Thanks in advance for your help.? Regards,Qin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jan 14 16:13:13 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jan 2021 17:13:13 -0500 Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: <527768665.1223531.1610660328028@mail.yahoo.com> References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> <527768665.1223531.1610660328028@mail.yahoo.com> Message-ID: On Thu, Jan 14, 2021 at 4:38 PM Qin Lu wrote: > Hi Matthew, > > Thanks for your response! Does CHKERRQ work for FORTRAN-90? > Yes, see the examples. > I linked my program with debug version of Petsc lib and got more error > messages as attached below. I only used 8 processes, but the message said a > big number, as well as a huge memory size, which must be wrong since the > problem size is not very big. Does this mean the non-zero sizes passed to > the subroutine are not correct? > Yes, the sizes passed are too large. Thanks, Matt > Thanks, > Qin > > ==== error messages ===== > > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [2]PETSC ERROR: Out of memory. This could be due to allocating > > [2]PETSC ERROR: too large an object or bleeding by not properly > > [2]PETSC ERROR: destroying unneeded objects. > > [2]PETSC ERROR: Memory allocated 0 Memory used by process 2049617920 > > [2]PETSC ERROR: Try running with -malloc_dump or -malloc_view for info. > > [2]PETSC ERROR: Memory requested 18446744073458206720 > > [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [2]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > Sent from Yahoo Mail for iPhone > > > On Thursday, January 14, 2021, 2:30 PM, Matthew Knepley > wrote: > > On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello, > > My program got error 55 from calling MatMPIAIJSetPreallocation with Petsc > version 3.12.5, what does this error mean? Interestingly, the same program > using an older version 3.4.2 works fine. I did make necessary interface > change in my program when I upgraded to 3.12.5. > > > 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will > get the full error message and stack to send in. > > 2) Your error code is here: > > https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 > > 3) It is a memory error. You asked for more memory than your machine could > allocate. > > Thanks, > > Matt > > > Thanks in advance for your help. > > Regards, > Qin > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jan 14 17:18:56 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 14 Jan 2021 17:18:56 -0600 Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> <527768665.1223531.1610660328028@mail.yahoo.com> Message-ID: <2F718F59-BE22-4107-88EE-14C55C854D20@petsc.dev> These giant sizes are sometime an indication of integer overflow. Did you previously build PETSc with the --with-64-bit-indices ? Are you sure the d_nnz and o_nnz arguments are correct and not too large? Barry > On Jan 14, 2021, at 4:13 PM, Matthew Knepley wrote: > > On Thu, Jan 14, 2021 at 4:38 PM Qin Lu > wrote: > Hi Matthew, > > Thanks for your response! Does CHKERRQ work for FORTRAN-90? > > Yes, see the examples. > > I linked my program with debug version of Petsc lib and got more error messages as attached below. I only used 8 processes, but the message said a big number, as well as a huge memory size, which must be wrong since the problem size is not very big. Does this mean the non-zero sizes passed to the subroutine are not correct? > > Yes, the sizes passed are too large. > > Thanks, > > Matt > > Thanks, > Qin > > ==== error messages ===== > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [2]PETSC ERROR: Out of memory. This could be due to allocating > > [2]PETSC ERROR: too large an object or bleeding by not properly > > [2]PETSC ERROR: destroying unneeded objects. > > [2]PETSC ERROR: Memory allocated 0 Memory used by process 2049617920 > [2]PETSC ERROR: Try running with -malloc_dump or -malloc_view for info. > > [2]PETSC ERROR: Memory requested 18446744073458206720 > > [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [2]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > Sent from Yahoo Mail for iPhone > > On Thursday, January 14, 2021, 2:30 PM, Matthew Knepley > wrote: > > On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users > wrote: > Hello, > > My program got error 55 from calling MatMPIAIJSetPreallocation with Petsc version 3.12.5, what does this error mean? Interestingly, the same program using an older version 3.4.2 works fine. I did make necessary interface change in my program when I upgraded to 3.12.5. > > 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will get the full error message and stack to send in. > > 2) Your error code is here: > > https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 > > 3) It is a memory error. You asked for more memory than your machine could allocate. > > Thanks, > > Matt > > Thanks in advance for your help. > > Regards, > Qin > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jan 14 17:40:11 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 14 Jan 2021 17:40:11 -0600 Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: <2F718F59-BE22-4107-88EE-14C55C854D20@petsc.dev> References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> <527768665.1223531.1610660328028@mail.yahoo.com> <2F718F59-BE22-4107-88EE-14C55C854D20@petsc.dev> Message-ID: <135B5B4A-6740-45C9-8C40-BD56BF0B94CA@petsc.dev> A merge request to catch more cases of integer overflow https://gitlab.com/petsc/petsc/-/merge_requests/3562 > On Jan 14, 2021, at 5:18 PM, Barry Smith wrote: > > > These giant sizes are sometime an indication of integer overflow. Did you previously build PETSc with the --with-64-bit-indices ? Are you sure the d_nnz and o_nnz arguments are correct and not too large? > > Barry > > > >> On Jan 14, 2021, at 4:13 PM, Matthew Knepley > wrote: >> >> On Thu, Jan 14, 2021 at 4:38 PM Qin Lu > wrote: >> Hi Matthew, >> >> Thanks for your response! Does CHKERRQ work for FORTRAN-90? >> >> Yes, see the examples. >> >> I linked my program with debug version of Petsc lib and got more error messages as attached below. I only used 8 processes, but the message said a big number, as well as a huge memory size, which must be wrong since the problem size is not very big. Does this mean the non-zero sizes passed to the subroutine are not correct? >> >> Yes, the sizes passed are too large. >> >> Thanks, >> >> Matt >> >> Thanks, >> Qin >> >> ==== error messages ===== >> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> >> [2]PETSC ERROR: Out of memory. This could be due to allocating >> >> [2]PETSC ERROR: too large an object or bleeding by not properly >> >> [2]PETSC ERROR: destroying unneeded objects. >> >> [2]PETSC ERROR: Memory allocated 0 Memory used by process 2049617920 >> [2]PETSC ERROR: Try running with -malloc_dump or -malloc_view for info. >> >> [2]PETSC ERROR: Memory requested 18446744073458206720 >> >> [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> >> [2]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> >> >> Sent from Yahoo Mail for iPhone >> >> On Thursday, January 14, 2021, 2:30 PM, Matthew Knepley > wrote: >> >> On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users > wrote: >> Hello, >> >> My program got error 55 from calling MatMPIAIJSetPreallocation with Petsc version 3.12.5, what does this error mean? Interestingly, the same program using an older version 3.4.2 works fine. I did make necessary interface change in my program when I upgraded to 3.12.5. >> >> 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will get the full error message and stack to send in. >> >> 2) Your error code is here: >> >> https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 >> >> 3) It is a memory error. You asked for more memory than your machine could allocate. >> >> Thanks, >> >> Matt >> >> Thanks in advance for your help. >> >> Regards, >> Qin >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Fri Jan 15 14:33:23 2021 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Fri, 15 Jan 2021 20:33:23 +0000 (UTC) Subject: [petsc-users] Error of MatMPIAIJSetPreallocation In-Reply-To: <135B5B4A-6740-45C9-8C40-BD56BF0B94CA@petsc.dev> References: <559521720.1160167.1610643511909.ref@mail.yahoo.com> <559521720.1160167.1610643511909@mail.yahoo.com> <527768665.1223531.1610660328028@mail.yahoo.com> <2F718F59-BE22-4107-88EE-14C55C854D20@petsc.dev> <135B5B4A-6740-45C9-8C40-BD56BF0B94CA@petsc.dev> Message-ID: <2089408966.1442611.1610742803388@mail.yahoo.com> It turns out some values in d_nnz were wrong. The program runs fine after I fixed the issue.? Thank you all for your help! Regards,Qin Sent from Yahoo Mail for iPhone On Thursday, January 14, 2021, 5:40 PM, Barry Smith wrote: A merge request to catch more cases of integer overflow https://gitlab.com/petsc/petsc/-/merge_requests/3562 On Jan 14, 2021, at 5:18 PM, Barry Smith wrote: ? These giant sizes are sometime an indication of integer overflow. Did you previously build PETSc with the --with-64-bit-indices ? Are you sure the d_nnz and o_nnz arguments are correct and not too large? ? Barry On Jan 14, 2021, at 4:13 PM, Matthew Knepley wrote: On Thu, Jan 14, 2021 at 4:38 PM Qin Lu wrote: Hi Matthew, Thanks for your response! Does CHKERRQ work for FORTRAN-90? Yes, see the examples.? I linked my program with debug version of Petsc lib and got more error messages as attached below. I only used 8 processes, but the message said a big number, as well as a huge memory size, which must be wrong since the problem size is not very big. Does this mean the non-zero sizes passed to the subroutine are not correct? Yes, the sizes passed are too large. ? Thanks, ? ? ?Matt? Thanks,Qin ==== error messages ===== [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Out of memory. This could be due to allocating [2]PETSC ERROR: too large an object or bleeding by not properly [2]PETSC ERROR: destroying unneeded objects. [2]PETSC ERROR: Memory allocated 0 Memory used by process?2049617920 [2]PETSC ERROR: Try running with -malloc_dump or -malloc_view for info. [2]PETSC ERROR: Memory requested 18446744073458206720 [2]PETSC ERROR: See?https://www.mcs.anl.gov/petsc/documentation/faq.html?for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 Sent from Yahoo Mail for iPhone On Thursday, January 14, 2021, 2:30 PM, Matthew Knepley wrote: On Thu, Jan 14, 2021 at 11:58 AM Qin Lu via petsc-users wrote: Hello, My program got error 55 from calling?MatMPIAIJSetPreallocation with Petsc version 3.12.5, what does this error mean? Interestingly, the same program using an older version 3.4.2 works fine. I did make necessary interface change in my program when I upgraded to 3.12.5.? 1) You should always call CHKERRQ(ierr) after a Petsc call. Then you will get the full error message and stack to send in. 2) Your error code is here: ??https://gitlab.com/petsc/petsc/-/blob/v3.12.5/include/petscerror.h#L17 3) It is a memory error. You asked for more memory than your machine could allocate. ? Thanks, ? ? ?Matt? Thanks in advance for your help.? Regards,Qin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Sun Jan 17 12:28:20 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Sun, 17 Jan 2021 19:28:20 +0100 Subject: [petsc-users] PetscSFBcastAndOpBegin() fails when called from MatDiagonalScale() for n_threads > 1 with "Wrong type of object" Message-ID: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> Dear all, I am currently encountering issues in my program when calling MatDiagonalScale(matrix, NULL, vector) in an MPI-context for at least two threads. The backtrace from gdb (and the error output) tells me that the problem lies apparently in PetscSFBcastAndOpBegin(), with the exact error listed as /petsc/src/vec/is/sf/interface/sf.c Wrong type of object: Parameter # 1 / When printing the involved matrix and vector, everything looks as expected (the matrix is split over the processes equally with continuous rows, the vector is distributed over both processes). Is the issue that my vector is distributed over several processes? Or is there another issue I am not seeing here? I tried to narrow the problem down by writing a MWE, but was not able to reproduce it in a smaller scale. Therefore, are there other approaches I could try, too? Thanks! Regards, Roland Richter -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Sun Jan 17 12:40:34 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sun, 17 Jan 2021 21:40:34 +0300 Subject: [petsc-users] PetscSFBcastAndOpBegin() fails when called from MatDiagonalScale() for n_threads > 1 with "Wrong type of object" In-Reply-To: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> References: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> Message-ID: Valgrind https://valgrind.org/ Il Dom 17 Gen 2021, 21:28 Roland Richter ha scritto: > Dear all, > > I am currently encountering issues in my program when calling > MatDiagonalScale(matrix, NULL, vector) in an MPI-context for at least two > threads. The backtrace from gdb (and the error output) tells me that the > problem lies apparently in PetscSFBcastAndOpBegin(), with the exact error > listed as > *petsc/src/vec/is/sf/interface/sf.c Wrong type of object: Parameter # 1 * > > When printing the involved matrix and vector, everything looks as expected > (the matrix is split over the processes equally with continuous rows, the > vector is distributed over both processes). > > Is the issue that my vector is distributed over several processes? Or is > there another issue I am not seeing here? > > I tried to narrow the problem down by writing a MWE, but was not able to > reproduce it in a smaller scale. Therefore, are there other approaches I > could try, too? > > Thanks! > > Regards, > > Roland Richter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jan 17 14:05:23 2021 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 17 Jan 2021 15:05:23 -0500 Subject: [petsc-users] PetscSFBcastAndOpBegin() fails when called from MatDiagonalScale() for n_threads > 1 with "Wrong type of object" In-Reply-To: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> References: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> Message-ID: On Sun, Jan 17, 2021 at 1:28 PM Roland Richter wrote: > Dear all, > > I am currently encountering issues in my program when calling > MatDiagonalScale(matrix, NULL, vector) in an MPI-context for at least two > threads. The backtrace from gdb (and the error output) tells me that the > problem lies apparently in PetscSFBcastAndOpBegin(), with the exact error > listed as > *petsc/src/vec/is/sf/interface/sf.c Wrong type of object: Parameter # 1 * > > 1) Always send the entire error trace 2) You are using right scaling, so it would use the matvec scatter internally. If MatSetUp() has not been called, this scatter will not have been created. I think we are missing a check here that the matrix is setup. Thanks, Matt > When printing the involved matrix and vector, everything looks as expected > (the matrix is split over the processes equally with continuous rows, the > vector is distributed over both processes). > > Is the issue that my vector is distributed over several processes? Or is > there another issue I am not seeing here? > > I tried to narrow the problem down by writing a MWE, but was not able to > reproduce it in a smaller scale. Therefore, are there other approaches I > could try, too? > > Thanks! > > Regards, > > Roland Richter > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jan 17 15:08:08 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 17 Jan 2021 16:08:08 -0500 Subject: [petsc-users] Kokkos with OpenMP Message-ID: I am getting this Kokkos build failure with OpenMP. Also, can you build Kokkos with OMP and Cuda? I know you can't have two execution spaces but can you use Kokkos-Cuda with OMP in the same executable? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 5058693 bytes Desc: not available URL: From bsmith at petsc.dev Sun Jan 17 19:10:10 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 17 Jan 2021 19:10:10 -0600 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: References: Message-ID: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> Mark, > On Jan 17, 2021, at 3:08 PM, Mark Adams wrote: > > I am getting this Kokkos build failure with OpenMP. Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS. It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you. But yes it should still work. I don't know how to fix configure to handle this case cleanly. We could not use the CXXPPFLAGS but always use the compiler flags for checking includes, this would change a 20 year old decision, since CXXPPFLAGS is for the preprocess :-(. Please remove the -fPIC -fopenmp from your ./configure options and see if it succeeds. (see below for the answer to your second question). ----- TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86) Checks if a particular include file can be found along particular include paths Checking for header files ['KokkosBlas.hpp', 'KokkosSparse_CrsMatrix.hpp'] in ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include'] Checking include with compiler flags var CXXPPFLAGS ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include', '/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include'] Preprocessing source: #include "confdefs.h" #include "conffix.h" #include Executing: mpicxx -E -I/tmp/petsc-jEwQ8b/config.setCompilers -I/tmp/petsc-jEwQ8b/config.packages.kokkos -I/tmp/petsc-jEwQ8b/config.headers -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include /tmp/petsc-jEwQ8b/config.headers/conftest.cc Possible ERROR while running preprocessor: exit code 1 stdout: ---- > Also, can you build Kokkos with OMP and Cuda? I know you can't have two execution spaces but can you use Kokkos-Cuda with OMP in the same executable? In theory you can do this but Barry enforced a packages design decision (for the sake of less configure options;) that does not allow it by default. Edit kokkos.py and remove the lines. if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration") if self.openmp.found: args.append('-DKokkos_ENABLE_OPENMP=ON') self.system = 'OpenMP' I guess we will need to add an option like --with-kokkos-device=CUDA or OpenMP or prethread or HIP to allow the user to set a particular device when several are possible. Stefano pushed for this long ago but I resisted. Barry > Thanks, > Mark > From mfadams at lbl.gov Sun Jan 17 21:39:16 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 17 Jan 2021 22:39:16 -0500 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> References: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> Message-ID: > > > > Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to > indicate how to compile for OpenMP but when ./configure checks the > installed KokkosKernel include files it uses CXXPPFLAGS. (Since you > provided the flag CXXOPTFLAGS ./configure never determines the flag to put > into CXXPPFLAGS. > > It is more desirable if you do not provide -fPIC -fopenmp. ./configure > will figure them out for you. > :( -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 886061 bytes Desc: not available URL: From bsmith at petsc.dev Mon Jan 18 00:00:46 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 18 Jan 2021 00:00:46 -0600 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: References: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> Message-ID: <06C45F26-AC0D-46CE-8182-129077AF83B9@petsc.dev> Mark, Looks like you tried to do two things at once, you removed the fopenmp and put back the --with-cuda. Like I said if you want PETSc to build OpenMP and CUDA you need to remove this line (and some others) from kokkos.py if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration") > On Jan 17, 2021, at 9:39 PM, Mark Adams wrote: > > > > Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS. > > It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you. > > :( > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jan 18 08:16:09 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 18 Jan 2021 09:16:09 -0500 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: <06C45F26-AC0D-46CE-8182-129077AF83B9@petsc.dev> References: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> <06C45F26-AC0D-46CE-8182-129077AF83B9@petsc.dev> Message-ID: OK, this running and using OMP Fieldsplit/additive. Kokkos does not have any solvers hooked up yet, but I have added an LU interface that just uses AIJ's LU. Junchao, I asked Kokkos about adding LU and they said that it is on the todo list. Until they do that I can't use it for the paper, but if you want to work on integrating solvers into aijkokkos I can do that and test with GMRES/Jacobi. Thanks, Mark On Mon, Jan 18, 2021 at 1:00 AM Barry Smith wrote: > > Mark, > > Looks like you tried to do two things at once, you removed the fopenmp > and put back the --with-cuda. > > > > Like I said if you want PETSc to build OpenMP and CUDA you need to > remove this line (and some others) from kokkos.py > > > > if self.openmp.found + pthreadfound + self.cuda.found > 1: > raise RuntimeError("Kokkos only supports a single parallel system > during its configuration") > > > > On Jan 17, 2021, at 9:39 PM, Mark Adams wrote: > > >> >> Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to >> indicate how to compile for OpenMP but when ./configure checks the >> installed KokkosKernel include files it uses CXXPPFLAGS. (Since you >> provided the flag CXXOPTFLAGS ./configure never determines the flag to put >> into CXXPPFLAGS. >> >> It is more desirable if you do not provide -fPIC -fopenmp. ./configure >> will figure them out for you. >> > > :( > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 19 05:58:46 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 19 Jan 2021 12:58:46 +0100 Subject: [petsc-users] PetscSFBcastAndOpBegin() fails when called from MatDiagonalScale() for n_threads > 1 with "Wrong type of object" In-Reply-To: References: <5183e5b6-07b5-bfc7-92f5-13a5f238f8d6@ntnu.no> Message-ID: <8f1d3a86-ee0e-9e16-3a64-c848437b919b@ntnu.no> Hei, thanks for the idea, that solved my problem! I noticed that I was writing out of bounds in an earlier loop, which then resulted in the behavior observed below. Regards, Roland Am 17.01.21 um 19:40 schrieb Stefano Zampini: > Valgrind https://valgrind.org/ > > Il Dom 17 Gen 2021, 21:28 Roland Richter > ha scritto: > > Dear all, > > I am currently encountering issues in my program when calling > MatDiagonalScale(matrix, NULL, vector) in an MPI-context for at > least two threads. The backtrace from gdb (and the error output) > tells me that the problem lies apparently in > PetscSFBcastAndOpBegin(), with the exact error listed as > /petsc/src/vec/is/sf/interface/sf.c Wrong type of object: > Parameter # 1 > / > > When printing the involved matrix and vector, everything looks as > expected (the matrix is split over the processes equally with > continuous rows, the vector is distributed over both processes). > > Is the issue that my vector is distributed over several processes? > Or is there another issue I am not seeing here? > > I tried to narrow the problem down by writing a MWE, but was not > able to reproduce it in a smaller scale. Therefore, are there > other approaches I could try, too? > > Thanks! > > Regards, > > Roland Richter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 19 06:28:19 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 19 Jan 2021 13:28:19 +0100 Subject: [petsc-users] Compilation error of test program: "MatDenseGetArray" was not declared in this scope Message-ID: <71941992-1609-616e-5276-461498637c19@ntnu.no> Hei, I just encountered an issue which I can not solve on my own. I wrote a test program on my development machine, and compilation works fine, but when trying to compile it on the production machine compilation fails with /[ 25%] Building CXX object CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o// ///share/apps/software/OpenMPI/3.1.3-gcccuda-2019a/bin/mpicxx -DBOOST_NO_AUTO_PTR -DUSE_CUDA -DUSE_PHYSICS_VARIABLES -Dtest_program_EXPORTS -I~/work_directory/full_physics/Test-Controller/library/include -I~/work_directory/full_physics/physics_variables/include -I~/local_opt_GCC7/interp2d/include -I/usr/include/eigen3 -I/share/apps/software/Compiler/GCC/8.2.0-2.31.1/GSL/2.5/include -I/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/imkl/2018.3.222/compilers_and_libraries_2018.3.222/linux/mkl/include -isystem ~/local_opt_GCC7/dealii/include -isystem ~/local_opt_GCC7/dealii/include/deal.II/bundled -isystem /share/apps/software/Core/icc/2019.1.144-GCC-8.2.0-2.31.1/tbb/include -isystem ~/local_opt_GCC7/boost/include -isystem ~/local_opt_GCC7/petsc/include -isystem ~/local_opt_GCC7/trilinos/include -isystem ~/local_opt_GCC7/arrayfire/include -isystem /share/apps/software/imkl/2019.5.281-iimpi-2019b/mkl/include -isystem ~/local_opt_GCC7/suitesparse/include -isystem /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/HDF5/1.10.5/include -isystem /share/apps/software/Compiler/GCCcore/8.2.0/zlib/1.2.11/include -isystem /share/apps/software/Compiler/GCCcore/8.2.0/Szip/2.1.1/include -isystem /share/apps/software/Compiler/GCC/8.2.0-2.31.1/CUDA/10.1.105/include -isystem /share/apps/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/include -isystem ~/local_opt_GCC7/adol-c/include -isystem ~/local_opt_GCC7/p4est/include -isystem ~/local_opt_GCC7/slepc/include -isystem ~/local_opt_GCC7/armadillo/include -isystem /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/FFTW/3.3.8/include -march=native -fopenmp-simd? -fPIC -flto -march=native -fopenmp -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -DMKL_LP64 -m64 -O3 -DNDEBUG -fPIC -pedantic -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd -march=native -mavx -Wno-unused-local-typedefs -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -Werror -flto -std=gnu++17 -MD -MT CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -MF CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o.d -o CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -c ~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: In member function 'void tester_helper::nonlinear_step(double&, double)':// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: error: 'MatDenseGetArrayWrite' was not declared in this scope// //???? MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);// //???? ^~~~~~~~~~~~~~~~~~~~~// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: note: suggested alternative: 'MatDenseGetArrayRead'// //???? MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);// //???? ^~~~~~~~~~~~~~~~~~~~~// //???? MatDenseGetArrayRead// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: error: 'MatDenseRestoreArrayWrite' was not declared in this scope// //???? MatDenseRestoreArrayWrite (petsc_field.pc_spectral, &write_ptr);// //???? ^~~~~~~~~~~~~~~~~~~~~~~~~// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: note: suggested alternative: 'MatDenseRestoreArrayRead'// //???? MatDenseRestoreArrayWrite (petsc_field.pc_spectral, &write_ptr);// //???? ^~~~~~~~~~~~~~~~~~~~~~~~~// //???? MatDenseRestoreArrayRead// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: In member function 'void tester_helper::nonlinear_step_material_transition()':// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: error: 'MatDenseGetArrayWrite' was not declared in this scope// //??? MatDenseGetArrayWrite(split_mat, &write_mat_ptr);// //??? ^~~~~~~~~~~~~~~~~~~~~// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: note: suggested alternative: 'MatDenseGetArrayRead'// //??? MatDenseGetArrayWrite(split_mat, &write_mat_ptr);// //??? ^~~~~~~~~~~~~~~~~~~~~// //??? MatDenseGetArrayRead// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: error: 'MatDenseRestoreArrayWrite' was not declared in this scope// //??? MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);// //??? ^~~~~~~~~~~~~~~~~~~~~~~~~// //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: note: suggested alternative: 'MatDenseRestoreArrayRead'// //??? MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);// //??? ^~~~~~~~~~~~~~~~~~~~~~~~~// //??? MatDenseRestoreArrayRead/ Where does that error come from, and how can I fix it/circumvent it? I compile the program using gcc 8.2.0 (including mpi-wrappers) on the production machine, and with gcc 9.3.1 on the dev machine. Thanks, regards, Roland -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 19 07:55:28 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jan 2021 08:55:28 -0500 Subject: [petsc-users] Compilation error of test program: "MatDenseGetArray" was not declared in this scope In-Reply-To: <71941992-1609-616e-5276-461498637c19@ntnu.no> References: <71941992-1609-616e-5276-461498637c19@ntnu.no> Message-ID: On Tue, Jan 19, 2021 at 7:28 AM Roland Richter wrote: > Hei, > > I just encountered an issue which I can not solve on my own. I wrote a > test program on my development machine, and compilation works fine, but > when trying to compile it on the production machine compilation fails with > > I believe your production machine has PETSc version < 3.14, since MatDenseGetArrayWrite() was not introduced until 3,14. Thanks, Matt > *[ 25%] Building CXX object > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o* > */share/apps/software/OpenMPI/3.1.3-gcccuda-2019a/bin/mpicxx > -DBOOST_NO_AUTO_PTR -DUSE_CUDA -DUSE_PHYSICS_VARIABLES > -Dtest_program_EXPORTS > -I~/work_directory/full_physics/Test-Controller/library/include > -I~/work_directory/full_physics/physics_variables/include > -I~/local_opt_GCC7/interp2d/include -I/usr/include/eigen3 > -I/share/apps/software/Compiler/GCC/8.2.0-2.31.1/GSL/2.5/include > -I/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/imkl/2018.3.222/compilers_and_libraries_2018.3.222/linux/mkl/include > -isystem ~/local_opt_GCC7/dealii/include -isystem > ~/local_opt_GCC7/dealii/include/deal.II/bundled -isystem > /share/apps/software/Core/icc/2019.1.144-GCC-8.2.0-2.31.1/tbb/include > -isystem ~/local_opt_GCC7/boost/include -isystem > ~/local_opt_GCC7/petsc/include -isystem ~/local_opt_GCC7/trilinos/include > -isystem ~/local_opt_GCC7/arrayfire/include -isystem > /share/apps/software/imkl/2019.5.281-iimpi-2019b/mkl/include -isystem > ~/local_opt_GCC7/suitesparse/include -isystem > /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/HDF5/1.10.5/include > -isystem /share/apps/software/Compiler/GCCcore/8.2.0/zlib/1.2.11/include > -isystem /share/apps/software/Compiler/GCCcore/8.2.0/Szip/2.1.1/include > -isystem > /share/apps/software/Compiler/GCC/8.2.0-2.31.1/CUDA/10.1.105/include > -isystem > /share/apps/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/include > -isystem ~/local_opt_GCC7/adol-c/include -isystem > ~/local_opt_GCC7/p4est/include -isystem ~/local_opt_GCC7/slepc/include > -isystem ~/local_opt_GCC7/armadillo/include -isystem > /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/FFTW/3.3.8/include > -march=native -fopenmp-simd -fPIC -flto -march=native -fopenmp -O2 > -funroll-loops -funroll-all-loops -fstrict-aliasing -DMKL_LP64 -m64 -O3 > -DNDEBUG -fPIC -pedantic -fPIC -Wall -Wextra -Wmissing-braces > -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wswitch -Wsynth > -Wwrite-strings -Wno-placement-new -Wno-literal-suffix -Wno-psabi > -Wno-class-memaccess -fopenmp-simd -march=native -mavx > -Wno-unused-local-typedefs -O2 -funroll-loops -funroll-all-loops > -fstrict-aliasing -Werror -flto -std=gnu++17 -MD -MT > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -MF > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o.d -o > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -c > ~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: > In member function 'void tester_helper::nonlinear_step(double&, double)':* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: > error: 'MatDenseGetArrayWrite' was not declared in this scope* > * MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: > note: suggested alternative: 'MatDenseGetArrayRead'* > * MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~* > * MatDenseGetArrayRead* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: > error: 'MatDenseRestoreArrayWrite' was not declared in this scope* > * MatDenseRestoreArrayWrite (petsc_field.pc_spectral, &write_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~~~~~* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: > note: suggested alternative: 'MatDenseRestoreArrayRead'* > * MatDenseRestoreArrayWrite (petsc_field.pc_spectral, &write_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~~~~~* > * MatDenseRestoreArrayRead* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: > In member function 'void > tester_helper::nonlinear_step_material_transition()':* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: > error: 'MatDenseGetArrayWrite' was not declared in this scope* > * MatDenseGetArrayWrite(split_mat, &write_mat_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: > note: suggested alternative: 'MatDenseGetArrayRead'* > * MatDenseGetArrayWrite(split_mat, &write_mat_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~* > * MatDenseGetArrayRead* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: > error: 'MatDenseRestoreArrayWrite' was not declared in this scope* > * MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~~~~~* > *~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: > note: suggested alternative: 'MatDenseRestoreArrayRead'* > * MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);* > * ^~~~~~~~~~~~~~~~~~~~~~~~~* > * MatDenseRestoreArrayRead* > > Where does that error come from, and how can I fix it/circumvent it? I > compile the program using gcc 8.2.0 (including mpi-wrappers) on the > production machine, and with gcc 9.3.1 on the dev machine. > > Thanks, > > regards, > > Roland > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 19 08:33:40 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 19 Jan 2021 15:33:40 +0100 Subject: [petsc-users] Compilation error of test program: "MatDenseGetArray" was not declared in this scope In-Reply-To: References: <71941992-1609-616e-5276-461498637c19@ntnu.no> Message-ID: <54389c7c-e230-481c-6f92-d617a573985b@ntnu.no> Yes, that's it, thanks for the fast help! Regards, Roland Am 19.01.21 um 14:55 schrieb Matthew Knepley: > On Tue, Jan 19, 2021 at 7:28 AM Roland Richter > wrote: > > Hei, > > I just encountered an issue which I can not solve on my own. I > wrote a test program on my development machine, and compilation > works fine, but when trying to compile it on the production > machine compilation fails with > > I believe your production machine has PETSc version < 3.14, since > MatDenseGetArrayWrite() was not introduced until 3,14. > > ? Thanks, > > ? ? ?Matt? > > /[ 25%] Building CXX object > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o// > ///share/apps/software/OpenMPI/3.1.3-gcccuda-2019a/bin/mpicxx > -DBOOST_NO_AUTO_PTR -DUSE_CUDA -DUSE_PHYSICS_VARIABLES > -Dtest_program_EXPORTS > -I~/work_directory/full_physics/Test-Controller/library/include > -I~/work_directory/full_physics/physics_variables/include > -I~/local_opt_GCC7/interp2d/include -I/usr/include/eigen3 > -I/share/apps/software/Compiler/GCC/8.2.0-2.31.1/GSL/2.5/include > -I/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/imkl/2018.3.222/compilers_and_libraries_2018.3.222/linux/mkl/include > -isystem ~/local_opt_GCC7/dealii/include -isystem > ~/local_opt_GCC7/dealii/include/deal.II/bundled -isystem > /share/apps/software/Core/icc/2019.1.144-GCC-8.2.0-2.31.1/tbb/include > -isystem ~/local_opt_GCC7/boost/include -isystem > ~/local_opt_GCC7/petsc/include -isystem > ~/local_opt_GCC7/trilinos/include -isystem > ~/local_opt_GCC7/arrayfire/include -isystem > /share/apps/software/imkl/2019.5.281-iimpi-2019b/mkl/include > -isystem ~/local_opt_GCC7/suitesparse/include -isystem > /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/HDF5/1.10.5/include > -isystem > /share/apps/software/Compiler/GCCcore/8.2.0/zlib/1.2.11/include > -isystem > /share/apps/software/Compiler/GCCcore/8.2.0/Szip/2.1.1/include > -isystem > /share/apps/software/Compiler/GCC/8.2.0-2.31.1/CUDA/10.1.105/include > -isystem > /share/apps/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/include > -isystem ~/local_opt_GCC7/adol-c/include -isystem > ~/local_opt_GCC7/p4est/include -isystem > ~/local_opt_GCC7/slepc/include -isystem > ~/local_opt_GCC7/armadillo/include -isystem > /share/apps/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/FFTW/3.3.8/include > -march=native -fopenmp-simd? -fPIC -flto -march=native -fopenmp > -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -DMKL_LP64 > -m64 -O3 -DNDEBUG -fPIC -pedantic -fPIC -Wall -Wextra > -Wmissing-braces -Woverloaded-virtual -Wpointer-arith > -Wsign-compare -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new > -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd > -march=native -mavx -Wno-unused-local-typedefs -O2 -funroll-loops > -funroll-all-loops -fstrict-aliasing -Werror -flto -std=gnu++17 > -MD -MT > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -MF > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o.d -o > CMakeFiles/test_program.dir/source/core/tester_helper.cpp.o -c > ~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: > In member function 'void tester_helper::nonlinear_step(double&, > double)':// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: > error: 'MatDenseGetArrayWrite' was not declared in this scope// > //???? MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);// > //???? ^~~~~~~~~~~~~~~~~~~~~// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:765:5: > note: suggested alternative: 'MatDenseGetArrayRead'// > //???? MatDenseGetArrayWrite (petsc_field.pc_spectral, &write_ptr);// > //???? ^~~~~~~~~~~~~~~~~~~~~// > //???? MatDenseGetArrayRead// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: > error: 'MatDenseRestoreArrayWrite' was not declared in this scope// > //???? MatDenseRestoreArrayWrite (petsc_field.pc_spectral, > &write_ptr);// > //???? ^~~~~~~~~~~~~~~~~~~~~~~~~// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:772:5: > note: suggested alternative: 'MatDenseRestoreArrayRead'// > //???? MatDenseRestoreArrayWrite (petsc_field.pc_spectral, > &write_ptr);// > //???? ^~~~~~~~~~~~~~~~~~~~~~~~~// > //???? MatDenseRestoreArrayRead// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp: > In member function 'void > tester_helper::nonlinear_step_material_transition()':// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: > error: 'MatDenseGetArrayWrite' was not declared in this scope// > //??? MatDenseGetArrayWrite(split_mat, &write_mat_ptr);// > //??? ^~~~~~~~~~~~~~~~~~~~~// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:870:4: > note: suggested alternative: 'MatDenseGetArrayRead'// > //??? MatDenseGetArrayWrite(split_mat, &write_mat_ptr);// > //??? ^~~~~~~~~~~~~~~~~~~~~// > //??? MatDenseGetArrayRead// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: > error: 'MatDenseRestoreArrayWrite' was not declared in this scope// > //??? MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);// > //??? ^~~~~~~~~~~~~~~~~~~~~~~~~// > //~/work_directory/full_physics/Test-Controller/library/source/core/tester_helper.cpp:874:4: > note: suggested alternative: 'MatDenseRestoreArrayRead'// > //??? MatDenseRestoreArrayWrite(split_mat, &write_mat_ptr);// > //??? ^~~~~~~~~~~~~~~~~~~~~~~~~// > //??? MatDenseRestoreArrayRead/ > > Where does that error come from, and how can I fix it/circumvent > it? I compile the program using gcc 8.2.0 (including mpi-wrappers) > on the production machine, and with gcc 9.3.1 on the dev machine. > > Thanks, > > regards, > > Roland > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Tue Jan 19 09:15:53 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Tue, 19 Jan 2021 16:15:53 +0100 Subject: [petsc-users] Step size for adaptive stepper when running on several MPI threads Message-ID: <238a2a54-40a9-2a78-89cb-e5ae22af52e3@ntnu.no> Hei, during recent experiments with the adaptive stepper in odeint I noticed that the step size used for each step can vary if I am using more than one thread at the same time (i.e. thread 1 can have a different step size than thread 2 when doing the same step). This is not suitable for my project, after the function calculating the right hand side relies on having the same step size for each thread. Before starting deeper experiments with the adaptive steppers in PETSc, I therefore wanted to ask if those steppers synchronize the step size of the sub-steps between the different mpi processes at each step, or if not? Thanks! Regards, Roland From knepley at gmail.com Tue Jan 19 09:18:52 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jan 2021 10:18:52 -0500 Subject: [petsc-users] Step size for adaptive stepper when running on several MPI threads In-Reply-To: <238a2a54-40a9-2a78-89cb-e5ae22af52e3@ntnu.no> References: <238a2a54-40a9-2a78-89cb-e5ae22af52e3@ntnu.no> Message-ID: On Tue, Jan 19, 2021 at 10:16 AM Roland Richter wrote: > Hei, > > during recent experiments with the adaptive stepper in odeint I noticed > that the step size used for each step can vary if I am using more than > one thread at the same time (i.e. thread 1 can have a different step > size than thread 2 when doing the same step). This is not suitable for > my project, after the function calculating the right hand side relies on > having the same step size for each thread. > > Before starting deeper experiments with the adaptive steppers in PETSc, > I therefore wanted to ask if those steppers synchronize the step size of > the sub-steps between the different mpi processes at each step, or if not? > Yes, the timestep is the same across the communicator that you create the TS with. Thanks, Matt > Thanks! > > Regards, > > Roland > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jan 19 09:27:20 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 19 Jan 2021 09:27:20 -0600 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: References: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> <06C45F26-AC0D-46CE-8182-129077AF83B9@petsc.dev> Message-ID: Mark, I hope I can take up the Kokkos work in early February once we have the ecp-tpds paper done. --Junchao Zhang On Mon, Jan 18, 2021 at 8:16 AM Mark Adams wrote: > OK, this running and using OMP Fieldsplit/additive. > > Kokkos does not have any solvers hooked up yet, but I have added an LU > interface that just uses AIJ's LU. > > Junchao, I asked Kokkos about adding LU and they said that it is on the > todo list. Until they do that I can't use it for the paper, but if you want > to work on integrating solvers into aijkokkos I can do that and test with > GMRES/Jacobi. > > Thanks, > Mark > > On Mon, Jan 18, 2021 at 1:00 AM Barry Smith wrote: > >> >> Mark, >> >> Looks like you tried to do two things at once, you removed the fopenmp >> and put back the --with-cuda. >> >> >> >> Like I said if you want PETSc to build OpenMP and CUDA you need to >> remove this line (and some others) from kokkos.py >> >> >> >> if self.openmp.found + pthreadfound + self.cuda.found > 1: >> raise RuntimeError("Kokkos only supports a single parallel system >> during its configuration") >> >> >> >> On Jan 17, 2021, at 9:39 PM, Mark Adams wrote: >> >> >>> >>> Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to >>> indicate how to compile for OpenMP but when ./configure checks the >>> installed KokkosKernel include files it uses CXXPPFLAGS. (Since you >>> provided the flag CXXOPTFLAGS ./configure never determines the flag to put >>> into CXXPPFLAGS. >>> >>> It is more desirable if you do not provide -fPIC -fopenmp. ./configure >>> will figure them out for you. >>> >> >> :( >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sblondel at utk.edu Tue Jan 19 10:13:55 2021 From: sblondel at utk.edu (Blondel, Sophie) Date: Tue, 19 Jan 2021 16:13:55 +0000 Subject: [petsc-users] TSSetEventHandler and TSSetPostEventIntervalStep In-Reply-To: References: <52CD65C8-DB69-4799-ACC7-0B2E5C32FE54@petsc.dev> , , , Message-ID: Hi Barry and Matt and happy new year, I wanted to check again if there was anything new with the TSSetEventHandler and TSSetPostEventIntervalStep. If you think there could be an alternative way to obtain the behavior I want in PETSc I'd be happy to try it. Cheers, Sophie ________________________________ From: Blondel, Sophie Sent: Tuesday, December 1, 2020 16:14 To: Barry Smith ; Matthew Knepley Cc: petsc-users at mcs.anl.gov ; xolotl-psi-development at lists.sourceforge.net Subject: Re: [petsc-users] TSSetEventHandler and TSSetPostEventIntervalStep Hi again Barry and Matt, Just wanted to check the status for the TSSetEventHandler. Let me know if I can help in any way. Cheers, Sophie ________________________________ From: Blondel, Sophie Sent: Monday, November 16, 2020 11:38 To: Barry Smith ; Matthew Knepley Cc: petsc-users at mcs.anl.gov ; xolotl-psi-development at lists.sourceforge.net Subject: Re: [petsc-users] TSSetEventHandler and TSSetPostEventIntervalStep Hi Matt and Barry, I wanted to check if you wanted me to test anything with the branch from Barry, it was not clear from the previous emails. Cheers, Sophie ________________________________ From: Barry Smith Sent: Tuesday, October 27, 2020 17:01 To: Matthew Knepley Cc: Blondel, Sophie ; petsc-users at mcs.anl.gov ; xolotl-psi-development at lists.sourceforge.net Subject: Re: [petsc-users] TSSetEventHandler and TSSetPostEventIntervalStep Pushed On Oct 27, 2020, at 3:41 PM, Matthew Knepley > wrote: On Tue, Oct 27, 2020 at 4:24 PM Barry Smith > wrote: I'm sorry the code is still fundamentally broken, I know I promised a long time ago to fix it all up but it is actually pretty hard to get right. It detects the zero by finding a small value when it should detect it by find a small region where it changes sign but surprising it is so hardwired to the size test that fixing it and testing the new code has been very difficult to me. My branch is barry/2019-08-18/fix-tsevent-posteventdt Barry, I do not see this branch on gitlab. Can you give a URL? Thanks, Matt Barry On Oct 27, 2020, at 3:02 PM, Blondel, Sophie via petsc-users > wrote: Hi Matt, With the ex40 I attached in my previous email here is what I get printed on screen when running "./ex40 -ts_monitor -ts_event_monitor": 0 TS dt 0.1 time 0. 1 TS dt 0.5 time 0.1 2 TS dt 0.5 time 0.6 3 TS dt 0.5 time 1.1 4 TS dt 0.5 time 1.6 5 TS dt 0.5 time 2.1 6 TS dt 0.5 time 2.6 7 TS dt 0.5 time 3.1 8 TS dt 0.5 time 3.6 9 TS dt 0.5 time 4.1 10 TS dt 0.5 time 4.6 11 TS dt 0.5 time 5.1 12 TS dt 0.5 time 5.6 13 TS dt 0.5 time 6.1 14 TS dt 0.5 time 6.6 15 TS dt 0.5 time 7.1 TSEvent: Event 0 zero crossing at time 7.6 located in 0 iterations Ball hit the ground at t = 7.60 seconds 16 TS dt 0.5 time 7.6 17 TS dt 0.5 time 8.1 18 TS dt 0.5 time 8.6 19 TS dt 0.5 time 9.1 20 TS dt 0.5 time 9.6 21 TS dt 0.5 time 10.1 22 TS dt 0.5 time 10.6 23 TS dt 0.5 time 11.1 24 TS dt 0.5 time 11.6 25 TS dt 0.5 time 12.1 26 TS dt 0.5 time 12.6 27 TS dt 0.5 time 13.1 28 TS dt 0.5 time 13.6 29 TS dt 0.5 time 14.1 30 TS dt 0.5 time 14.6 31 TS dt 0.5 time 15.1 32 TS dt 0.5 time 15.6 33 TS dt 0.5 time 16.1 34 TS dt 0.5 time 16.6 35 TS dt 0.5 time 17.1 36 TS dt 0.5 time 17.6 37 TS dt 0.5 time 18.1 38 TS dt 0.5 time 18.6 39 TS dt 0.5 time 19.1 40 TS dt 0.5 time 19.6 41 TS dt 0.5 time 20.1 42 TS dt 0.5 time 20.6 43 TS dt 0.5 time 21.1 44 TS dt 0.5 time 21.6 45 TS dt 0.5 time 22.1 46 TS dt 0.5 time 22.6 47 TS dt 0.5 time 23.1 48 TS dt 0.5 time 23.6 49 TS dt 0.5 time 24.1 50 TS dt 0.5 time 24.6 51 TS dt 0.5 time 25.1 TSEvent: Event 0 zero crossing at time 25.6 located in 0 iterations Ball hit the ground at t = 25.60 seconds 52 TS dt 0.5 time 25.6 53 TS dt 0.5 time 26.1 54 TS dt 0.5 time 26.6 55 TS dt 0.5 time 27.1 56 TS dt 0.5 time 27.6 57 TS dt 0.5 time 28.1 58 TS dt 0.5 time 28.6 59 TS dt 0.5 time 29.1 60 TS dt 0.5 time 29.6 61 TS dt 0.5 time 30.1 0 TS dt 0.1 time 0. 1 TS dt 0.5 time 0.1 2 TS dt 0.5 time 0.6 3 TS dt 0.5 time 1.1 4 TS dt 0.5 time 1.6 5 TS dt 0.5 time 2.1 6 TS dt 0.5 time 2.6 7 TS dt 0.5 time 3.1 8 TS dt 0.5 time 3.6 9 TS dt 0.5 time 4.1 10 TS dt 0.5 time 4.6 11 TS dt 0.5 time 5.1 12 TS dt 0.5 time 5.6 13 TS dt 0.5 time 6.1 14 TS dt 0.5 time 6.6 15 TS dt 0.5 time 7.1 16 TS dt 0.5 time 7.6 17 TS dt 0.5 time 8.1 18 TS dt 0.5 time 8.6 19 TS dt 0.5 time 9.1 20 TS dt 0.5 time 9.6 21 TS dt 0.5 time 10.1 22 TS dt 0.5 time 10.6 23 TS dt 0.5 time 11.1 24 TS dt 0.5 time 11.6 25 TS dt 0.5 time 12.1 26 TS dt 0.5 time 12.6 TSEvent: Event 0 zero crossing at time 13.1 located in 0 iterations Ball hit the ground at t = 13.10 seconds 27 TS dt 0.5 time 13.1 28 TS dt 0.5 time 13.6 29 TS dt 0.5 time 14.1 30 TS dt 0.5 time 14.6 31 TS dt 0.5 time 15.1 32 TS dt 0.5 time 15.6 33 TS dt 0.5 time 16.1 34 TS dt 0.5 time 16.6 35 TS dt 0.5 time 17.1 36 TS dt 0.5 time 17.6 37 TS dt 0.5 time 18.1 38 TS dt 0.5 time 18.6 39 TS dt 0.5 time 19.1 40 TS dt 0.5 time 19.6 41 TS dt 0.5 time 20.1 42 TS dt 0.5 time 20.6 43 TS dt 0.5 time 21.1 44 TS dt 0.5 time 21.6 45 TS dt 0.5 time 22.1 46 TS dt 0.5 time 22.6 47 TS dt 0.5 time 23.1 TSEvent: Event 0 zero crossing at time 23.6 located in 0 iterations Ball hit the ground at t = 23.60 seconds 48 TS dt 0.5 time 23.6 49 TS dt 0.5 time 24.1 50 TS dt 0.5 time 24.6 51 TS dt 0.5 time 25.1 52 TS dt 0.5 time 25.6 53 TS dt 0.5 time 26.1 TSEvent: Event 0 zero crossing at time 26.6 located in 0 iterations Ball hit the ground at t = 26.60 seconds 54 TS dt 0.5 time 26.6 55 TS dt 0.5 time 27.1 56 TS dt 0.5 time 27.6 57 TS dt 0.5 time 28.1 58 TS dt 0.5 time 28.6 59 TS dt 0.5 time 29.1 60 TS dt 0.5 time 29.6 61 TS dt 0. time 30.1 I don't see the 0.001 timestep here, do you get a different behavior? Thank you, Sophie ________________________________ From: Matthew Knepley > Sent: Tuesday, October 27, 2020 15:34 To: Blondel, Sophie > Cc: petsc-users at mcs.anl.gov >; xolotl-psi-development at lists.sourceforge.net > Subject: Re: [petsc-users] TSSetEventHandler and TSSetPostEventIntervalStep [External Email] On Tue, Oct 27, 2020 at 3:09 PM Blondel, Sophie via petsc-users > wrote: Hi, I am currently using TSSetEventHandler in my code to detect a random event where the solution vector gets modified during the event. Ideally, after the event happens I want the solver to use a much smaller timestep using TSSetPostEventIntervalStep. However, when I use TSSetPostEventIntervalStep the solver doesn't use the set value. I managed to reproduce the behavior by modifying ex40.c as attached. I stepped through ex40, and it does indeed change the timestep to 0.001. Can you be more specific, perhaps with monitors, about what you think is wrong? Thanks, Matt I think the issue is related to the fact that the fvalue is not technically "approaching" 0 with a random event, it is more of a step function instead. Do you have any recommendation on how to implement the behavior I'm looking for? Let me know if I can provide additional information. Best, Sophie -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jan 19 11:39:50 2021 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 19 Jan 2021 12:39:50 -0500 Subject: [petsc-users] Kokkos with OpenMP In-Reply-To: References: <8262DB88-AFC1-49FE-884D-6DCE21BB9C8D@petsc.dev> <06C45F26-AC0D-46CE-8182-129077AF83B9@petsc.dev> Message-ID: That would be great and early Feb would be perfect. Not having LU in Kokkos is not perfect but having a GPU ILU would be much much better than the current CPU LU. I am now suspecting that I am not running the job correctly for OMP at NERSc. I am going to read the docs more carefully. On Tue, Jan 19, 2021 at 10:27 AM Junchao Zhang wrote: > Mark, > I hope I can take up the Kokkos work in early February once we have the > ecp-tpds paper done. > --Junchao Zhang > > > On Mon, Jan 18, 2021 at 8:16 AM Mark Adams wrote: > >> OK, this running and using OMP Fieldsplit/additive. >> >> Kokkos does not have any solvers hooked up yet, but I have added an LU >> interface that just uses AIJ's LU. >> >> Junchao, I asked Kokkos about adding LU and they said that it is on the >> todo list. Until they do that I can't use it for the paper, but if you want >> to work on integrating solvers into aijkokkos I can do that and test with >> GMRES/Jacobi. >> >> Thanks, >> Mark >> >> On Mon, Jan 18, 2021 at 1:00 AM Barry Smith wrote: >> >>> >>> Mark, >>> >>> Looks like you tried to do two things at once, you removed the fopenmp >>> and put back the --with-cuda. >>> >>> >>> >>> Like I said if you want PETSc to build OpenMP and CUDA you need to >>> remove this line (and some others) from kokkos.py >>> >>> >>> >>> if self.openmp.found + pthreadfound + self.cuda.found > 1: >>> raise RuntimeError("Kokkos only supports a single parallel system >>> during its configuration") >>> >>> >>> >>> On Jan 17, 2021, at 9:39 PM, Mark Adams wrote: >>> >>> >>>> >>>> Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to >>>> indicate how to compile for OpenMP but when ./configure checks the >>>> installed KokkosKernel include files it uses CXXPPFLAGS. (Since you >>>> provided the flag CXXOPTFLAGS ./configure never determines the flag to put >>>> into CXXPPFLAGS. >>>> >>>> It is more desirable if you do not provide -fPIC -fopenmp. >>>> ./configure will figure them out for you. >>>> >>> >>> :( >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tangqi at msu.edu Thu Jan 21 01:52:04 2021 From: tangqi at msu.edu (Tang, Qi) Date: Thu, 21 Jan 2021 07:52:04 +0000 Subject: [petsc-users] MatCreateSchurComplementPmat to create a Schur complement Message-ID: Hi, When I use the MatCreateSchurComplementPmat to create a Schur complement for my own usage, I do not understand the logic when preuse=MAT_INITIAL_MATRIX. If all the matrixes are not empty, the first thing it did is 809: MatDestroy(Spmat); So that means I cannot pass in an empty matrix. It works if I first do MatCreate a dummy matrix and then pass in. Is that just some inconsistency or is there something I misunderstood? Qi -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Thu Jan 21 06:09:11 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Thu, 21 Jan 2021 13:09:11 +0100 Subject: [petsc-users] Program spends a large amount of time in PetscStrcmp and PetscCheckPointer - Why? Message-ID: <6ee16cfa-c0b6-78c6-c57c-237912a0c245@ntnu.no> Hei, I noticed while benchmarking and profiling that my program spends ~2800 s in PetscStrcmp and ~2300 s in PetscCheckPointer (for a total runtime of 21000 s) when running it on four threads. I do not recall using those functions actively, especially for PetscStrcmp. The first function which is doing something useful (i.e. I know where it is in use) is zgemv_, with 574 s total runtime. Why are those functions called, and where are they necessary? Thanks! Regards, Roland Richter From jacob.fai at gmail.com Thu Jan 21 06:37:38 2021 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Thu, 21 Jan 2021 07:37:38 -0500 Subject: [petsc-users] Program spends a large amount of time in PetscStrcmp and PetscCheckPointer - Why? In-Reply-To: <6ee16cfa-c0b6-78c6-c57c-237912a0c245@ntnu.no> References: <6ee16cfa-c0b6-78c6-c57c-237912a0c245@ntnu.no> Message-ID: <6DEBC00C-CAB2-4436-9184-AC39BB7C4864@gmail.com> > PetscCheckPointer (for a total runtime > of 21000 s) when running it on four threads. I do not recall using those > functions actively, especially for PetscStrcmp. PetscCheckPointer() is one of a chain of sanity-check functions called in the preambles to most petsc functions ? see PetscValidIntPointer() for example ? when PETSc is configured with debugging turned on. You can turn this off even for debugging builds by running with the option: -check_pointer_intensity 0 PetscStrcmp() is similarly used in these preambles to check that two PETSc objects are of the same type (since the type is stored as a string), see PetscObjectTypeCompare(), usually called from PetscValidHeaderSpecific() or PetscCheckTypeNames(). Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Jan 21, 2021, at 07:09, Roland Richter wrote: > > Hei, > > I noticed while benchmarking and profiling that my program spends ~2800 > s in PetscStrcmp and ~2300 s in PetscCheckPointer (for a total runtime > of 21000 s) when running it on four threads. I do not recall using those > functions actively, especially for PetscStrcmp. The first function which > is doing something useful (i.e. I know where it is in use) is zgemv_, > with 574 s total runtime. Why are those functions called, and where are > they necessary? > > Thanks! > > Regards, > > Roland Richter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hqhviet at tohoku.ac.jp Thu Jan 21 06:52:51 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Thu, 21 Jan 2021 21:52:51 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations Message-ID: Hello Petsc developers and supporters, I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100. Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? Thank you very much for anything you can provide. Sincerely, Viet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jan 21 07:04:19 2021 From: jed at jedbrown.org (Jed Brown) Date: Thu, 21 Jan 2021 06:04:19 -0700 Subject: [petsc-users] Program spends a large amount of time in PetscStrcmp and PetscCheckPointer - Why? In-Reply-To: <6DEBC00C-CAB2-4436-9184-AC39BB7C4864@gmail.com> References: <6ee16cfa-c0b6-78c6-c57c-237912a0c245@ntnu.no> <6DEBC00C-CAB2-4436-9184-AC39BB7C4864@gmail.com> Message-ID: <87sg6ua1bg.fsf@jedbrown.org> Jacob Faibussowitsch writes: >> PetscCheckPointer (for a total runtime >> of 21000 s) when running it on four threads. I do not recall using those >> functions actively, especially for PetscStrcmp. > > PetscCheckPointer() is one of a chain of sanity-check functions called in the preambles to most petsc functions ? see PetscValidIntPointer() for example ? when PETSc is configured with debugging turned on. You can turn this off even for debugging builds by running with the option: > -check_pointer_intensity 0 Indeed, though you should really configure --with-debugging=0 when you care about performance (usually with a different value of PETSC_ARCH so you can switch back and forth easily). It makes a huge difference, and not just because of these aggressive debug-mode checks. There are big warning banners if you use -log_view (which has profiling information) with a debugging build. From knepley at gmail.com Thu Jan 21 08:29:58 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Jan 2021 09:29:58 -0500 Subject: [petsc-users] MatCreateSchurComplementPmat to create a Schur complement In-Reply-To: References: Message-ID: On Thu, Jan 21, 2021 at 2:52 AM Tang, Qi wrote: > Hi, > When I use the MatCreateSchurComplementPmat to create a Schur complement > for my own usage, I do not understand the logic > when preuse=MAT_INITIAL_MATRIX. If all the matrixes are not empty, the > first thing it did is > 809: MatDestroy(Spmat); > So that means I cannot pass in an empty matrix. It works if I first do > MatCreate a dummy matrix and then pass in. Is that just some inconsistency > or is there something I misunderstood? > Yes, it wants you to pass &Sp with Sp == NULL. Sorry this is not more clear. Thanks, Matt > Qi > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eijkhout at tacc.utexas.edu Thu Jan 21 09:30:25 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Thu, 21 Jan 2021 15:30:25 +0000 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: Message-ID: On , 2021Jan21, at 06:52, Viet H.Q.H. > wrote: The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 Your runtime is so short that you probably only measure a slight increase in overhead. How many processors and what was the problem size? Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tangqi at msu.edu Thu Jan 21 13:13:15 2021 From: tangqi at msu.edu (Tang, Qi) Date: Thu, 21 Jan 2021 19:13:15 +0000 Subject: [petsc-users] MatCreateSchurComplementPmat to create a Schur complement In-Reply-To: References: , Message-ID: Thanks, Matt. That works now. ________________________________ From: Matthew Knepley Sent: Thursday, January 21, 2021 9:29 AM To: Tang, Qi Cc: PETSc users list Subject: Re: [petsc-users] MatCreateSchurComplementPmat to create a Schur complement On Thu, Jan 21, 2021 at 2:52 AM Tang, Qi > wrote: Hi, When I use the MatCreateSchurComplementPmat to create a Schur complement for my own usage, I do not understand the logic when preuse=MAT_INITIAL_MATRIX. If all the matrixes are not empty, the first thing it did is 809: MatDestroy(Spmat); So that means I cannot pass in an empty matrix. It works if I first do MatCreate a dummy matrix and then pass in. Is that just some inconsistency or is there something I misunderstood? Yes, it wants you to pass &Sp with Sp == NULL. Sorry this is not more clear. Thanks, Matt Qi -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jan 21 20:20:40 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 21 Jan 2021 20:20:40 -0600 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: Message-ID: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. comb.c has /* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only The idea for this and much of the initial code is contributed by Victor Eijkhout. Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *); Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */ #include /*I "petscvec.h" I*/ static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr; PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); } So first check if $PETSC_DIR/include/petscconf.h has PETSC_HAVE_MPI_IALLREDUCE if it does not then the standard MPI reduce is called. If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. Barry > On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: > > > Hello Petsc developers and supporters, > > I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. > > > PetscLogDouble tt1,tt2; > KSP ksp; > //ierr = VecSet(c,one); > ierr = VecSet(c,one); > ierr = VecSet(u,one); > ierr = VecSet(b,one); > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); > ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); > > > ierr = PetscTime(&tt1);CHKERRQ(ierr); > ierr = VecNormBegin(u,NORM_2,&norm1); > ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); > ierr = KSP_MatMult(ksp,A,c,Ac); > ierr = VecNormEnd(u,NORM_2,&norm1); > ierr = PetscTime(&tt2);CHKERRQ(ierr); > > ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); > > > ierr = PetscTime(&tt1);CHKERRQ(ierr); > ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); > ierr = KSP_MatMult(ksp,A,c,Ac); > ierr = PetscTime(&tt2);CHKERRQ(ierr); > > ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); > > > On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. > > The time used for the asynchronous calculation: 0.000203 > + |u| = 100. > The time used for the synchronous calculation: 0.000006 > + |b| = 100. > > Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? > > Thank you very much for anything you can provide. > Sincerely, > Viet. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hqhviet at tohoku.ac.jp Fri Jan 22 11:20:15 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Sat, 23 Jan 2021 02:20:15 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> Message-ID: Dear Victor and Berry, Thank you so much for your answers. I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000. Asynchronous computation still takes a longer time. I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h I built Petsc by using the following script #!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc" cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install Intel 2018 also complies with the MPI-3 standard. Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? Sincerely, Viet. On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: > > ierr = VecNormBegin(u,NORM_2,&norm1); > ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); > > How come you call this on Ax and not on u? For clarity, if nothing else, I > think you should call it on u. > > comb.c has > > /* > Split phase global vector reductions with support for combining the > communication portion of several operations. Using MPI-1.1 support only > > The idea for this and much of the initial code is contributed by > Victor Eijkhout. > > Usage: > VecDotBegin(Vec,Vec,PetscScalar *); > VecNormBegin(Vec,NormType,PetscReal *); > .... > VecDotEnd(Vec,Vec,PetscScalar *); > VecNormEnd(Vec,NormType,PetscReal *); > > Limitations: > - The order of the xxxEnd() functions MUST be in the same order > as the xxxBegin(). There is extensive error checking to try to > insure that the user calls the routines in the correct order > */ > > #include /*I "petscvec.h" I*/ > > static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void > *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm > comm,MPI_Request *request) > { > PETSC_UNUSED PetscErrorCode ierr; > > PetscFunctionBegin; > #if defined(PETSC_HAVE_MPI_IALLREDUCE) > ierr = > MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); > #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) > ierr = > MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); > #else > ierr = > MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); > *request = MPI_REQUEST_NULL; > #endif > PetscFunctionReturn(0); > } > > > So first check if $PETSC_DIR/include/petscconf.h has > > PETSC_HAVE_MPI_IALLREDUCE > > if it does not then the standard MPI reduce is called. > > If this is set then any improvement depends on the implementation of > iallreduce inside the MPI you are using. > > Barry > > > On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: > > > Hello Petsc developers and supporters, > > I would like to confirm the performance of asynchronous computations of > inner product computation overlapping with matrix-vector multiplication > computation by the below code. > > > PetscLogDouble tt1,tt2; > KSP ksp; > //ierr = VecSet(c,one); > ierr = VecSet(c,one); > ierr = VecSet(u,one); > ierr = VecSet(b,one); > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); > ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); > > > ierr = PetscTime(&tt1);CHKERRQ(ierr); > ierr = VecNormBegin(u,NORM_2,&norm1); > ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); > ierr = KSP_MatMult(ksp,A,c,Ac); > ierr = VecNormEnd(u,NORM_2,&norm1); > ierr = PetscTime(&tt2);CHKERRQ(ierr); > > ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the > asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); > CHKERRQ(ierr); > > > ierr = PetscTime(&tt1);CHKERRQ(ierr); > ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); > ierr = KSP_MatMult(ksp,A,c,Ac); > ierr = PetscTime(&tt2);CHKERRQ(ierr); > > ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the > synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); > CHKERRQ(ierr); > > > On a cluster with two or four nodes, the asynchronous computation is > always much slower than synchronous computation. > > The time used for the asynchronous calculation: 0.000203 > + |u| = 100. > The time used for the synchronous calculation: 0.000006 > + |b| = 100. > > Are there any necessary settings on MPI or Petsc to gain performance of > asynchronous computation? > > Thank you very much for anything you can provide. > Sincerely, > Viet. > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jan 22 12:01:08 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 22 Jan 2021 12:01:08 -0600 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> Message-ID: <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps. > On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. wrote: > > > Dear Victor and Berry, > > Thank you so much for your answers. > > I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. > > ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); > > It was also a mistake to set the vector size too small. > I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows > > The time used for the asynchronous calculation: 0.022043 > + | u | = 10000. > The time used for the synchronous calculation: 0.016188 > + | b | = 10000. > > Asynchronous computation still takes a longer time. > > I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h > > I built Petsc by using the following script > > #!/usr/bin/bash > set -e > DATE="21.01.18" > MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" > MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" > INSTL_DIR="${HOME}/local/petsc-3.14.3" > BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" > PETSC_DIR="${HOME}/tmp/petsc" > > cd ${PETSC_DIR} > ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' > > make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all > make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install > > > Intel 2018 also complies with the MPI-3 standard. > > Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? > > Sincerely, > Viet. > > > On Fri, Jan 22, 2021 at 11:20 AM Barry Smith > wrote: > > ierr = VecNormBegin(u,NORM_2,&norm1); > ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); > > How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. > > comb.c has > > /* > Split phase global vector reductions with support for combining the > communication portion of several operations. Using MPI-1.1 support only > > The idea for this and much of the initial code is contributed by > Victor Eijkhout. > > Usage: > VecDotBegin(Vec,Vec,PetscScalar *); > VecNormBegin(Vec,NormType,PetscReal *); > .... > VecDotEnd(Vec,Vec,PetscScalar *); > VecNormEnd(Vec,NormType,PetscReal *); > > Limitations: > - The order of the xxxEnd() functions MUST be in the same order > as the xxxBegin(). There is extensive error checking to try to > insure that the user calls the routines in the correct order > */ > > #include /*I "petscvec.h" I*/ > > static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) > { > PETSC_UNUSED PetscErrorCode ierr; > > PetscFunctionBegin; > #if defined(PETSC_HAVE_MPI_IALLREDUCE) > ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); > #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) > ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); > #else > ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); > *request = MPI_REQUEST_NULL; > #endif > PetscFunctionReturn(0); > } > > > So first check if $PETSC_DIR/include/petscconf.h has > > PETSC_HAVE_MPI_IALLREDUCE > > if it does not then the standard MPI reduce is called. > > If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. > > Barry > > >> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. > wrote: >> >> >> Hello Petsc developers and supporters, >> >> I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. >> >> >> PetscLogDouble tt1,tt2; >> KSP ksp; >> //ierr = VecSet(c,one); >> ierr = VecSet(c,one); >> ierr = VecSet(u,one); >> ierr = VecSet(b,one); >> >> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >> >> >> ierr = PetscTime(&tt1);CHKERRQ(ierr); >> ierr = VecNormBegin(u,NORM_2,&norm1); >> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >> ierr = KSP_MatMult(ksp,A,c,Ac); >> ierr = VecNormEnd(u,NORM_2,&norm1); >> ierr = PetscTime(&tt2);CHKERRQ(ierr); >> >> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); >> >> >> ierr = PetscTime(&tt1);CHKERRQ(ierr); >> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >> ierr = KSP_MatMult(ksp,A,c,Ac); >> ierr = PetscTime(&tt2);CHKERRQ(ierr); >> >> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); >> >> >> On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. >> >> The time used for the asynchronous calculation: 0.000203 >> + |u| = 100. >> The time used for the synchronous calculation: 0.000006 >> + |b| = 100. >> >> Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? >> >> Thank you very much for anything you can provide. >> Sincerely, >> Viet. >> >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sajidsyed2021 at u.northwestern.edu Sat Jan 23 18:45:21 2021 From: sajidsyed2021 at u.northwestern.edu (Sajid Ali) Date: Sat, 23 Jan 2021 18:45:21 -0600 Subject: [petsc-users] Convert a 3D DMDA sub-vector to a natural 2D vector Message-ID: Hi PETSc-developers, For an application, I'd like to extract a 2D slice from a 3D DMDA vector, perform a `MatMult` on it (a discretized rotation on the 2D vector) and place the resulting vector back into the 3D vector. The approach I'd taken was to use `DMDACreatePatchIS` to create an IS that selects the slice (say this slice is [:,:,n_z] with no loss of generality). This IS is then used to extract the 2D slice via the `VecGetSubVector` utility. The issue that arises from this scheme is that the extracted 2D vector does not represent a flattened 2D vector, it is instead ordered as a DMDA vector (aka each rank having the portion of vector it owns in a column major ordering). How do I convert this DMDA ordered 2D subvector to a vector that represents a flattened 2D array? Since I can't predict how the extracted sub-vector While PETSc provides `DMDAGlobalToNatural` routines, those don't apply to extracted sub-vectors and extracting the 2D slice from a natural vector (filled by a scatter from the global vector) does not work either. On the other hand, converting the output of DMDACreatePatchIS to a natural IS via `AOPetscToApplicationIS` before extracting a subvector extracts the wrong indices from the 3D vector. Could someone tell me how I can achieve the desired goal (converting a 2D vector ordered as per a 3D DMDA grid onto a natural flattened index) ? Thanks in advance for the help! PS : Should it help, I'm attaching a slightly modified version of `src/dm/tests/ex53.c` to demonstrate the issue. The array I wish to obtain (the [:,:,2]) slice is : `[196., 194., 192., 190., 198., 197., 196., 195., 200., 200., 200.,200., 202., 203., 204., 205.]` (a row-major flattening of the 2D slice or alternatively a column-major flattening). What I obtain from the program (by creating the PatchIS and extracting the subvector) instead is : `[0 (A)] 196,198,194,197 [1 (B)] 200,202,200,203 [2 (C)] 192,196,190,195 [3 (D)] 200,204,200,205` (where each square bracket represents the rank and portion of the 2D vector being flattened via a column-major formatting with the layout shown below) |---------| | A | C | | B | D | |---------| Thank You, Sajid Ali (he/him) | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex53.c Type: application/octet-stream Size: 7275 bytes Desc: not available URL: From rlmackie862 at gmail.com Sat Jan 23 20:09:41 2021 From: rlmackie862 at gmail.com (Randall Mackie) Date: Sat, 23 Jan 2021 18:09:41 -0800 Subject: [petsc-users] Convert a 3D DMDA sub-vector to a natural 2D vector In-Reply-To: References: Message-ID: <90FF5B38-852E-4BF9-B4FA-01823A4FF02F@gmail.com> Several years ago I asked a similar question and this is what Barry told me: >>>>>>>>>> Randy, Take a look at DMDAGetRay() in src/dm/impls/da/dasub.c (now called DMDACreateRay()) this takes a row or column from a 2d DADM. You can use the same kind of approach to get a slice from a 3d DMDA. Barry <<<<<<<<<< Maybe this would help you? Randy > On Jan 23, 2021, at 4:45 PM, Sajid Ali wrote: > > Hi PETSc-developers, > > For an application, I'd like to extract a 2D slice from a 3D DMDA vector, perform a `MatMult` on it (a discretized rotation on the 2D vector) and place the resulting vector back into the 3D vector. The approach I'd taken was to use `DMDACreatePatchIS` to create an IS that selects the slice (say this slice is [:,:,n_z] with no loss of generality). This IS is then used to extract the 2D slice via the `VecGetSubVector` utility. The issue that arises from this scheme is that the extracted 2D vector does not represent a flattened 2D vector, it is instead ordered as a DMDA vector (aka each rank having the portion of vector it owns in a column major ordering). How do I convert this DMDA ordered 2D subvector to a vector that represents a flattened 2D array? Since I can't predict how the extracted sub-vector > > While PETSc provides `DMDAGlobalToNatural` routines, those don't apply to extracted sub-vectors and extracting the 2D slice from a natural vector (filled by a scatter from the global vector) does not work either. On the other hand, converting the output of DMDACreatePatchIS to a natural IS via `AOPetscToApplicationIS` before extracting a subvector extracts the wrong indices from the 3D vector. > > Could someone tell me how I can achieve the desired goal (converting a 2D vector ordered as per a 3D DMDA grid onto a natural flattened index) ? Thanks in advance for the help! > > PS : Should it help, I'm attaching a slightly modified version of `src/dm/tests/ex53.c` to demonstrate the issue. The array I wish to obtain (the [:,:,2]) slice is : `[196., 194., 192., 190., 198., 197., 196., 195., 200., 200., 200.,200., 202., 203., 204., 205.]` (a row-major flattening of the 2D slice or alternatively a column-major flattening). What I obtain from the program (by creating the PatchIS and extracting the subvector) instead is : `[0 (A)] 196,198,194,197 [1 (B)] 200,202,200,203 [2 (C)] 192,196,190,195 [3 (D)] 200,204,200,205` (where each square bracket represents the rank and portion of the 2D vector being flattened via a column-major formatting with the layout shown below) > |---------| > | A | C | > | B | D | > |---------| > > > Thank You, > Sajid Ali (he/him) | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Sun Jan 24 10:02:44 2021 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Sun, 24 Jan 2021 17:02:44 +0100 Subject: [petsc-users] DMPlex with dof in cells AND at nodes In-Reply-To: References: Message-ID: Hi everyone, I have continued working on my problem, and I have made some progress ... although it is not working in parallel as well as I would like yet. I have troubles with the normals at the faces : it appears that for a given mesh, the normals at the faces depend on the partitioning, especially for those faces that belong to the boundary between two MPI-proc domains. If I read correctly everything I printed out to try and understand, it seems that those faces have normals pointing outside of the MPI-proc domain, no matter what their normal was in the sequential version. My problem is that my algorithms heavily rely on those normals, and they have to have consistent orientations ... how can I recover for a given face belonging to a given MPI-process the normal it would have had in sequential ? Thank you very much in advance !! Thibault Le mer. 13 janv. 2021 ? 18:36, Thibault Bridel-Bertomeu < thibault.bridelbertomeu at gmail.com> a ?crit : > Hello Matt, > > Thank you for the follow up. > Here are the print out for the two DMs as you suggested (the smallest > tetrahedral mesh possible in a cube). > > :: [DEBUG] Visualizing DM in console :: > > DM Object: 2 MPI processes > > type: plex > > DM_0x7faa62d416f0_1 in 3 dimensions: > > 0-cells: 14 14 > > 1-cells: 49 49 > > 2-cells: 60 60 > > 3-cells: 36 (12) [12] 36 (12) [12] > > Labels: > > depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (36)) > > celltype: 5 strata with value/size (0 (14), 1 (49), 3 (60), 6 (24), 12 > (12)) > > Cell Sets: 1 strata with value/size (7 (24)) > > Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), > 6 (4)) > > vtk: 1 strata with value/size (1 (12)) > > ghost: 2 strata with value/size (2 (12), 1 (27)) > > > :: [DEBUG] Visualizing DM_nodal in console :: > > DM Object: 2 MPI processes > > type: plex > > DM_0x7faa62d416f0_3 in 3 dimensions: > > 0-cells: 14 14 > > 1-cells: 49 49 > > 2-cells: 60 60 > > 3-cells: 36 (12) [12] 36 (12) [12] > > Labels: > > depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (36)) > > celltype: 5 strata with value/size (0 (14), 1 (49), 3 (60), 6 (24), 12 > (12)) > > Cell Sets: 1 strata with value/size (7 (24)) > > Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), > 6 (4)) > > vtk: 1 strata with value/size (1 (12)) > > ghost: 2 strata with value/size (2 (12), 1 (27)) > > As you predicted, the sum of the numbers of local cells is greater than > the total number of cells. > > I was doing some more digging, and it appears that I have issues because > the normals of some faces attached to the vertices are not populated. > In other words, "call DMPlexGetGeometryFVM(dm, facegeom, cellgeom, > minradius, ierr); CHKERRA(ierr)" yields a facegeom Vec that is a local Vec. > For the mesh that are described above, that Vec has a global size of 720 > (12 * 60 faces), and also a local size of 720 (12 * 60) because (I think .. > ?) the mesh is so small, with the distribution chosen, both processes know > the existence of all the entities (vertices, edges, faces & cells). > However, for, say MPI proc no.1, a lot of places in this Vec are filled > with zeros. As I use the normal normalized by the face area, I get NaNs > because I am trying to divide by zero. > Now the question is, a vertex on a given MPI proc knows that it has, say, > 6 faces attached, and I can get that number along with the indices of those > faces with "call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, > vertexclosure, ierr); CHKERRA(ierr)", but then when I fetch the face > characteristics in the facegeom Vec, some are not populated (full of > zeros). Is this the consequence of what you were explaining, "only the > point closure is guaranteed when distributing" ? > Or is there a way to do some equivalent to GlobalToLocal for that facegeom > Vec ? > > Thank you very much for your support, > > Thibault > > Le mer. 13 janv. 2021 ? 15:00, Matthew Knepley a > ?crit : > >> On Tue, Jan 12, 2021 at 4:12 PM Thibault Bridel-Bertomeu < >> thibault.bridelbertomeu at gmail.com> wrote: >> >>> Good evening Matthew, >>> >>> Thank you for your answer, and sorry for the delay in mine ! I am trying >>> to figure things out but when one gets in the distribution of a DMPlex, it >>> becomes quite complex! >>> >>> Le lun. 11 janv. 2021 ? 16:24, Matthew Knepley a >>> ?crit : >>> >>>> On Mon, Jan 11, 2021 at 6:41 AM Thibault Bridel-Bertomeu < >>>> thibault.bridelbertomeu at gmail.com> wrote: >>>> >>>>> Dear all, >>>>> >>>>> I haven't been on this mailing list since december, so first of all, >>>>> happy new year to everyone ! >>>>> >>>>> I am facing another challenge with the DMPlex structure : although I >>>>> am doing cell-centered finite volume (using PetscFV), I also need to do >>>>> some manipulations and store information that are vertex-based. >>>>> The problem I have is when I run the code in parallel : when I >>>>> visualize the nodal data using a vtk viewer, it shows NaN at all the >>>>> vertices in close proximity to the MPI partition line. >>>>> Here is how I proceed to work on the vertex-based data : >>>>> >>>>> First comes the creation of the DM that I'll be using for the >>>>> cell-centered FVM >>>>> >>>>> call DMPlexCreateFromFile(comm, name, PETSC_TRUE, dm, ierr) >>>>> call DMSetBasicAdjacency(dm, PETSC_TRUE, PETSC_TRUE, ierr) >>>>> call DMPlexDistribute(dm, 1, PETSC_NULL_SF, dmDist, ierr) >>>>> ... >>>>> dm = dmDist >>>>> ... >>>>> call DMConstructGhostCells(dm, ..., ..., dmGhost, ierr) >>>>> ... >>>>> dm = dmGhost >>>>> ... >>>>> >>>>> I set both boolean to PETSC_TRUE in the SetBasicAdjacency so even in >>>>> MPI the neighbouring relationships are conserved for all strata. Is that >>>>> true ? Like, does doing this actually enable to get all the transitive >>>>> closure of a given vertex, even if part of it is in another MPI block ? >>>>> >>>> >>>> What you say should be true, but it is not. When I designed everything, >>>> I was very much stuck in the traditional mesh way of looking at things, and >>>> made some limiting decisions. You can still get what you want (almost), but >>>> you need to distribute the mesh with an overlap of 1. If you do, then you >>>> can get the transitive support of vertices. >>>> >>> >>> I actually already distribute the mesh with an overlap of 1, but the >>> result is the same. Do I need to re-distribute the clone "dmNodal" somehow >>> ? Is the cloning process maybe not preserving the MPI support of the >>> different strata ? >>> >> >> No. However, it is easy to check what you have using -dm_view. I would >> encourage you to put >> >> DMViewFromOptions(dm, NULL, "-dm_view"); >> DMViewFromOptions(dmNodal, NULL, "-nodal_dm_view"); >> >> so that you can give those options and send the output (for a very simple >> mesh) with the emails. What you should >> see is that for both meshes, the number of local cells adds up to more >> than the number of global cells. >> >> >>> The problem is basically that many places in the code assume that the >>>> overlap always contains the closure of any point in it. It is difficult to >>>> be able to switch this assumption to the star (transitive support)so I have >>>> not done that yet. >>>> >>> >>>> >>>>> Then I need a vertex-based DM, so I figure it's pretty much what's >>>>> done in CreateMassMatrix in TS ex11.c. >>>>> >>>>> call DMClone(dm, dmNodal, ierr) >>>>> call DMGetCoordinateSection(dm, coordSection, ierr); CHKERRA(ierr) >>>>> call DMGetCoordinatesLocal(dm, coordinates, ierr); CHKERRA(ierr) >>>>> call DMSetCoordinateSection(dmNodal, PETSC_DETERMINE, coordSection, >>>>> ierr); >>>>> call PetscSectionCreate(PETSC_COMM_WORLD, sectionNodal, ierr); >>>>> CHKERRA(ierr) >>>>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr); CHKERRA(ierr) >>>>> call PetscSectionSetChart(sectionNodal, vstart, vend, ierr); >>>>> CHKERRA(ierr) >>>>> do inode = vstart, vend-1 >>>>> ! 3 dof per node, ndim = 3 >>>>> call PetscSectionSetDof(sectionNodal, inode, ndim, ierr); CHKERRA(ierr) >>>>> end do >>>>> call PetscSectionSetUp(sectionNodal, ierr); CHKERRA(ierr) >>>>> call DMSetLocalSection(dmNodal, sectionNodal, ierr); CHKERRA(ierr) >>>>> call PetscSectionDestroy(sectionNodal, ierr); CHKERRA(ierr) >>>>> >>>>> From what I understood, this new dmNodal basically has all the >>>>> characteristics of the ori dm, except it's primary layer is vertex-based >>>>> with 3 DOF per vertex. >>>>> >>>> >>>> Take out the coordinate stuff above. This is now handled internally in >>>> a better way. Other than that, it looks good. >>>> >>> >>> You mean the SetCoordinateSection is already handled by the cloning >>> process ? >>> >> >> Yes. >> >> >>> >>>> >>>>> Then I get a local vector and work on it : >>>>> >>>>> call DMGetLocalVector(dmNodal, nodalvec, ierr) >>>>> call VecGetArrayF90(nodalvec, nodalvecarray, ierr) >>>>> call DMPlexGetDepthStratum(dm, 0, vstart, vend, ierr) >>>>> call DMPlexGetDepthStratum(dm, 2, fstart, fend, ierr) >>>>> ! For each node ... >>>>> do inode = vstart, vend-1 >>>>> A = 0 ! matrix, ndim x ndim >>>>> b = 0 ! vector, ndim >>>>> call DMPlexGetTransitiveClosure(dmNodal, inode, PETSC_FALSE, >>>>> vertexclosure, ierr); >>>>> ! For each face attached to that node >>>>> do iface = 1, size(vertexclosure) >>>>> if ((vertexclosure(iface) < fstart) .or. (vertexclosure(iface) >= >>>>> fend)) cycle >>>>> A = A + ... >>>>> b = b + ... >>>>> end do >>>>> nodalvecarray(1+(inode-vstart)*ndim:3+(inode-vstart)*ndim) = matmul(A, >>>>> b) >>>>> ... >>>>> end do >>>>> >>>> >>>> There are some subtleties here I think. You are looping over all >>>> vertices, but you only own some of them. >>>> >>> >>> Here I am quite surprised by your remark : from the random print I did, >>> the DMPlexGetDepthStratum returns different vstart and vend for different >>> MPI processes, hence I assumed it was a local numbering. >>> >> >> It is a local numbering. If you have a mesh partition, it is usually to >> have vertices present in multiple local meshes. We think of one process as >> "owning" the vertex. That is what I meant here. >> >> >>> And then I assumed that, with the DMPlexDistribute, it ensured that the >>> "vertexclosure" >>> >> >> This is the wrong terminology. Closures mean including the boundary of >> something. Vertices have no boundary. You want the dual notion "support", >> or its transitive closure "star". So is the vertex star included when >> distributing? No, only the point closure is guaranteed when distributing. >> This was the point of my explanation. However, if you request overlap = 1, >> then you get everything in the star of the original boundary. >> >> >>> contained (among others) all the faces connected to each "inode", even >>> if the closure is spread among multiple MPI processes. So you mean that all >>> these assumptions are actually wrong to some degree ? >>> >>> >>>> What do you do when putting these values in the global vector? I think >>>> there are at least three choices: >>>> >>>> 1) Everyone has everything, so you compute all the vertex stuff >>>> redundantly, and then use INSERT_VALUES for the scatter >>>> >>>> 2) Only vertices are redundant, so the transitive supports of vertices >>>> are partial but non-overlapping, and you use ADD_VALUES for the scatter >>>> >>>> 3) Only cells are redundant, so you do the complete vertex computation >>>> only for local vertices >>>> >>> >>> I distributed with overlap = 1, so I expect I have a halo of redundant >>> cells in each MPI block (option no. 3). >>> >> >> Yes. >> >> >>> With the (..., PETSC_TRUE, PETSC_TRUE, ...) setting of the distribute >>> however, I also expected to have a halo of vertices and a halo of edges and >>> a halo of faces. Therefore, completing the vertex computation only for >>> local vertices should have been fine since anyways even local vertices know >>> their complete closure in terms of faces, >>> >> >> Yes, this should work. >> >> >>> but I am missing something because it does not turn out right ... >>> >> >> Start with a very simple mesh, say 2 cells or 4 cells. Then you can print >> everything out. >> >> Thanks, >> >> Matt >> >> >>> >>>> >>>>> Then I want to visualize it : >>>>> >>>>> call VecRestoreArrayF90(nodalvec, nodalvecarray, ierr) >>>>> call DMCreateGlobalVector(dmNodal, globalnodalvec, ierr) >>>>> call DMLocalToGlobal(dmNodal, nodalvec, INSERT_VALUES, globalnodalvec, >>>>> ierr) >>>>> call PetscViewerCreate(PETSC_COMM_WORLD, vtkViewer, ierr); >>>>> CHKERRA(ierr) >>>>> call PetscViewerSetType(vtkViewer, PETSCVIEWERVTK, ierr); CHKERRA(ierr) >>>>> call PetscViewerFileSetName(vtkViewer, "toto.vtk", ierr); CHKERRA(ierr) >>>>> call VecView(globalnodalvec, vtkViewer, ierr); CHKERRA(ierr) >>>>> call PetscViewerDestroy(vtkViewer, ierr); CHKERRA(ierr) >>>>> >>>>> In the toto.vtk file, I get vertex-based values as expected, and the >>>>> whole shebang works in sequential. Now when I run it in parallel, I get >>>>> vertex-based values inside the MPI blocks, but in the neighbourhood of the >>>>> join between the MPI blocks, all the vertex-based values are NaN (see >>>>> attachment). >>>>> >>>> >>>> If you have redundant cells, meaning you distributed with overlap = 1, >>>> did you remember to full up the values for those ghost cells using >>>> DMGlobalToLocal() or something like that? If not, they will start as >>>> NaNs. >>>> >>> >>> Indeed, in the vertex computation, at some point I use the values in the >>> cells. But I got the solution vector from TSGetSolution, then I do a >>> GlobalToLocal, then I even add a InsertBoundaryValues to fill the ghost >>> cells correctly. >>> Even by doing this, when I print the nodalvecarray on each process, they >>> somehow contain some NaNs. This means, I think, that at some point, a given >>> vertex tries to get the information of a face that is supposed to be in its >>> closure but is actually not .. (?) >>> >>> Thank you for your help and feedback ! >>> >>> Cheers, >>> >>> Thibault >>> >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> All in all, I think I do not understand fully the consequences of the >>>>> DMPlexDistribute with useCone=true & useClosure=true ... but I cannot >>>>> figure out where the NaN are coming from : is it because the vertices do >>>>> not actually know all their closure through the MPI join, or is it some >>>>> Global / Local error I made ? >>>>> >>>>> Thank you very much in advance for your support !! >>>>> >>>>> Thibault >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hqhviet at tohoku.ac.jp Mon Jan 25 02:17:51 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Mon, 25 Jan 2021 17:17:51 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> Message-ID: Dear Barry, Thank you very much for your information. It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI. $ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet. On Sat, Jan 23, 2021 at 3:01 AM Barry Smith wrote: > > > https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html > > It states "and a partial support for non-blocking collectives ( MPI_Ibcas > t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial > support means but you can try setting the variables and see if that helps. > > > > On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. wrote: > > > Dear Victor and Berry, > > Thank you so much for your answers. > > I fixed the code with the bug in the PetscCommSplitReductionBegin function > as commented by Brave. > > ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) > u)); > > It was also a mistake to set the vector size too small. > I just set a vector size of 100000000 and ran the code on 4 nodes with 2 > processors per node. The result is as follows > > The time used for the asynchronous calculation: 0.022043 > + | u | = 10000. > The time used for the synchronous calculation: 0.016188 > + | b | = 10000. > > Asynchronous computation still takes a longer time. > > I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the > file $PETSC_DIR/include/petscconf.h > > I built Petsc by using the following script > > #!/usr/bin/bash > set -e > DATE="21.01.18" > > MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" > > MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" > INSTL_DIR="${HOME}/local/petsc-3.14.3" > BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" > PETSC_DIR="${HOME}/tmp/petsc" > > cd ${PETSC_DIR} > ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} > --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec > --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} > --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native > -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 > -march=native -mtune=native' > > make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all > make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install > > > Intel 2018 also complies with the MPI-3 standard. > > Are there specific settings for Intel MPI to obtain the performance of the > MPI_IALLREDUCE function? > > Sincerely, > Viet. > > > On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: > >> >> ierr = VecNormBegin(u,NORM_2,&norm1); >> ierr = >> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >> >> How come you call this on Ax and not on u? For clarity, if nothing else, >> I think you should call it on u. >> >> comb.c has >> >> /* >> Split phase global vector reductions with support for combining the >> communication portion of several operations. Using MPI-1.1 support only >> >> The idea for this and much of the initial code is contributed by >> Victor Eijkhout. >> >> Usage: >> VecDotBegin(Vec,Vec,PetscScalar *); >> VecNormBegin(Vec,NormType,PetscReal *); >> .... >> VecDotEnd(Vec,Vec,PetscScalar *); >> VecNormEnd(Vec,NormType,PetscReal *); >> >> Limitations: >> - The order of the xxxEnd() functions MUST be in the same order >> as the xxxBegin(). There is extensive error checking to try to >> insure that the user calls the routines in the correct order >> */ >> >> #include /*I "petscvec.h" I*/ >> >> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void >> *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm >> comm,MPI_Request *request) >> { >> PETSC_UNUSED PetscErrorCode ierr; >> >> PetscFunctionBegin; >> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >> ierr = >> MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >> ierr = >> MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >> #else >> ierr = >> MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >> *request = MPI_REQUEST_NULL; >> #endif >> PetscFunctionReturn(0); >> } >> >> >> So first check if $PETSC_DIR/include/petscconf.h has >> >> PETSC_HAVE_MPI_IALLREDUCE >> >> if it does not then the standard MPI reduce is called. >> >> If this is set then any improvement depends on the implementation of >> iallreduce inside the MPI you are using. >> >> Barry >> >> >> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: >> >> >> Hello Petsc developers and supporters, >> >> I would like to confirm the performance of asynchronous computations of >> inner product computation overlapping with matrix-vector multiplication >> computation by the below code. >> >> >> PetscLogDouble tt1,tt2; >> KSP ksp; >> //ierr = VecSet(c,one); >> ierr = VecSet(c,one); >> ierr = VecSet(u,one); >> ierr = VecSet(b,one); >> >> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >> >> >> ierr = PetscTime(&tt1);CHKERRQ(ierr); >> ierr = VecNormBegin(u,NORM_2,&norm1); >> ierr = >> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >> ierr = KSP_MatMult(ksp,A,c,Ac); >> ierr = VecNormEnd(u,NORM_2,&norm1); >> ierr = PetscTime(&tt2);CHKERRQ(ierr); >> >> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >> asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); >> CHKERRQ(ierr); >> >> >> ierr = PetscTime(&tt1);CHKERRQ(ierr); >> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >> ierr = KSP_MatMult(ksp,A,c,Ac); >> ierr = PetscTime(&tt2);CHKERRQ(ierr); >> >> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >> synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); >> CHKERRQ(ierr); >> >> >> On a cluster with two or four nodes, the asynchronous computation is >> always much slower than synchronous computation. >> >> The time used for the asynchronous calculation: 0.000203 >> + |u| = 100. >> The time used for the synchronous calculation: 0.000006 >> + |b| = 100. >> >> Are there any necessary settings on MPI or Petsc to gain performance of >> asynchronous computation? >> >> Thank you very much for anything you can provide. >> Sincerely, >> Viet. >> >> >> >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland.richter at ntnu.no Mon Jan 25 03:40:24 2021 From: roland.richter at ntnu.no (Roland Richter) Date: Mon, 25 Jan 2021 10:40:24 +0100 Subject: [petsc-users] Gathering distributed dense matrix only on rank 0 Message-ID: <0fdad010-6d01-0550-e958-d77f21a70d05@ntnu.no> Hei, is there a way to gather a distributed dense matrix only on rank 0, not on the other ranks? My matrix processing/storage routines in my program currently are single-threaded and only operate on rank 0, and therefore I assume that I can ignore all other ranks. This should also save a bit time, I hope (MatCreateRedundantMatrix is currently the by far most expensive part of my program). When executing MatCreateRedundantMatrix(A, size, MPI_COMM_NULL, MAT_INITIAL_MATRIX, &B), usually all ranks get the full matrices, something I would like to avoid. Therefore, is there another way of handling this? Thanks! Regards, Roland Richter From patrick.sanan at gmail.com Mon Jan 25 04:47:20 2021 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Mon, 25 Jan 2021 11:47:20 +0100 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> Message-ID: <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Sorry about the delay in responding, but I'll add a couple of points here: 1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case). 2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation. For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system). I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right. > Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. : > > > Dear Barry, > > Thank you very much for your information. > > It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI. > > $ export I_MPI_ASYNC_PROGRESS = 1 > $ export I_MPI_ASYNC_PROGRESS_PIN = > > https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ > > I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. > Best, > Viet. > > > > > On Sat, Jan 23, 2021 at 3:01 AM Barry Smith > wrote: > > https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html > > It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps. > > > >> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. > wrote: >> >> >> Dear Victor and Berry, >> >> Thank you so much for your answers. >> >> I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. >> >> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); >> >> It was also a mistake to set the vector size too small. >> I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows >> >> The time used for the asynchronous calculation: 0.022043 >> + | u | = 10000. >> The time used for the synchronous calculation: 0.016188 >> + | b | = 10000. >> >> Asynchronous computation still takes a longer time. >> >> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h >> >> I built Petsc by using the following script >> >> #!/usr/bin/bash >> set -e >> DATE="21.01.18" >> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >> INSTL_DIR="${HOME}/local/petsc-3.14.3" >> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >> PETSC_DIR="${HOME}/tmp/petsc" >> >> cd ${PETSC_DIR} >> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' >> >> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >> >> >> Intel 2018 also complies with the MPI-3 standard. >> >> Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? >> >> Sincerely, >> Viet. >> >> >> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith > wrote: >> >> ierr = VecNormBegin(u,NORM_2,&norm1); >> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >> >> How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. >> >> comb.c has >> >> /* >> Split phase global vector reductions with support for combining the >> communication portion of several operations. Using MPI-1.1 support only >> >> The idea for this and much of the initial code is contributed by >> Victor Eijkhout. >> >> Usage: >> VecDotBegin(Vec,Vec,PetscScalar *); >> VecNormBegin(Vec,NormType,PetscReal *); >> .... >> VecDotEnd(Vec,Vec,PetscScalar *); >> VecNormEnd(Vec,NormType,PetscReal *); >> >> Limitations: >> - The order of the xxxEnd() functions MUST be in the same order >> as the xxxBegin(). There is extensive error checking to try to >> insure that the user calls the routines in the correct order >> */ >> >> #include /*I "petscvec.h" I*/ >> >> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) >> { >> PETSC_UNUSED PetscErrorCode ierr; >> >> PetscFunctionBegin; >> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >> ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >> ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >> #else >> ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >> *request = MPI_REQUEST_NULL; >> #endif >> PetscFunctionReturn(0); >> } >> >> >> So first check if $PETSC_DIR/include/petscconf.h has >> >> PETSC_HAVE_MPI_IALLREDUCE >> >> if it does not then the standard MPI reduce is called. >> >> If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. >> >> Barry >> >> >>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. > wrote: >>> >>> >>> Hello Petsc developers and supporters, >>> >>> I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. >>> >>> >>> PetscLogDouble tt1,tt2; >>> KSP ksp; >>> //ierr = VecSet(c,one); >>> ierr = VecSet(c,one); >>> ierr = VecSet(u,one); >>> ierr = VecSet(b,one); >>> >>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>> >>> >>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>> ierr = VecNormBegin(u,NORM_2,&norm1); >>> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>> ierr = KSP_MatMult(ksp,A,c,Ac); >>> ierr = VecNormEnd(u,NORM_2,&norm1); >>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>> >>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); >>> >>> >>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>> ierr = KSP_MatMult(ksp,A,c,Ac); >>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>> >>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); >>> >>> >>> On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. >>> >>> The time used for the asynchronous calculation: 0.000203 >>> + |u| = 100. >>> The time used for the synchronous calculation: 0.000006 >>> + |b| = 100. >>> >>> Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? >>> >>> Thank you very much for anything you can provide. >>> Sincerely, >>> Viet. >>> >>> >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: allreduce_test.c Type: application/octet-stream Size: 9127 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jan 25 07:09:49 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Jan 2021 08:09:49 -0500 Subject: [petsc-users] Gathering distributed dense matrix only on rank 0 In-Reply-To: <0fdad010-6d01-0550-e958-d77f21a70d05@ntnu.no> References: <0fdad010-6d01-0550-e958-d77f21a70d05@ntnu.no> Message-ID: On Mon, Jan 25, 2021 at 4:40 AM Roland Richter wrote: > Hei, > > is there a way to gather a distributed dense matrix only on rank 0, not > on the other ranks? My matrix processing/storage routines in my program > currently are single-threaded and only operate on rank 0, and therefore > I assume that I can ignore all other ranks. This should also save a bit > time, I hope (MatCreateRedundantMatrix is currently the by far most > expensive part of my program). > > When executing MatCreateRedundantMatrix(A, size, MPI_COMM_NULL, > MAT_INITIAL_MATRIX, &B), usually all ranks get the full matrices, > something I would like to avoid. Therefore, is there another way of > handling this? > I am skeptical that it will save time, but you can do what you want using https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html Thanks, Matt > Thanks! > > Regards, > > Roland Richter > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sajidsyed2021 at u.northwestern.edu Mon Jan 25 18:36:40 2021 From: sajidsyed2021 at u.northwestern.edu (Sajid Ali) Date: Mon, 25 Jan 2021 18:36:40 -0600 Subject: [petsc-users] Convert a 3D DMDA sub-vector to a natural 2D vector In-Reply-To: <90FF5B38-852E-4BF9-B4FA-01823A4FF02F@gmail.com> References: <90FF5B38-852E-4BF9-B4FA-01823A4FF02F@gmail.com> Message-ID: Hi Randall, Thanks for providing a pointer to the DMDAGetRay function! After looking at its implementation, I came up with a solution that creates a natural ordered slice vector on the same subset of processors as the DMDA ordered slice vector (by scattering from the DMDA order slice to a natural ordered slice by using an AO associated with a temporary 2D DMDA object that lives only on the subset of ranks where the slice vector lives). I've attached the code for the same should it be of interest to anyone who reads this. -- Sajid Ali (he/him) | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex_slice_nat.c Type: application/octet-stream Size: 13363 bytes Desc: not available URL: From hqhviet at tohoku.ac.jp Mon Jan 25 22:31:18 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Tue, 26 Jan 2021 13:31:18 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: Dear Patrick Sanan, Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? Thanks again. Viet On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan wrote: > Sorry about the delay in responding, but I'll add a couple of points here: > > > 1) It's important to have some reason to believe that pipelining will > actually help your problem. Pipelined Krylov methods work by overlapping > reductions with operator and preconditioner applications. So, to see > speedup, the time for a reduction needs to be comparable to the time for > the operator/preconditioner application. This will only be true in some > cases - typical cases are when you have a large number of ranks/nodes, a > slow network, or very fast operator/preconditioner applications (assuming > that these require the same time on each rank - it's an interesting case > when they don't, but unless you say otherwise I'll assume this doesn't > apply to your use case). > > 2) As you're discovering, simply ensuring that asynchronous progress > works, at the pure MPI level, isn't as easy as it might be, as it's so > dependent on the MPI implementation. > > > For both of these reasons, I suggest setting up a test that just directly > uses MPI (which you can of course do from a PETSc-style code) and allows > you to compare times for blocking and non-blocking reductions, overlapping > some (useless) local work. You should make sure to run multiple iterations > within the script, and also run the script multiple times on the cluster > (bearing in mind that it's possible that the performance will be affected > by other users of the system). > > I attach an old script I found that I used to test some of these things, > to give a more concrete idea of what I mean. Note that this was used early > on in our own exploration of these topics so I'm only offering it to give > an idea, not as a meaningful benchmark in its own right. > > Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. : > > > Dear Barry, > > Thank you very much for your information. > > It seems complicated to set environment variables to allow asynchronous > progress and pinning threads to cores when using Intel MPI. > > $ export I_MPI_ASYNC_PROGRESS = 1 > $ export I_MPI_ASYNC_PROGRESS_PIN = > > > > https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ > > I'm still not sure how to get an appropriate "CPU list" when running MPI > with multiple nodes and multiple processes on one node. > Best, > Viet. > > > > > On Sat, Jan 23, 2021 at 3:01 AM Barry Smith wrote: > >> >> >> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >> >> It states "and a partial support for non-blocking collectives ( MPI_Ibcas >> t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial >> support means but you can try setting the variables and see if that helps. >> >> >> >> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. wrote: >> >> >> Dear Victor and Berry, >> >> Thank you so much for your answers. >> >> I fixed the code with the bug in the PetscCommSplitReductionBegin >> function as commented by Brave. >> >> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) >> u)); >> >> It was also a mistake to set the vector size too small. >> I just set a vector size of 100000000 and ran the code on 4 nodes with 2 >> processors per node. The result is as follows >> >> The time used for the asynchronous calculation: 0.022043 >> + | u | = 10000. >> The time used for the synchronous calculation: 0.016188 >> + | b | = 10000. >> >> Asynchronous computation still takes a longer time. >> >> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the >> file $PETSC_DIR/include/petscconf.h >> >> I built Petsc by using the following script >> >> #!/usr/bin/bash >> set -e >> DATE="21.01.18" >> >> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >> >> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >> INSTL_DIR="${HOME}/local/petsc-3.14.3" >> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >> PETSC_DIR="${HOME}/tmp/petsc" >> >> cd ${PETSC_DIR} >> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} >> --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec >> --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} >> --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native >> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 >> -march=native -mtune=native' >> >> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >> >> >> Intel 2018 also complies with the MPI-3 standard. >> >> Are there specific settings for Intel MPI to obtain the performance of >> the MPI_IALLREDUCE function? >> >> Sincerely, >> Viet. >> >> >> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: >> >>> >>> ierr = VecNormBegin(u,NORM_2,&norm1); >>> ierr = >>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>> >>> How come you call this on Ax and not on u? For clarity, if nothing else, >>> I think you should call it on u. >>> >>> comb.c has >>> >>> /* >>> Split phase global vector reductions with support for combining the >>> communication portion of several operations. Using MPI-1.1 support >>> only >>> >>> The idea for this and much of the initial code is contributed by >>> Victor Eijkhout. >>> >>> Usage: >>> VecDotBegin(Vec,Vec,PetscScalar *); >>> VecNormBegin(Vec,NormType,PetscReal *); >>> .... >>> VecDotEnd(Vec,Vec,PetscScalar *); >>> VecNormEnd(Vec,NormType,PetscReal *); >>> >>> Limitations: >>> - The order of the xxxEnd() functions MUST be in the same order >>> as the xxxBegin(). There is extensive error checking to try to >>> insure that the user calls the routines in the correct order >>> */ >>> >>> #include /*I "petscvec.h" I*/ >>> >>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void >>> *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm >>> comm,MPI_Request *request) >>> { >>> PETSC_UNUSED PetscErrorCode ierr; >>> >>> PetscFunctionBegin; >>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>> ierr = >>> MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>> ierr = >>> MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>> #else >>> ierr = >>> MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>> *request = MPI_REQUEST_NULL; >>> #endif >>> PetscFunctionReturn(0); >>> } >>> >>> >>> So first check if $PETSC_DIR/include/petscconf.h has >>> >>> PETSC_HAVE_MPI_IALLREDUCE >>> >>> if it does not then the standard MPI reduce is called. >>> >>> If this is set then any improvement depends on the implementation of >>> iallreduce inside the MPI you are using. >>> >>> Barry >>> >>> >>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: >>> >>> >>> Hello Petsc developers and supporters, >>> >>> I would like to confirm the performance of asynchronous computations of >>> inner product computation overlapping with matrix-vector multiplication >>> computation by the below code. >>> >>> >>> PetscLogDouble tt1,tt2; >>> KSP ksp; >>> //ierr = VecSet(c,one); >>> ierr = VecSet(c,one); >>> ierr = VecSet(u,one); >>> ierr = VecSet(b,one); >>> >>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>> >>> >>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>> ierr = VecNormBegin(u,NORM_2,&norm1); >>> ierr = >>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>> ierr = KSP_MatMult(ksp,A,c,Ac); >>> ierr = VecNormEnd(u,NORM_2,&norm1); >>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>> >>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>> asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); >>> CHKERRQ(ierr); >>> >>> >>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>> ierr = KSP_MatMult(ksp,A,c,Ac); >>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>> >>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>> synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); >>> CHKERRQ(ierr); >>> >>> >>> On a cluster with two or four nodes, the asynchronous computation is >>> always much slower than synchronous computation. >>> >>> The time used for the asynchronous calculation: 0.000203 >>> + |u| = 100. >>> The time used for the synchronous calculation: 0.000006 >>> + |b| = 100. >>> >>> Are there any necessary settings on MPI or Petsc to gain performance of >>> asynchronous computation? >>> >>> Thank you very much for anything you can provide. >>> Sincerely, >>> Viet. >>> >>> >>> >>> >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 26 05:01:26 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Jan 2021 06:01:26 -0500 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. wrote: > Dear Patrick Sanan, > > Thank you very much for your answer, especially for your code. > I was able to compile and run your code on 8 nodes with 20 processes per > node. Below is the result > > Testing with 160 MPI ranks > reducing an array of size 32 (256 bytes) > Running 5 burnin runs and 100 tests ... Done. > For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over > all ranks: > MPI timer resolution: 1.0000e-06 seconds > MPI timer resolution/#trials: 1.0000e-08 seconds > B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds > N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds > Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds > Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds > Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds > Observe to see if the local time is enough to hide the reduction, and see > if the reduction is indeed hidden > > > It appears that the non-blocking computation with this test is no faster > than the blocking computation. > I think I am missing some suitable Intel MPI environment settings. > I am now thinking about using MPICH, which does not require any > environment settings for non-blocking computation. > Could you please let me know which MPI (MPICH or OpenMPI) you used in your > tests? > Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application? Thanks, Matt > Thanks again. > Viet > > > On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan > wrote: > >> Sorry about the delay in responding, but I'll add a couple of points here: >> >> >> 1) It's important to have some reason to believe that pipelining will >> actually help your problem. Pipelined Krylov methods work by overlapping >> reductions with operator and preconditioner applications. So, to see >> speedup, the time for a reduction needs to be comparable to the time for >> the operator/preconditioner application. This will only be true in some >> cases - typical cases are when you have a large number of ranks/nodes, a >> slow network, or very fast operator/preconditioner applications (assuming >> that these require the same time on each rank - it's an interesting case >> when they don't, but unless you say otherwise I'll assume this doesn't >> apply to your use case). >> >> 2) As you're discovering, simply ensuring that asynchronous progress >> works, at the pure MPI level, isn't as easy as it might be, as it's so >> dependent on the MPI implementation. >> >> >> For both of these reasons, I suggest setting up a test that just directly >> uses MPI (which you can of course do from a PETSc-style code) and allows >> you to compare times for blocking and non-blocking reductions, overlapping >> some (useless) local work. You should make sure to run multiple iterations >> within the script, and also run the script multiple times on the cluster >> (bearing in mind that it's possible that the performance will be affected >> by other users of the system). >> >> I attach an old script I found that I used to test some of these things, >> to give a more concrete idea of what I mean. Note that this was used early >> on in our own exploration of these topics so I'm only offering it to give >> an idea, not as a meaningful benchmark in its own right. >> >> Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. : >> >> >> Dear Barry, >> >> Thank you very much for your information. >> >> It seems complicated to set environment variables to allow asynchronous >> progress and pinning threads to cores when using Intel MPI. >> >> $ export I_MPI_ASYNC_PROGRESS = 1 >> $ export I_MPI_ASYNC_PROGRESS_PIN = >> >> >> >> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ >> >> I'm still not sure how to get an appropriate "CPU list" when running MPI >> with multiple nodes and multiple processes on one node. >> Best, >> Viet. >> >> >> >> >> On Sat, Jan 23, 2021 at 3:01 AM Barry Smith wrote: >> >>> >>> >>> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >>> >>> It states "and a partial support for non-blocking collectives ( >>> MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what >>> partial support means but you can try setting the variables and see if that >>> helps. >>> >>> >>> >>> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. wrote: >>> >>> >>> Dear Victor and Berry, >>> >>> Thank you so much for your answers. >>> >>> I fixed the code with the bug in the PetscCommSplitReductionBegin >>> function as commented by Brave. >>> >>> ierr = PetscCommSplitReductionBegin (PetscObjectComm >>> ((PetscObject) u)); >>> >>> It was also a mistake to set the vector size too small. >>> I just set a vector size of 100000000 and ran the code on 4 nodes with 2 >>> processors per node. The result is as follows >>> >>> The time used for the asynchronous calculation: 0.022043 >>> + | u | = 10000. >>> The time used for the synchronous calculation: 0.016188 >>> + | b | = 10000. >>> >>> Asynchronous computation still takes a longer time. >>> >>> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the >>> file $PETSC_DIR/include/petscconf.h >>> >>> I built Petsc by using the following script >>> >>> #!/usr/bin/bash >>> set -e >>> DATE="21.01.18" >>> >>> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >>> >>> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >>> INSTL_DIR="${HOME}/local/petsc-3.14.3" >>> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >>> PETSC_DIR="${HOME}/tmp/petsc" >>> >>> cd ${PETSC_DIR} >>> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} >>> --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec >>> --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} >>> --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native >>> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 >>> -march=native -mtune=native' >>> >>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >>> >>> >>> Intel 2018 also complies with the MPI-3 standard. >>> >>> Are there specific settings for Intel MPI to obtain the performance of >>> the MPI_IALLREDUCE function? >>> >>> Sincerely, >>> Viet. >>> >>> >>> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: >>> >>>> >>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>> ierr = >>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>> >>>> How come you call this on Ax and not on u? For clarity, if nothing >>>> else, I think you should call it on u. >>>> >>>> comb.c has >>>> >>>> /* >>>> Split phase global vector reductions with support for combining >>>> the >>>> communication portion of several operations. Using MPI-1.1 support >>>> only >>>> >>>> The idea for this and much of the initial code is contributed by >>>> Victor Eijkhout. >>>> >>>> Usage: >>>> VecDotBegin(Vec,Vec,PetscScalar *); >>>> VecNormBegin(Vec,NormType,PetscReal *); >>>> .... >>>> VecDotEnd(Vec,Vec,PetscScalar *); >>>> VecNormEnd(Vec,NormType,PetscReal *); >>>> >>>> Limitations: >>>> - The order of the xxxEnd() functions MUST be in the same order >>>> as the xxxBegin(). There is extensive error checking to try >>>> to >>>> insure that the user calls the routines in the correct order >>>> */ >>>> >>>> #include /*I "petscvec.h" I*/ >>>> >>>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void >>>> *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm >>>> comm,MPI_Request *request) >>>> { >>>> PETSC_UNUSED PetscErrorCode ierr; >>>> >>>> PetscFunctionBegin; >>>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>>> ierr = >>>> MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>>> ierr = >>>> MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>>> #else >>>> ierr = >>>> MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>>> *request = MPI_REQUEST_NULL; >>>> #endif >>>> PetscFunctionReturn(0); >>>> } >>>> >>>> >>>> So first check if $PETSC_DIR/include/petscconf.h has >>>> >>>> PETSC_HAVE_MPI_IALLREDUCE >>>> >>>> if it does not then the standard MPI reduce is called. >>>> >>>> If this is set then any improvement depends on the implementation of >>>> iallreduce inside the MPI you are using. >>>> >>>> Barry >>>> >>>> >>>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: >>>> >>>> >>>> Hello Petsc developers and supporters, >>>> >>>> I would like to confirm the performance of asynchronous computations of >>>> inner product computation overlapping with matrix-vector multiplication >>>> computation by the below code. >>>> >>>> >>>> PetscLogDouble tt1,tt2; >>>> KSP ksp; >>>> //ierr = VecSet(c,one); >>>> ierr = VecSet(c,one); >>>> ierr = VecSet(u,one); >>>> ierr = VecSet(b,one); >>>> >>>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>>> >>>> >>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>> ierr = >>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>> ierr = VecNormEnd(u,NORM_2,&norm1); >>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>> asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) >>>> norm1); CHKERRQ(ierr); >>>> >>>> >>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>> synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) >>>> norm2); CHKERRQ(ierr); >>>> >>>> >>>> On a cluster with two or four nodes, the asynchronous computation is >>>> always much slower than synchronous computation. >>>> >>>> The time used for the asynchronous calculation: 0.000203 >>>> + |u| = 100. >>>> The time used for the synchronous calculation: 0.000006 >>>> + |b| = 100. >>>> >>>> Are there any necessary settings on MPI or Petsc to gain performance of >>>> asynchronous computation? >>>> >>>> Thank you very much for anything you can provide. >>>> Sincerely, >>>> Viet. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Tue Jan 26 11:53:10 2021 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Tue, 26 Jan 2021 18:53:10 +0100 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: > Am 26.01.2021 um 12:01 schrieb Matthew Knepley : > > On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. > wrote: > Dear Patrick Sanan, > > Thank you very much for your answer, especially for your code. > I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result > > Testing with 160 MPI ranks > reducing an array of size 32 (256 bytes) > Running 5 burnin runs and 100 tests ... Done. > For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: > MPI timer resolution: 1.0000e-06 seconds > MPI timer resolution/#trials: 1.0000e-08 seconds > B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds > N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds > Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds > Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds > Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds > Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden > > It appears that the non-blocking computation with this test is no faster than the blocking computation. > I think I am missing some suitable Intel MPI environment settings. > I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. > Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? I used Cray MPI, which is based on MPICH. > > Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely > larger in your application? This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application? > > Thanks, > > Matt > > Thanks again. > Viet > > > On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan > wrote: > Sorry about the delay in responding, but I'll add a couple of points here: > > > 1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case). > > 2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation. > > > For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system). > > I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right. > >> Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. >: >> >> >> Dear Barry, >> >> Thank you very much for your information. >> >> It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI. >> >> $ export I_MPI_ASYNC_PROGRESS = 1 >> $ export I_MPI_ASYNC_PROGRESS_PIN = >> >> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ >> >> I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. >> Best, >> Viet. >> >> >> >> >> On Sat, Jan 23, 2021 at 3:01 AM Barry Smith > wrote: >> >> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >> >> It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps. >> >> >> >>> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. > wrote: >>> >>> >>> Dear Victor and Berry, >>> >>> Thank you so much for your answers. >>> >>> I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. >>> >>> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); >>> >>> It was also a mistake to set the vector size too small. >>> I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows >>> >>> The time used for the asynchronous calculation: 0.022043 >>> + | u | = 10000. >>> The time used for the synchronous calculation: 0.016188 >>> + | b | = 10000. >>> >>> Asynchronous computation still takes a longer time. >>> >>> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h >>> >>> I built Petsc by using the following script >>> >>> #!/usr/bin/bash >>> set -e >>> DATE="21.01.18" >>> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >>> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >>> INSTL_DIR="${HOME}/local/petsc-3.14.3" >>> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >>> PETSC_DIR="${HOME}/tmp/petsc" >>> >>> cd ${PETSC_DIR} >>> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' >>> >>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >>> >>> >>> Intel 2018 also complies with the MPI-3 standard. >>> >>> Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? >>> >>> Sincerely, >>> Viet. >>> >>> >>> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith > wrote: >>> >>> ierr = VecNormBegin(u,NORM_2,&norm1); >>> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>> >>> How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. >>> >>> comb.c has >>> >>> /* >>> Split phase global vector reductions with support for combining the >>> communication portion of several operations. Using MPI-1.1 support only >>> >>> The idea for this and much of the initial code is contributed by >>> Victor Eijkhout. >>> >>> Usage: >>> VecDotBegin(Vec,Vec,PetscScalar *); >>> VecNormBegin(Vec,NormType,PetscReal *); >>> .... >>> VecDotEnd(Vec,Vec,PetscScalar *); >>> VecNormEnd(Vec,NormType,PetscReal *); >>> >>> Limitations: >>> - The order of the xxxEnd() functions MUST be in the same order >>> as the xxxBegin(). There is extensive error checking to try to >>> insure that the user calls the routines in the correct order >>> */ >>> >>> #include /*I "petscvec.h" I*/ >>> >>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) >>> { >>> PETSC_UNUSED PetscErrorCode ierr; >>> >>> PetscFunctionBegin; >>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>> ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>> ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>> #else >>> ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>> *request = MPI_REQUEST_NULL; >>> #endif >>> PetscFunctionReturn(0); >>> } >>> >>> >>> So first check if $PETSC_DIR/include/petscconf.h has >>> >>> PETSC_HAVE_MPI_IALLREDUCE >>> >>> if it does not then the standard MPI reduce is called. >>> >>> If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. >>> >>> Barry >>> >>> >>>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. > wrote: >>>> >>>> >>>> Hello Petsc developers and supporters, >>>> >>>> I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. >>>> >>>> >>>> PetscLogDouble tt1,tt2; >>>> KSP ksp; >>>> //ierr = VecSet(c,one); >>>> ierr = VecSet(c,one); >>>> ierr = VecSet(u,one); >>>> ierr = VecSet(b,one); >>>> >>>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>>> >>>> >>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>> ierr = VecNormEnd(u,NORM_2,&norm1); >>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); >>>> >>>> >>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); >>>> >>>> >>>> On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. >>>> >>>> The time used for the asynchronous calculation: 0.000203 >>>> + |u| = 100. >>>> The time used for the synchronous calculation: 0.000006 >>>> + |b| = 100. >>>> >>>> Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? >>>> >>>> Thank you very much for anything you can provide. >>>> Sincerely, >>>> Viet. >>>> >>>> >>>> >>>> >>>> >>>> >>> >> > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hqhviet at tohoku.ac.jp Wed Jan 27 09:50:10 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Thu, 28 Jan 2021 00:50:10 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: Dear Patrick and Matthew, Thank you very much for your answers. My test code is just the code that contains the inner product calculation and the calculation of the matrix-vector multiplication. It is easy to measure the time of each calculation. I'm going to revise the code to measure the time as it's done in Patrick's code. I will also switch to Mpich so as not to worry about the MPI environment settings. With the current number of nodes (8), the reduction time for calculating the inner product is relatively small, but I think it would get bigger if I increased the number of nodes to 100 or more. In this case, the latency of the inner product calculation would increase, since the reduced values are transmitted via a complex network structure with cables and switches. Then, hopefully, the non-blocking communication can show its effectiveness. Best, Viet On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan wrote: > > > Am 26.01.2021 um 12:01 schrieb Matthew Knepley : > > On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. wrote: > >> Dear Patrick Sanan, >> >> Thank you very much for your answer, especially for your code. >> I was able to compile and run your code on 8 nodes with 20 processes per >> node. Below is the result >> >> Testing with 160 MPI ranks >> reducing an array of size 32 (256 bytes) >> Running 5 burnin runs and 100 tests ... Done. >> For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over >> all ranks: >> MPI timer resolution: 1.0000e-06 seconds >> MPI timer resolution/#trials: 1.0000e-08 seconds >> B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds >> N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds >> Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds >> Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds >> Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds >> Observe to see if the local time is enough to hide the reduction, and see >> if the reduction is indeed hidden >> >> >> It appears that the non-blocking computation with this test is no faster >> than the blocking computation. >> I think I am missing some suitable Intel MPI environment settings. >> I am now thinking about using MPICH, which does not require any >> environment settings for non-blocking computation. >> Could you please let me know which MPI (MPICH or OpenMPI) you used in >> your tests? >> > I used Cray MPI, which is based on MPICH. > > > Note that in the test, the cost of the reduction is only 3% of the total > cost. Is that worth putting time into? Is the cost definitely > larger in your application? > > This is very important! Note that the reductions are quite fast compared > to the local work. Even on a much larger number of ranks, they might still > be quite fast. The local work here is just some useless computation (here > tuned to be comparable to how long it took to do a reduction on a few > thousand ranks at the time). As Matt says, having some estimate of how long > it takes to apply your operator (matrix) and preconditioner is essential. > Do you know how to estimate that for your application? > > > Thanks, > > Matt > > >> Thanks again. >> Viet >> >> >> On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan >> wrote: >> >>> Sorry about the delay in responding, but I'll add a couple of points >>> here: >>> >>> >>> 1) It's important to have some reason to believe that pipelining will >>> actually help your problem. Pipelined Krylov methods work by overlapping >>> reductions with operator and preconditioner applications. So, to see >>> speedup, the time for a reduction needs to be comparable to the time for >>> the operator/preconditioner application. This will only be true in some >>> cases - typical cases are when you have a large number of ranks/nodes, a >>> slow network, or very fast operator/preconditioner applications (assuming >>> that these require the same time on each rank - it's an interesting case >>> when they don't, but unless you say otherwise I'll assume this doesn't >>> apply to your use case). >>> >>> 2) As you're discovering, simply ensuring that asynchronous progress >>> works, at the pure MPI level, isn't as easy as it might be, as it's so >>> dependent on the MPI implementation. >>> >>> >>> For both of these reasons, I suggest setting up a test that just >>> directly uses MPI (which you can of course do from a PETSc-style code) and >>> allows you to compare times for blocking and non-blocking reductions, >>> overlapping some (useless) local work. You should make sure to run multiple >>> iterations within the script, and also run the script multiple times on the >>> cluster (bearing in mind that it's possible that the performance will be >>> affected by other users of the system). >>> >>> I attach an old script I found that I used to test some of these things, >>> to give a more concrete idea of what I mean. Note that this was used early >>> on in our own exploration of these topics so I'm only offering it to give >>> an idea, not as a meaningful benchmark in its own right. >>> >>> Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. : >>> >>> >>> Dear Barry, >>> >>> Thank you very much for your information. >>> >>> It seems complicated to set environment variables to allow asynchronous >>> progress and pinning threads to cores when using Intel MPI. >>> >>> $ export I_MPI_ASYNC_PROGRESS = 1 >>> $ export I_MPI_ASYNC_PROGRESS_PIN = >>> >>> >>> >>> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ >>> >>> I'm still not sure how to get an appropriate "CPU list" when running MPI >>> with multiple nodes and multiple processes on one node. >>> Best, >>> Viet. >>> >>> >>> >>> >>> On Sat, Jan 23, 2021 at 3:01 AM Barry Smith wrote: >>> >>>> >>>> >>>> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >>>> >>>> It states "and a partial support for non-blocking collectives ( >>>> MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what >>>> partial support means but you can try setting the variables and see if that >>>> helps. >>>> >>>> >>>> >>>> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. wrote: >>>> >>>> >>>> Dear Victor and Berry, >>>> >>>> Thank you so much for your answers. >>>> >>>> I fixed the code with the bug in the PetscCommSplitReductionBegin >>>> function as commented by Brave. >>>> >>>> ierr = PetscCommSplitReductionBegin (PetscObjectComm >>>> ((PetscObject) u)); >>>> >>>> It was also a mistake to set the vector size too small. >>>> I just set a vector size of 100000000 and ran the code on 4 nodes with >>>> 2 processors per node. The result is as follows >>>> >>>> The time used for the asynchronous calculation: 0.022043 >>>> + | u | = 10000. >>>> The time used for the synchronous calculation: 0.016188 >>>> + | b | = 10000. >>>> >>>> Asynchronous computation still takes a longer time. >>>> >>>> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the >>>> file $PETSC_DIR/include/petscconf.h >>>> >>>> I built Petsc by using the following script >>>> >>>> #!/usr/bin/bash >>>> set -e >>>> DATE="21.01.18" >>>> >>>> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >>>> >>>> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >>>> INSTL_DIR="${HOME}/local/petsc-3.14.3" >>>> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >>>> PETSC_DIR="${HOME}/tmp/petsc" >>>> >>>> cd ${PETSC_DIR} >>>> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} >>>> --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec >>>> --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} >>>> --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native >>>> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 >>>> -march=native -mtune=native' >>>> >>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >>>> >>>> >>>> Intel 2018 also complies with the MPI-3 standard. >>>> >>>> Are there specific settings for Intel MPI to obtain the performance of >>>> the MPI_IALLREDUCE function? >>>> >>>> Sincerely, >>>> Viet. >>>> >>>> >>>> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: >>>> >>>>> >>>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>>> ierr = >>>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>>> >>>>> How come you call this on Ax and not on u? For clarity, if nothing >>>>> else, I think you should call it on u. >>>>> >>>>> comb.c has >>>>> >>>>> /* >>>>> Split phase global vector reductions with support for combining >>>>> the >>>>> communication portion of several operations. Using MPI-1.1 support >>>>> only >>>>> >>>>> The idea for this and much of the initial code is contributed by >>>>> Victor Eijkhout. >>>>> >>>>> Usage: >>>>> VecDotBegin(Vec,Vec,PetscScalar *); >>>>> VecNormBegin(Vec,NormType,PetscReal *); >>>>> .... >>>>> VecDotEnd(Vec,Vec,PetscScalar *); >>>>> VecNormEnd(Vec,NormType,PetscReal *); >>>>> >>>>> Limitations: >>>>> - The order of the xxxEnd() functions MUST be in the same >>>>> order >>>>> as the xxxBegin(). There is extensive error checking to try >>>>> to >>>>> insure that the user calls the routines in the correct order >>>>> */ >>>>> >>>>> #include /*I "petscvec.h" I*/ >>>>> >>>>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void >>>>> *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm >>>>> comm,MPI_Request *request) >>>>> { >>>>> PETSC_UNUSED PetscErrorCode ierr; >>>>> >>>>> PetscFunctionBegin; >>>>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>>>> ierr = >>>>> MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>>>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>>>> ierr = >>>>> MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>>>> #else >>>>> ierr = >>>>> MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>>>> *request = MPI_REQUEST_NULL; >>>>> #endif >>>>> PetscFunctionReturn(0); >>>>> } >>>>> >>>>> >>>>> So first check if $PETSC_DIR/include/petscconf.h has >>>>> >>>>> PETSC_HAVE_MPI_IALLREDUCE >>>>> >>>>> if it does not then the standard MPI reduce is called. >>>>> >>>>> If this is set then any improvement depends on the implementation of >>>>> iallreduce inside the MPI you are using. >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. wrote: >>>>> >>>>> >>>>> Hello Petsc developers and supporters, >>>>> >>>>> I would like to confirm the performance of asynchronous computations >>>>> of inner product computation overlapping with matrix-vector multiplication >>>>> computation by the below code. >>>>> >>>>> >>>>> PetscLogDouble tt1,tt2; >>>>> KSP ksp; >>>>> //ierr = VecSet(c,one); >>>>> ierr = VecSet(c,one); >>>>> ierr = VecSet(u,one); >>>>> ierr = VecSet(b,one); >>>>> >>>>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>>>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>>>> >>>>> >>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>>> ierr = >>>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>> ierr = VecNormEnd(u,NORM_2,&norm1); >>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>> >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>>> asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) >>>>> norm1); CHKERRQ(ierr); >>>>> >>>>> >>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>> >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>>> synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) >>>>> norm2); CHKERRQ(ierr); >>>>> >>>>> >>>>> On a cluster with two or four nodes, the asynchronous computation is >>>>> always much slower than synchronous computation. >>>>> >>>>> The time used for the asynchronous calculation: 0.000203 >>>>> + |u| = 100. >>>>> The time used for the synchronous calculation: 0.000006 >>>>> + |b| = 100. >>>>> >>>>> Are there any necessary settings on MPI or Petsc to gain performance >>>>> of asynchronous computation? >>>>> >>>>> Thank you very much for anything you can provide. >>>>> Sincerely, >>>>> Viet. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 27 10:30:11 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Jan 2021 11:30:11 -0500 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: On Wed, Jan 27, 2021 at 10:51 AM Viet H.Q.H. wrote: > Dear Patrick and Matthew, > > Thank you very much for your answers. > My test code is just the code that contains the inner product calculation > and the calculation of the matrix-vector multiplication. It is easy to > measure the time of each calculation. I'm going to revise the code to > measure the time as it's done in Patrick's code. I will also switch to > Mpich so as not to worry about the MPI environment settings. > > With the current number of nodes (8), the reduction time for calculating > the inner product is relatively small, but I think it would get bigger if I > increased the number of nodes to 100 or more. In this case, the latency of > the inner product calculation would increase, since the reduced values are > transmitted via a complex network structure with cables and switches. Then, > hopefully, the non-blocking communication can show its effectiveness. > This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on. Thanks Matt > Best, > Viet > > > On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan > wrote: > >> >> >> Am 26.01.2021 um 12:01 schrieb Matthew Knepley : >> >> On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. >> wrote: >> >>> Dear Patrick Sanan, >>> >>> Thank you very much for your answer, especially for your code. >>> I was able to compile and run your code on 8 nodes with 20 processes per >>> node. Below is the result >>> >>> Testing with 160 MPI ranks >>> reducing an array of size 32 (256 bytes) >>> Running 5 burnin runs and 100 tests ... Done. >>> For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times >>> over all ranks: >>> MPI timer resolution: 1.0000e-06 seconds >>> MPI timer resolution/#trials: 1.0000e-08 seconds >>> B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds >>> N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds >>> Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds >>> Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds >>> Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds >>> Observe to see if the local time is enough to hide the reduction, and >>> see if the reduction is indeed hidden >>> >>> >>> It appears that the non-blocking computation with this test is no faster >>> than the blocking computation. >>> I think I am missing some suitable Intel MPI environment settings. >>> I am now thinking about using MPICH, which does not require any >>> environment settings for non-blocking computation. >>> Could you please let me know which MPI (MPICH or OpenMPI) you used in >>> your tests? >>> >> I used Cray MPI, which is based on MPICH. >> >> >> Note that in the test, the cost of the reduction is only 3% of the total >> cost. Is that worth putting time into? Is the cost definitely >> larger in your application? >> >> This is very important! Note that the reductions are quite fast compared >> to the local work. Even on a much larger number of ranks, they might still >> be quite fast. The local work here is just some useless computation (here >> tuned to be comparable to how long it took to do a reduction on a few >> thousand ranks at the time). As Matt says, having some estimate of how long >> it takes to apply your operator (matrix) and preconditioner is essential. >> Do you know how to estimate that for your application? >> >> >> Thanks, >> >> Matt >> >> >>> Thanks again. >>> Viet >>> >>> >>> On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan >>> wrote: >>> >>>> Sorry about the delay in responding, but I'll add a couple of points >>>> here: >>>> >>>> >>>> 1) It's important to have some reason to believe that pipelining will >>>> actually help your problem. Pipelined Krylov methods work by overlapping >>>> reductions with operator and preconditioner applications. So, to see >>>> speedup, the time for a reduction needs to be comparable to the time for >>>> the operator/preconditioner application. This will only be true in some >>>> cases - typical cases are when you have a large number of ranks/nodes, a >>>> slow network, or very fast operator/preconditioner applications (assuming >>>> that these require the same time on each rank - it's an interesting case >>>> when they don't, but unless you say otherwise I'll assume this doesn't >>>> apply to your use case). >>>> >>>> 2) As you're discovering, simply ensuring that asynchronous progress >>>> works, at the pure MPI level, isn't as easy as it might be, as it's so >>>> dependent on the MPI implementation. >>>> >>>> >>>> For both of these reasons, I suggest setting up a test that just >>>> directly uses MPI (which you can of course do from a PETSc-style code) and >>>> allows you to compare times for blocking and non-blocking reductions, >>>> overlapping some (useless) local work. You should make sure to run multiple >>>> iterations within the script, and also run the script multiple times on the >>>> cluster (bearing in mind that it's possible that the performance will be >>>> affected by other users of the system). >>>> >>>> I attach an old script I found that I used to test some of these >>>> things, to give a more concrete idea of what I mean. Note that this was >>>> used early on in our own exploration of these topics so I'm only offering >>>> it to give an idea, not as a meaningful benchmark in its own right. >>>> >>>> Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. : >>>> >>>> >>>> Dear Barry, >>>> >>>> Thank you very much for your information. >>>> >>>> It seems complicated to set environment variables to allow asynchronous >>>> progress and pinning threads to cores when using Intel MPI. >>>> >>>> $ export I_MPI_ASYNC_PROGRESS = 1 >>>> $ export I_MPI_ASYNC_PROGRESS_PIN = >>>> >>>> >>>> >>>> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ >>>> >>>> I'm still not sure how to get an appropriate "CPU list" when running >>>> MPI with multiple nodes and multiple processes on one node. >>>> Best, >>>> Viet. >>>> >>>> >>>> >>>> >>>> On Sat, Jan 23, 2021 at 3:01 AM Barry Smith wrote: >>>> >>>>> >>>>> >>>>> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >>>>> >>>>> It states "and a partial support for non-blocking collectives ( >>>>> MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what >>>>> partial support means but you can try setting the variables and see if that >>>>> helps. >>>>> >>>>> >>>>> >>>>> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. >>>>> wrote: >>>>> >>>>> >>>>> Dear Victor and Berry, >>>>> >>>>> Thank you so much for your answers. >>>>> >>>>> I fixed the code with the bug in the PetscCommSplitReductionBegin >>>>> function as commented by Brave. >>>>> >>>>> ierr = PetscCommSplitReductionBegin (PetscObjectComm >>>>> ((PetscObject) u)); >>>>> >>>>> It was also a mistake to set the vector size too small. >>>>> I just set a vector size of 100000000 and ran the code on 4 nodes with >>>>> 2 processors per node. The result is as follows >>>>> >>>>> The time used for the asynchronous calculation: 0.022043 >>>>> + | u | = 10000. >>>>> The time used for the synchronous calculation: 0.016188 >>>>> + | b | = 10000. >>>>> >>>>> Asynchronous computation still takes a longer time. >>>>> >>>>> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the >>>>> file $PETSC_DIR/include/petscconf.h >>>>> >>>>> I built Petsc by using the following script >>>>> >>>>> #!/usr/bin/bash >>>>> set -e >>>>> DATE="21.01.18" >>>>> >>>>> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >>>>> >>>>> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >>>>> INSTL_DIR="${HOME}/local/petsc-3.14.3" >>>>> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >>>>> PETSC_DIR="${HOME}/tmp/petsc" >>>>> >>>>> cd ${PETSC_DIR} >>>>> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} >>>>> --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec >>>>> --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} >>>>> --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native >>>>> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 >>>>> -march=native -mtune=native' >>>>> >>>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >>>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >>>>> >>>>> >>>>> Intel 2018 also complies with the MPI-3 standard. >>>>> >>>>> Are there specific settings for Intel MPI to obtain the performance of >>>>> the MPI_IALLREDUCE function? >>>>> >>>>> Sincerely, >>>>> Viet. >>>>> >>>>> >>>>> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith wrote: >>>>> >>>>>> >>>>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>>>> ierr = >>>>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>>>> >>>>>> How come you call this on Ax and not on u? For clarity, if nothing >>>>>> else, I think you should call it on u. >>>>>> >>>>>> comb.c has >>>>>> >>>>>> /* >>>>>> Split phase global vector reductions with support for >>>>>> combining the >>>>>> communication portion of several operations. Using MPI-1.1 support >>>>>> only >>>>>> >>>>>> The idea for this and much of the initial code is contributed >>>>>> by >>>>>> Victor Eijkhout. >>>>>> >>>>>> Usage: >>>>>> VecDotBegin(Vec,Vec,PetscScalar *); >>>>>> VecNormBegin(Vec,NormType,PetscReal *); >>>>>> .... >>>>>> VecDotEnd(Vec,Vec,PetscScalar *); >>>>>> VecNormEnd(Vec,NormType,PetscReal *); >>>>>> >>>>>> Limitations: >>>>>> - The order of the xxxEnd() functions MUST be in the same >>>>>> order >>>>>> as the xxxBegin(). There is extensive error checking to >>>>>> try to >>>>>> insure that the user calls the routines in the correct >>>>>> order >>>>>> */ >>>>>> >>>>>> #include /*I "petscvec.h" I*/ >>>>>> >>>>>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void >>>>>> *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm >>>>>> comm,MPI_Request *request) >>>>>> { >>>>>> PETSC_UNUSED PetscErrorCode ierr; >>>>>> >>>>>> PetscFunctionBegin; >>>>>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>>>>> ierr = >>>>>> MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>>>>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>>>>> ierr = >>>>>> MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>>>>> #else >>>>>> ierr = >>>>>> MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>>>>> *request = MPI_REQUEST_NULL; >>>>>> #endif >>>>>> PetscFunctionReturn(0); >>>>>> } >>>>>> >>>>>> >>>>>> So first check if $PETSC_DIR/include/petscconf.h has >>>>>> >>>>>> PETSC_HAVE_MPI_IALLREDUCE >>>>>> >>>>>> if it does not then the standard MPI reduce is called. >>>>>> >>>>>> If this is set then any improvement depends on the implementation of >>>>>> iallreduce inside the MPI you are using. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. >>>>>> wrote: >>>>>> >>>>>> >>>>>> Hello Petsc developers and supporters, >>>>>> >>>>>> I would like to confirm the performance of asynchronous computations >>>>>> of inner product computation overlapping with matrix-vector multiplication >>>>>> computation by the below code. >>>>>> >>>>>> >>>>>> PetscLogDouble tt1,tt2; >>>>>> KSP ksp; >>>>>> //ierr = VecSet(c,one); >>>>>> ierr = VecSet(c,one); >>>>>> ierr = VecSet(u,one); >>>>>> ierr = VecSet(b,one); >>>>>> >>>>>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>>>>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>>>>> >>>>>> >>>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>>>> ierr = >>>>>> PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>>> ierr = VecNormEnd(u,NORM_2,&norm1); >>>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>>> >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>>>> asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) >>>>>> norm1); CHKERRQ(ierr); >>>>>> >>>>>> >>>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>>> >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the >>>>>> synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) >>>>>> norm2); CHKERRQ(ierr); >>>>>> >>>>>> >>>>>> On a cluster with two or four nodes, the asynchronous computation is >>>>>> always much slower than synchronous computation. >>>>>> >>>>>> The time used for the asynchronous calculation: 0.000203 >>>>>> + |u| = 100. >>>>>> The time used for the synchronous calculation: 0.000006 >>>>>> + |b| = 100. >>>>>> >>>>>> Are there any necessary settings on MPI or Petsc to gain performance >>>>>> of asynchronous computation? >>>>>> >>>>>> Thank you very much for anything you can provide. >>>>>> Sincerely, >>>>>> Viet. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Thu Jan 28 09:11:02 2021 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Thu, 28 Jan 2021 16:11:02 +0100 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: > Am 27.01.2021 um 17:30 schrieb Matthew Knepley : > > On Wed, Jan 27, 2021 at 10:51 AM Viet H.Q.H. > wrote: > Dear Patrick and Matthew, > > Thank you very much for your answers. > My test code is just the code that contains the inner product calculation and the calculation of the matrix-vector multiplication. It is easy to measure the time of each calculation. I'm going to revise the code to measure the time as it's done in Patrick's code. I will also switch to Mpich so as not to worry about the MPI environment settings. > > With the current number of nodes (8), the reduction time for calculating the inner product is relatively small, but I think it would get bigger if I increased the number of nodes to 100 or more. In this case, the latency of the inner product calculation would increase, since the reduced values are transmitted via a complex network structure with cables and switches. Then, hopefully, the non-blocking communication can show its effectiveness. > > This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on. > Seconded! A very simple performance model is to expect reductions to take C * log(P) time on P ranks, which is of course a slowly increasing function. > Thanks > > Matt > > Best, > Viet > > > On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan > wrote: > > >> Am 26.01.2021 um 12:01 schrieb Matthew Knepley >: >> >> On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. > wrote: >> Dear Patrick Sanan, >> >> Thank you very much for your answer, especially for your code. >> I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result >> >> Testing with 160 MPI ranks >> reducing an array of size 32 (256 bytes) >> Running 5 burnin runs and 100 tests ... Done. >> For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: >> MPI timer resolution: 1.0000e-06 seconds >> MPI timer resolution/#trials: 1.0000e-08 seconds >> B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds >> N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds >> Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds >> Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds >> Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds >> Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden >> >> It appears that the non-blocking computation with this test is no faster than the blocking computation. >> I think I am missing some suitable Intel MPI environment settings. >> I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. >> Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? > I used Cray MPI, which is based on MPICH. >> >> Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely >> larger in your application? > This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application? >> >> Thanks, >> >> Matt >> >> Thanks again. >> Viet >> >> >> On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan > wrote: >> Sorry about the delay in responding, but I'll add a couple of points here: >> >> >> 1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case). >> >> 2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation. >> >> >> For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system). >> >> I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right. >> >>> Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. >: >>> >>> >>> Dear Barry, >>> >>> Thank you very much for your information. >>> >>> It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI. >>> >>> $ export I_MPI_ASYNC_PROGRESS = 1 >>> $ export I_MPI_ASYNC_PROGRESS_PIN = >>> >>> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/ >>> >>> I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. >>> Best, >>> Viet. >>> >>> >>> >>> >>> On Sat, Jan 23, 2021 at 3:01 AM Barry Smith > wrote: >>> >>> https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html >>> >>> It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps. >>> >>> >>> >>>> On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. > wrote: >>>> >>>> >>>> Dear Victor and Berry, >>>> >>>> Thank you so much for your answers. >>>> >>>> I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. >>>> >>>> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); >>>> >>>> It was also a mistake to set the vector size too small. >>>> I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows >>>> >>>> The time used for the asynchronous calculation: 0.022043 >>>> + | u | = 10000. >>>> The time used for the synchronous calculation: 0.016188 >>>> + | b | = 10000. >>>> >>>> Asynchronous computation still takes a longer time. >>>> >>>> I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h >>>> >>>> I built Petsc by using the following script >>>> >>>> #!/usr/bin/bash >>>> set -e >>>> DATE="21.01.18" >>>> MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" >>>> MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" >>>> INSTL_DIR="${HOME}/local/petsc-3.14.3" >>>> BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" >>>> PETSC_DIR="${HOME}/tmp/petsc" >>>> >>>> cd ${PETSC_DIR} >>>> ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' >>>> >>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all >>>> make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install >>>> >>>> >>>> Intel 2018 also complies with the MPI-3 standard. >>>> >>>> Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? >>>> >>>> Sincerely, >>>> Viet. >>>> >>>> >>>> On Fri, Jan 22, 2021 at 11:20 AM Barry Smith > wrote: >>>> >>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>> >>>> How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. >>>> >>>> comb.c has >>>> >>>> /* >>>> Split phase global vector reductions with support for combining the >>>> communication portion of several operations. Using MPI-1.1 support only >>>> >>>> The idea for this and much of the initial code is contributed by >>>> Victor Eijkhout. >>>> >>>> Usage: >>>> VecDotBegin(Vec,Vec,PetscScalar *); >>>> VecNormBegin(Vec,NormType,PetscReal *); >>>> .... >>>> VecDotEnd(Vec,Vec,PetscScalar *); >>>> VecNormEnd(Vec,NormType,PetscReal *); >>>> >>>> Limitations: >>>> - The order of the xxxEnd() functions MUST be in the same order >>>> as the xxxBegin(). There is extensive error checking to try to >>>> insure that the user calls the routines in the correct order >>>> */ >>>> >>>> #include /*I "petscvec.h" I*/ >>>> >>>> static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) >>>> { >>>> PETSC_UNUSED PetscErrorCode ierr; >>>> >>>> PetscFunctionBegin; >>>> #if defined(PETSC_HAVE_MPI_IALLREDUCE) >>>> ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); >>>> #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) >>>> ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); >>>> #else >>>> ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); >>>> *request = MPI_REQUEST_NULL; >>>> #endif >>>> PetscFunctionReturn(0); >>>> } >>>> >>>> >>>> So first check if $PETSC_DIR/include/petscconf.h has >>>> >>>> PETSC_HAVE_MPI_IALLREDUCE >>>> >>>> if it does not then the standard MPI reduce is called. >>>> >>>> If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. >>>> >>>> Barry >>>> >>>> >>>>> On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. > wrote: >>>>> >>>>> >>>>> Hello Petsc developers and supporters, >>>>> >>>>> I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. >>>>> >>>>> >>>>> PetscLogDouble tt1,tt2; >>>>> KSP ksp; >>>>> //ierr = VecSet(c,one); >>>>> ierr = VecSet(c,one); >>>>> ierr = VecSet(u,one); >>>>> ierr = VecSet(b,one); >>>>> >>>>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); >>>>> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); >>>>> >>>>> >>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>> ierr = VecNormBegin(u,NORM_2,&norm1); >>>>> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); >>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>> ierr = VecNormEnd(u,NORM_2,&norm1); >>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>> >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); >>>>> >>>>> >>>>> ierr = PetscTime(&tt1);CHKERRQ(ierr); >>>>> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); >>>>> ierr = KSP_MatMult(ksp,A,c,Ac); >>>>> ierr = PetscTime(&tt2);CHKERRQ(ierr); >>>>> >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); >>>>> >>>>> >>>>> On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. >>>>> >>>>> The time used for the asynchronous calculation: 0.000203 >>>>> + |u| = 100. >>>>> The time used for the synchronous calculation: 0.000006 >>>>> + |b| = 100. >>>>> >>>>> Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? >>>>> >>>>> Thank you very much for anything you can provide. >>>>> Sincerely, >>>>> Viet. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wencel at gmail.com Thu Jan 28 10:11:41 2021 From: wencel at gmail.com (Lawrence Mitchell) Date: Thu, 28 Jan 2021 16:11:41 +0000 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> Message-ID: <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> > On 27 Jan 2021, at 16:30, Matthew Knepley wrote: > > This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on. Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. Although I guess Figure 2 from here https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue. Viet, I suspect that Matt thinks you should try and produce a figure like Figure 3 from that linked paper. Lawrence From jed at jedbrown.org Thu Jan 28 10:35:19 2021 From: jed at jedbrown.org (Jed Brown) Date: Thu, 28 Jan 2021 09:35:19 -0700 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> Message-ID: <87r1m5nho8.fsf@jedbrown.org> Lawrence Mitchell writes: >> On 27 Jan 2021, at 16:30, Matthew Knepley wrote: >> >> This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on. > > Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. I've been asking for it for years. They say if you care, you should just go run it. Never mind how wasteful that is, and the time commitment to doing so. I think they often avoid making a commitment because latency is super variable (depending on the partition you get and what other jobs are running elsewhere on the machine; Blue Gene famously didn't have that problem). Meanwhile, latency on cloud providers keeps dropping and they're sure to beat conventional HPC centers to publishing a dashboard of expected latency for different configurations. This page illustrates how hardware reductions scale much better than log(P). https://www.mcs.anl.gov/~fischer/gop/ > Although I guess Figure 2 from here https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue. > > Viet, I suspect that Matt thinks you should try and produce a figure like Figure 3 from that linked paper. > > Lawrence From hqhviet at tohoku.ac.jp Fri Jan 29 05:51:39 2021 From: hqhviet at tohoku.ac.jp (Viet H.Q.H.) Date: Fri, 29 Jan 2021 20:51:39 +0900 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <87r1m5nho8.fsf@jedbrown.org> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> <87r1m5nho8.fsf@jedbrown.org> Message-ID: Thank you everyone for your valuable materials and comments. Currently, I can use a maximum of 8 nodes on a computer system with a 10 Gb InfiniBand network. I am applying to use all the nodes in this computer system (about 300 nodes). It will take some time. I also hope 300 nodes are enough to check the effectiveness of a simple nonblocking computation test where the inner product overlaps the matrix-vector multiplication. Have a great weekend!! Viet On Fri, Jan 29, 2021 at 1:35 AM Jed Brown wrote: > Lawrence Mitchell writes: > > >> On 27 Jan 2021, at 16:30, Matthew Knepley wrote: > >> > >> This is very important to do _first_. It would probably only take you a > day to measure the Allreduce time on your target, say the whole machine you > run on. > > > > Why plots like this are not _absolutely standard_ on all HPC sites' > webpages is a source of continuing mystery to me. > > I've been asking for it for years. They say if you care, you should just > go run it. Never mind how wasteful that is, and the time commitment to > doing so. I think they often avoid making a commitment because latency is > super variable (depending on the partition you get and what other jobs are > running elsewhere on the machine; Blue Gene famously didn't have that > problem). > > Meanwhile, latency on cloud providers keeps dropping and they're sure to > beat conventional HPC centers to publishing a dashboard of expected latency > for different configurations. > > This page illustrates how hardware reductions scale much better than > log(P). > > https://www.mcs.anl.gov/~fischer/gop/ > > > Although I guess Figure 2 from here > https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue. > > > > Viet, I suspect that Matt thinks you should try and produce a figure > like Figure 3 from that linked paper. > > > > Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eijkhout at tacc.utexas.edu Fri Jan 29 09:09:22 2021 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Fri, 29 Jan 2021 15:09:22 +0000 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <87r1m5nho8.fsf@jedbrown.org> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> <87r1m5nho8.fsf@jedbrown.org> Message-ID: <96D03382-0B97-4011-A741-DE458EF051F9@tacc.utexas.edu> On , 2021Jan28, at 10:35, Jed Brown > wrote: Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. I've been asking for it for years. Curious. Why? What would it tell you? I mean, other than ?42?. I?ll ask around how this is done in TX. Considering all the tinkering Intel does to their MPI and what with all the UCX crap, we keep a close eye on MPI performance, especially collectives. latency is super variable (depending on the partition you get and what other jobs are running elsewhere on the machine; Blue Gene famously didn't have that problem). The harsh truth of fat-trees with static routing: https://arxiv.org/abs/1909.12195 Btw, I think we?ve turned off dynamic routing for now because of too many problems. Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jan 29 09:45:34 2021 From: jed at jedbrown.org (Jed Brown) Date: Fri, 29 Jan 2021 08:45:34 -0700 Subject: [petsc-users] How to confirm the performance of asynchronous computations In-Reply-To: <96D03382-0B97-4011-A741-DE458EF051F9@tacc.utexas.edu> References: <54C23024-BCC6-4053-8E6D-686A33E27FC7@petsc.dev> <8E50FAAA-2E67-4F75-AC80-CB9938DC8DA6@petsc.dev> <047EC928-FEEB-4C98-BD3A-B78570528FFD@gmail.com> <99EFDB69-AF37-47F5-8115-3484B773E41B@gmail.com> <87r1m5nho8.fsf@jedbrown.org> <96D03382-0B97-4011-A741-DE458EF051F9@tacc.utexas.edu> Message-ID: <87czxnagrl.fsf@jedbrown.org> Victor Eijkhout writes: > On , 2021Jan28, at 10:35, Jed Brown > wrote: > > Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. > > I've been asking for it for years. > > Curious. Why? What would it tell you? I mean, other than ?42?. Quite a bit about what sort of performance and variability can be expected for latency-sensitive jobs. Facilities publish HPL numbers despite far less relevance to application needs. People currently write proposals for Supercomputer X, and only find post-award that it delivers latency regressions compared to Humblecomputer Y and thus isn't suitable for the planned workload, so they fabricate a different workload to use the allocated hours.