<div dir="ltr">Thank you, I'll try that.<div><br></div><div>Best,</div><div>Li</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 5:34 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
From the code:<br>
<br>
if (snes->lagjacobian == -2) {<br>
snes->lagjacobian = -1;<br>
<br>
ierr = PetscInfo(snes,"Recomputing Jacobian/preconditioner because lag is -2 (means compute Jacobian, but then never again) \n");CHKERRQ(ierr);<br>
} else if (snes->lagjacobian == -1) {<br>
ierr = PetscInfo(snes,"Reusing Jacobian/preconditioner because lag is -1\n");CHKERRQ(ierr);<br>
ierr = PetscObjectTypeCompare((PetscObject)A,MATMFFD,&flag);CHKERRQ(ierr);<br>
if (flag) {<br>
ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
}<br>
PetscFunctionReturn(0);<br>
<br>
So it does what the manual page says. If you use -2 it will compute the Jacobian the next time it is needed but then will never compute it again,. This means it reuses the one computed for all time-steps. It can make sense depending on how much the Jacobian changes to use with -snes_mf_operator. <br>
<br>
To compute just once at the beginning of each new linear solver you can use for example 100 (this assumes that one nonlinear solver never needs more than 100 linear solvers.) This can also be used with -snes_mf_operator (or without).<br>
<br>
<br>
Barry<br>
<br>
<br>
> On Dec 3, 2019, at 6:33 AM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> <br>
> If use -snes_mf, the linear solver just never converges. And at the second time step, it reports an error as:<br>
> [0]PETSC ERROR: No support for this operation for this object type!<br>
> [0]PETSC ERROR: Mat type mffd!<br>
> <br>
> I still don't understand "-snes_lag_jacobian -2", from the manual, it seems Jacobian is computed only once by setting -1, but it is recomputed once by -2.<br>
> For a time-stepping scheme where each step calls SNESSolve, if using -snes_lag_jacobian -2, will the following time steps reuse the Jacobian built at the first time step?<br>
> <br>
> Best,<br>
> Li<br>
> <br>
> <br>
> <br>
> On Tue, Dec 3, 2019 at 12:10 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> <br>
> <br>
> > On Dec 2, 2019, at 2:30 PM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> > <br>
> > -snes_mf fails to converge in my case, but -ds_snes_mf_operator works, when the original analytic matrix is still used as the preconditioner. <br>
> > The timing is several times greater than using the analytic matrix for both Jacobian and preconditioner.<br>
> <br>
> ok, how does -snes_mf fail to converge? -ksp_monitor ? does the linear solver just never converge? <br>
> <br>
> Using -snes_mf_operator will also build the Jacobian so in your case doesn't make much sense by itself since it is very expensive<br>
> <br>
> > <br>
> > For an implicit time-stepping scheme, if using -snes_lag_jacobian -2, is the Jacobian built only twice at the first time step then it is used for all later time steps? Or it is built twice at every time step?<br>
> <br>
> Check the manual page for SNESSolve it should just compute the Jacobian once and reuse it forever. <br>
> <br>
> You can also try -snes_mf -snes_lag_jacobian -2 which should compute the Jacobian once, use that original one to build the preconditioner once and reuse the same preconditioner but use the matrix free to define the operator.<br>
> <br>
> <br>
> <br>
> Barry<br>
> <br>
> > <br>
> > Regards,<br>
> > Li<br>
> > <br>
> > On Mon, Dec 2, 2019 at 6:02 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> > <br>
> > Ok it is spending 99+ percent of the time computing the Jacobians.<br>
> > <br>
> > <br>
> > MatFDColorApply 106 1.0 8.7622e+03 1.0 1.31e+08 1.1 6.9e+07 2.6e+03 1.1e+06 99 0 97 81 99 99 0 97 81 99 1<br>
> > MatFDColorFunc 60950 1.0 8.7560e+03 1.0 0.00e+00 0.0 6.9e+07 2.6e+03 1.1e+06 99 0 97 81 99 99 0 97 81 99 0<br>
> > <br>
> > It is requiring on average 12 KSP iterations per linear solve so the resulting linear system appears well conditioned, this means even if you compute the Jacobian analytically likely most of the time in the run will still be computing Jacobians.<br>
> > <br>
> > Try using -snes_mf with the logging and see what happens. You can also try -snes_lag_jacobian_persists -snes_lag_jacobian -2<br>
> > <br>
> > Note there may be other ways of avoiding the costly computation of the Jacobian at each Newton step.<br>
> > <br>
> > Barry<br>
> > <br>
> > <br>
> > > On Dec 2, 2019, at 6:38 AM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> > > <br>
> > > Dear Barry,<br>
> > > <br>
> > > Here is my log.<br>
> > > Because I am using libMesh built on PETSc, there is more information from libMesh in the log file.<br>
> > > I ran 13 time steps for the simulation so there are repeated snes_view info.<br>
> > > The algorithm is simply NKS.<br>
> > > <br>
> > > Cheers,<br>
> > > Li<br>
> > > <br>
> > > On Mon, Dec 2, 2019 at 4:55 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> > > <br>
> > > Please send a run with optimization turned on (--with-debugging=0 in ./configure) and -log_view without the actual timing information we are just guessing where the time is spent.<br>
> > > <br>
> > > If your problem has a natural block size then using baij should be a bit faster than aij, but not dramatically better<br>
> > > <br>
> > > Barry<br>
> > > <br>
> > > <br>
> > > > On Dec 2, 2019, at 4:30 AM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> > > > <br>
> > > > Thank you very much! It looks forming an analytic Jacobian is the only choice.<br>
> > > > <br>
> > > > Best,<br>
> > > > Li<br>
> > > > <br>
> > > > On Mon, Dec 2, 2019 at 3:21 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> > > > On Mon, Dec 2, 2019 at 4:04 AM Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> > > > Thank you for your reply.<br>
> > > > <br>
> > > > The matrix is small with only 67500 rows, but is relatively dense since a second-order discontinuous Galerkin FEM is used, nonzeros=23,036,400.<br>
> > > > <br>
> > > > This is very dense, 0.5% fill or 340 nonzeros per row.<br>
> > > > <br>
> > > > The number of colors is 539 as shown by using -mat_fd_coloring_view:<br>
> > > > <br>
> > > > Coloring is not appropriate for this matrix since you have enormous dense blocks (I am guessing). It could work if you statically<br>
> > > > condense them out or had a fast analytic Jacobian. With 540 colors, it takes 540 matvecs to generate the action of the Jacobian.<br>
> > > > <br>
> > > > Thanks,<br>
> > > > <br>
> > > > Matt<br>
> > > > <br>
> > > > MatFDColoring Object: 64 MPI processes<br>
> > > > type not yet set<br>
> > > > Error tolerance=1.49012e-08<br>
> > > > Umin=1.49012e-06<br>
> > > > Number of colors=539<br>
> > > > Information for color 0<br>
> > > > Number of columns 1<br>
> > > > 378<br>
> > > > Number of rows 756<br>
> > > > 0 1188<br>
> > > > 1 1188<br>
> > > > 2 1188<br>
> > > > 3 1188<br>
> > > > 4 1188<br>
> > > > 5 1188<br>
> > > > ...<br>
> > > > <br>
> > > > Is this normal?<br>
> > > > When using MCFD, is there any difference using mpiaij and mpibaij?<br>
> > > > <br>
> > > > Best,<br>
> > > > Li<br>
> > > > <br>
> > > > On Mon, Dec 2, 2019 at 10:03 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> > > > <br>
> > > > How many colors is it requiring? And how long is the MatGetColoring() taking? Are you running in parallel? The MatGetColoring() MATCOLORINGSL uses a sequential coloring algorithm so if your matrix is large and parallel the coloring will take a long time. The parallel colorings are MATCOLORINGGREEDY and MATCOLORINGJP<br>
> > > > <br>
> > > > Barry<br>
> > > > <br>
> > > > <br>
> > > > > On Dec 1, 2019, at 12:56 AM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> > > > > <br>
> > > > > Dear Developers,<br>
> > > > > <br>
> > > > > I tried to use the multi-color finite-difference (MC-FD) method for constructing the Jacobians. However, I find it is very slow compared to the exact Jacobian. <br>
> > > > > My implementation of MC-FD Jacobian is posted below, would you please check whether I am correct? Anything missed? Thank you!<br>
> > > > > <br>
> > > > > ////////// Setup phase:<br>
> > > > > MatStructure flag;<br>
> > > > > ISColoring iscoloring;<br>
> > > > > ierr = MatGetColoring(Jac,MATCOLORINGSL,&iscoloring);<br>
> > > > > ierr = MatFDColoringCreate(Jac,iscoloring,&this->matfdcoloring);<br>
> > > > > ierr = MatFDColoringSetFunction(this->matfdcoloring,(PetscErrorCode (*)(void))__libmesh_petsc_snes_residual,(void *)this);<br>
> > > > > ierr = MatFDColoringSetFromOptions(this->matfdcoloring);<br>
> > > > > ierr = ISColoringDestroy(&iscoloring);<br>
> > > > > <br>
> > > > > //////////// Apply:<br>
> > > > > ierr = MatZeroEntries(*jac);CHKERRQ(ierr);<br>
> > > > > ierr = MatFDColoringApply(*jac,solver->matfdcoloring,x,msflag,snes);<br>
> > > > > <br>
> > > > > Best regards,<br>
> > > > > Li Luo<br>
> > > > > <br>
> > > > > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
> > > > <br>
> > > > <br>
> > > > <br>
> > > > -- <br>
> > > > Postdoctoral Fellow<br>
> > > > Extreme Computing Research Center<br>
> > > > King Abdullah University of Science & Technology<br>
> > > > <a href="https://sites.google.com/site/rolyliluo/" rel="noreferrer" target="_blank">https://sites.google.com/site/rolyliluo/</a><br>
> > > > <br>
> > > > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
> > > > <br>
> > > > <br>
> > > > -- <br>
> > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > > > -- Norbert Wiener<br>
> > > > <br>
> > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> > > > <br>
> > > > <br>
> > > > -- <br>
> > > > Postdoctoral Fellow<br>
> > > > Extreme Computing Research Center<br>
> > > > King Abdullah University of Science & Technology<br>
> > > > <a href="https://sites.google.com/site/rolyliluo/" rel="noreferrer" target="_blank">https://sites.google.com/site/rolyliluo/</a><br>
> > > > <br>
> > > > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
> > > <br>
> > > <br>
> > > <br>
> > > -- <br>
> > > Postdoctoral Fellow<br>
> > > Extreme Computing Research Center<br>
> > > King Abdullah University of Science & Technology<br>
> > > <a href="https://sites.google.com/site/rolyliluo/" rel="noreferrer" target="_blank">https://sites.google.com/site/rolyliluo/</a><br>
> > > <br>
> > > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<slurm-13575838.out><br>
> > <br>
> > <br>
> > <br>
> > -- <br>
> > Postdoctoral Fellow<br>
> > Extreme Computing Research Center<br>
> > King Abdullah University of Science & Technology<br>
> > <a href="https://sites.google.com/site/rolyliluo/" rel="noreferrer" target="_blank">https://sites.google.com/site/rolyliluo/</a><br>
> > <br>
> > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
> <br>
> <br>
> <br>
> -- <br>
> Postdoctoral Fellow<br>
> Extreme Computing Research Center<br>
> King Abdullah University of Science & Technology<br>
> <a href="https://sites.google.com/site/rolyliluo/" rel="noreferrer" target="_blank">https://sites.google.com/site/rolyliluo/</a><br>
> <br>
> This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><p style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif;border:medium none;margin:6pt 0in 0pt"><span style="font-family:"trebuchet ms",sans-serif"><span style="line-height:normal"><font size="2">Postdoctoral Fellow<br></font></span></span></p><font style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif" size="2"><font face="trebuchet ms, sans-serif">Extreme Computing Research Center</font><br><font face="trebuchet ms, sans-serif">King Abdullah University of Science & Technology</font></font><div><font size="2"><font face="trebuchet ms, sans-serif"><font color="#444444"><a href="https://sites.google.com/site/rolyliluo/" target="_blank">https://sites.google.com/site/rolyliluo/</a></font><br></font></font></div></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font>