<div dir="ltr">Thank you very much! It looks forming an analytic Jacobian is the only choice.<div><br></div><div>Best,</div><div>Li</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 2, 2019 at 3:21 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 2, 2019 at 4:04 AM Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for your reply.<div><br></div><div>The matrix is small with only 67500 rows, but is relatively dense since a second-order discontinuous Galerkin FEM is used, nonzeros=23,036,400.</div></div></blockquote><div><br></div><div>This is very dense, 0.5% fill or 340 nonzeros per row.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The number of colors is 539 as shown by using -mat_fd_coloring_view:</div></div></blockquote><div><br></div><div>Coloring is not appropriate for this matrix since you have enormous dense blocks (I am guessing). It could work if you statically</div><div>condense them out or had a fast analytic Jacobian. With 540 colors, it takes 540 matvecs to generate the action of the Jacobian.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>MatFDColoring Object: 64 MPI processes<br> type not yet set<br> Error tolerance=1.49012e-08<br> Umin=1.49012e-06<br> Number of colors=539<br> Information for color 0<br> Number of columns 1<br> 378<br> Number of rows 756<br> 0 1188<br> 1 1188<br> 2 1188<br> 3 1188<br> 4 1188<br> 5 1188<br></div><div> ...</div><div><br></div><div>Is this normal?</div><div>When using MCFD, is there any difference using mpiaij and mpibaij?</div><div><br></div><div>Best,</div><div>Li</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 2, 2019 at 10:03 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
How many colors is it requiring? And how long is the MatGetColoring() taking? Are you running in parallel? The MatGetColoring() MATCOLORINGSL uses a sequential coloring algorithm so if your matrix is large and parallel the coloring will take a long time. The parallel colorings are MATCOLORINGGREEDY and MATCOLORINGJP<br>
<br>
Barry<br>
<br>
<br>
> On Dec 1, 2019, at 12:56 AM, Li Luo <<a href="mailto:li.luo@kaust.edu.sa" target="_blank">li.luo@kaust.edu.sa</a>> wrote:<br>
> <br>
> Dear Developers,<br>
> <br>
> I tried to use the multi-color finite-difference (MC-FD) method for constructing the Jacobians. However, I find it is very slow compared to the exact Jacobian. <br>
> My implementation of MC-FD Jacobian is posted below, would you please check whether I am correct? Anything missed? Thank you!<br>
> <br>
> ////////// Setup phase:<br>
> MatStructure flag;<br>
> ISColoring iscoloring;<br>
> ierr = MatGetColoring(Jac,MATCOLORINGSL,&iscoloring);<br>
> ierr = MatFDColoringCreate(Jac,iscoloring,&this->matfdcoloring);<br>
> ierr = MatFDColoringSetFunction(this->matfdcoloring,(PetscErrorCode (*)(void))__libmesh_petsc_snes_residual,(void *)this);<br>
> ierr = MatFDColoringSetFromOptions(this->matfdcoloring);<br>
> ierr = ISColoringDestroy(&iscoloring);<br>
> <br>
> //////////// Apply:<br>
> ierr = MatZeroEntries(*jac);CHKERRQ(ierr);<br>
> ierr = MatFDColoringApply(*jac,solver->matfdcoloring,x,msflag,snes);<br>
> <br>
> Best regards,<br>
> Li Luo<br>
> <br>
> This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><p style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif;border:medium none;margin:6pt 0in 0pt"><span style="font-family:"trebuchet ms",sans-serif"><span style="line-height:normal"><font size="2">Postdoctoral Fellow<br></font></span></span></p><font style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif" size="2"><font face="trebuchet ms, sans-serif">Extreme Computing Research Center</font><br><font face="trebuchet ms, sans-serif">King Abdullah University of Science & Technology</font></font><div><font size="2"><font face="trebuchet ms, sans-serif"><font color="#444444"><a href="https://sites.google.com/site/rolyliluo/" target="_blank">https://sites.google.com/site/rolyliluo/</a></font><br></font></font></div></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><p style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif;border:medium none;margin:6pt 0in 0pt"><span style="font-family:"trebuchet ms",sans-serif"><span style="line-height:normal"><font size="2">Postdoctoral Fellow<br></font></span></span></p><font style="color:rgb(68,68,68);font-family:Verdana,Helvetica,Arial,sans-serif" size="2"><font face="trebuchet ms, sans-serif">Extreme Computing Research Center</font><br><font face="trebuchet ms, sans-serif">King Abdullah University of Science & Technology</font></font><div><font size="2"><font face="trebuchet ms, sans-serif"><font color="#444444"><a href="https://sites.google.com/site/rolyliluo/" target="_blank">https://sites.google.com/site/rolyliluo/</a></font><br></font></font></div></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font>