<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Danyang :</div><div class="gmail_quote">I tested your data.</div><div class="gmail_quote">Your matrices encountered zero pivots, e.g.</div><div class="gmail_quote"><div class="gmail_quote">petsc/src/ksp/ksp/examples/tutorials (master)</div><div class="gmail_quote">$ mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor -ksp_error_if_not_converged</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote">[15]PETSC ERROR: Zero pivot in LU factorization: <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot">http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot</a></div><div class="gmail_quote">[15]PETSC ERROR: Zero pivot row 1249 value 2.05808e-14 tolerance 2.22045e-14</div><div class="gmail_quote">...</div><div class="gmail_quote"><br></div><div class="gmail_quote">Adding option '-sub_pc_factor_shift_type nonzero', I got</div><div class="gmail_quote"><div class="gmail_quote">mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero -mat_view ascii::ascii_info</div><div class="gmail_quote"><br></div><div class="gmail_quote">Mat Object: 24 MPI processes</div><div class="gmail_quote"> type: mpiaij</div><div class="gmail_quote"> rows=450000, cols=450000</div><div class="gmail_quote"> total: nonzeros=6991400, allocated nonzeros=6991400</div><div class="gmail_quote"> total number of mallocs used during MatSetValues calls =0</div><div class="gmail_quote"> not using I-node (on process 0) routines</div><div class="gmail_quote"> 0 KSP Residual norm 5.849777711755e+01</div><div class="gmail_quote"> 1 KSP Residual norm 6.824179430230e-01</div><div class="gmail_quote"> 2 KSP Residual norm 3.994483555787e-02</div><div class="gmail_quote"> 3 KSP Residual norm 6.085841461433e-03</div><div class="gmail_quote"> 4 KSP Residual norm 8.876162583511e-04</div><div class="gmail_quote"> 5 KSP Residual norm 9.407780665278e-05</div><div class="gmail_quote">Number of iterations = 5</div><div class="gmail_quote">Residual norm 0.00542891</div><div class="gmail_quote"><br></div><div class="gmail_quote">Hong</div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>Yes. The matrix is 450000x450000 sparse. The hypre takes hundreds
of iterates, not for all but in most of the timesteps. The matrix
is not well conditioned, with nonzero entries range from 1.0e-29
to 1.0e2. I also made double check if there is anything wrong in
the parallel version, however, the matrix is the same with
sequential version except some round error which is relatively
very small. Usually for those not well conditioned matrix, direct
solver should be faster than iterative solver, right? But when I
use the sequential iterative solver with ILU prec developed almost
20 years go by others, the solver converge fast with appropriate
factorization level. In other words, when I use 24 processor using
hypre, the speed is almost the same as as the old sequential
iterative solver using 1 processor.</p>
<p>I use most of the default configuration for the general case with
pretty good speedup. And I am not sure if I miss something for
this problem.</p>
<p>Thanks,</p>
<p>Danyang<br>
</p><div><div class="gmail-h5">
<br>
<div class="gmail-m_1680376947984306833moz-cite-prefix">On 17-05-24 11:12 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, May 24, 2017 at 12:50 PM,
Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew and Barry,</p>
<p>Thanks for the quick response. <br>
</p>
<p>I also tried superlu and mumps, both work but it is
about four times slower than ILU(dt) prec through
hypre, with 24 processors I have tested.</p>
</div>
</blockquote>
<div>You mean the total time is 4x? And you are taking
hundreds of iterates? That seems hard to believe, unless
you are dropping</div>
<div>a huge number of elements. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> When I look into the convergence information, the
method using ILU(dt) still takes 200 to 3000 linear
iterations for each newton iteration. One reason is
this equation is hard to solve. As for the general
cases, the same method works awesome and get very good
speedup.</p>
</div>
</blockquote>
<div>I do not understand what you mean here. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> I also doubt if I use hypre correctly for this case.
Is there anyway to check this problem, or is it
possible to increase the factorization level through
hypre?</p>
</div>
</blockquote>
<div>I don't know.</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Danyang<br>
</p>
<br>
<div class="gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
17-05-24 04:59 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, May 24, 2017 at
2:21 AM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear All,<br>
<br>
I use PCFactorSetLevels for ILU and
PCFactorSetFill for other preconditioning in
my code to help solve the problems that the
default option is hard to solve. However, I
found the latter one, PCFactorSetFill does not
take effect for my problem. The matrices and
rhs as well as the solutions are attached from
the link below. I obtain the solution using
hypre preconditioner and it takes 7 and 38
iterations for matrix 1 and matrix 2. However,
if I use other preconditioner, the solver just
failed at the first matrix. I have tested this
matrix using the native sequential solver (not
PETSc) with ILU preconditioning. If I set the
incomplete factorization level to 0, this
sequential solver will take more than 100
iterations. If I increase the factorization
level to 1 or more, it just takes several
iterations. This remind me that the PC factor
for this matrices should be increased.
However, when I tried it in PETSc, it just
does not work.<br>
<br>
Matrix and rhs can be obtained from the link
below.<br>
<br>
<a href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R" rel="noreferrer" target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
<br>
Would anyone help to check if you can make
this work by increasing the PC factor level or
fill?<br>
</blockquote>
<div><br>
</div>
<div>We have ILU(k) supported in serial. However
ILU(dt) which takes a tolerance only works
through Hypre</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html" target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
<div><br>
</div>
<div>I recommend you try SuperLU or MUMPS, which
can both be downloaded automatically by
configure, and</div>
<div>do a full sparse LU.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks and
regards,<br>
<br>
Danyang<br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<span class="gmail-m_1680376947984306833HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail-m_1680376947984306833gmail_signature">
<div dir="ltr">
<div>What most experimenters take for granted before they
begin their experiments is infinitely more interesting
than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~<wbr>mk51/</a><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div>