<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Hong,</p>
<p>Awesome. Thanks for testing the case. I will try your options for
the code and get back to you later.</p>
<p>Regards,</p>
<p>Danyang<br>
</p>
<br>
<div class="moz-cite-prefix">On 17-05-24 12:21 PM, Hong wrote:<br>
</div>
<blockquote
cite="mid:CAGCphBvLSvn8NXFpQ2qjBtyXwtHA0dqziJVSdZSVtu9BtVxZ5g@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote">I tested your data.</div>
<div class="gmail_quote">Your matrices encountered zero
pivots, e.g.</div>
<div class="gmail_quote">
<div class="gmail_quote">petsc/src/ksp/ksp/examples/tutorials
(master)</div>
<div class="gmail_quote">$ mpiexec -n 24 ./ex10 -f0
a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">[15]PETSC ERROR: Zero pivot in LU
factorization: <a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot">http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot</a></div>
<div class="gmail_quote">[15]PETSC ERROR: Zero pivot row
1249 value 2.05808e-14 tolerance 2.22045e-14</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Adding option
'-sub_pc_factor_shift_type nonzero', I got</div>
<div class="gmail_quote">
<div class="gmail_quote">mpiexec -n 24 ./ex10 -f0
a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged -sub_pc_factor_shift_type
nonzero -mat_view ascii::ascii_info</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Mat Object: 24 MPI processes</div>
<div class="gmail_quote"> type: mpiaij</div>
<div class="gmail_quote"> rows=450000, cols=450000</div>
<div class="gmail_quote"> total: nonzeros=6991400,
allocated nonzeros=6991400</div>
<div class="gmail_quote"> total number of mallocs used
during MatSetValues calls =0</div>
<div class="gmail_quote"> not using I-node (on
process 0) routines</div>
<div class="gmail_quote"> 0 KSP Residual norm
5.849777711755e+01</div>
<div class="gmail_quote"> 1 KSP Residual norm
6.824179430230e-01</div>
<div class="gmail_quote"> 2 KSP Residual norm
3.994483555787e-02</div>
<div class="gmail_quote"> 3 KSP Residual norm
6.085841461433e-03</div>
<div class="gmail_quote"> 4 KSP Residual norm
8.876162583511e-04</div>
<div class="gmail_quote"> 5 KSP Residual norm
9.407780665278e-05</div>
<div class="gmail_quote">Number of iterations = 5</div>
<div class="gmail_quote">Residual norm 0.00542891</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>Yes. The matrix is 450000x450000 sparse. The hypre
takes hundreds of iterates, not for all but in most of
the timesteps. The matrix is not well conditioned,
with nonzero entries range from 1.0e-29 to 1.0e2. I
also made double check if there is anything wrong in
the parallel version, however, the matrix is the same
with sequential version except some round error which
is relatively very small. Usually for those not well
conditioned matrix, direct solver should be faster
than iterative solver, right? But when I use the
sequential iterative solver with ILU prec developed
almost 20 years go by others, the solver converge fast
with appropriate factorization level. In other words,
when I use 24 processor using hypre, the speed is
almost the same as as the old sequential iterative
solver using 1 processor.</p>
<p>I use most of the default configuration for the
general case with pretty good speedup. And I am not
sure if I miss something for this problem.</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
<div>
<div class="gmail-h5"> <br>
<div
class="gmail-m_1680376947984306833moz-cite-prefix">On
17-05-24 11:12 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, May 24, 2017
at 12:50 PM, Danyang Su <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew and Barry,</p>
<p>Thanks for the quick response. <br>
</p>
<p>I also tried superlu and mumps, both
work but it is about four times slower
than ILU(dt) prec through hypre, with
24 processors I have tested.</p>
</div>
</blockquote>
<div>You mean the total time is 4x? And you
are taking hundreds of iterates? That
seems hard to believe, unless you are
dropping</div>
<div>a huge number of elements. <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> When I look into the convergence
information, the method using ILU(dt)
still takes 200 to 3000 linear
iterations for each newton iteration.
One reason is this equation is hard to
solve. As for the general cases, the
same method works awesome and get very
good speedup.</p>
</div>
</blockquote>
<div>I do not understand what you mean
here. <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> I also doubt if I use hypre
correctly for this case. Is there
anyway to check this problem, or is it
possible to increase the factorization
level through hypre?</p>
</div>
</blockquote>
<div>I don't know.</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Danyang<br>
</p>
<br>
<div
class="gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
17-05-24 04:59 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed,
May 24, 2017 at 2:21 AM, Danyang
Su <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Dear
All,<br>
<br>
I use PCFactorSetLevels for
ILU and PCFactorSetFill for
other preconditioning in my
code to help solve the
problems that the default
option is hard to solve.
However, I found the latter
one, PCFactorSetFill does not
take effect for my problem.
The matrices and rhs as well
as the solutions are attached
from the link below. I obtain
the solution using hypre
preconditioner and it takes 7
and 38 iterations for matrix 1
and matrix 2. However, if I
use other preconditioner, the
solver just failed at the
first matrix. I have tested
this matrix using the native
sequential solver (not PETSc)
with ILU preconditioning. If I
set the incomplete
factorization level to 0, this
sequential solver will take
more than 100 iterations. If I
increase the factorization
level to 1 or more, it just
takes several iterations. This
remind me that the PC factor
for this matrices should be
increased. However, when I
tried it in PETSc, it just
does not work.<br>
<br>
Matrix and rhs can be obtained
from the link below.<br>
<br>
<a moz-do-not-send="true"
href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R"
rel="noreferrer"
target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
<br>
Would anyone help to check if
you can make this work by
increasing the PC factor level
or fill?<br>
</blockquote>
<div><br>
</div>
<div>We have ILU(k) supported in
serial. However ILU(dt) which
takes a tolerance only works
through Hypre</div>
<div><br>
</div>
<div> <a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html"
target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
<div><br>
</div>
<div>I recommend you try SuperLU
or MUMPS, which can both be
downloaded automatically by
configure, and</div>
<div>do a full sparse LU.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Thanks and regards,<br>
<br>
Danyang<br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<span
class="gmail-m_1680376947984306833HOEnZb"><font
color="#888888">
<div><br>
</div>
-- <br>
<div
class="gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their experiments
is infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div
class="gmail-m_1680376947984306833gmail_signature">
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/"
target="_blank">http://www.caam.rice.edu/~<wbr>mk51/</a><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>