<div dir="ltr"><span style="font-size:12.8px">Remove your option '</span><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">-vecload_block_size 10'.</span><br><div><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap"><br></span></div><div><font color="#000000" face="Courier New, Courier, monospace, arial, sans-serif"><span style="font-size:14px;white-space:pre-wrap">Hong</span></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 3:06 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Dear Hong,</p>
<p>I just tested with different number of processors for the same
matrix. It sometimes got "ERROR: Arguments are incompatible" for
different number of processors. It works fine using 4, 8, or 24
processors, but failed with "ERROR: Arguments are incompatible"
using 16 or 48 processors. The error information is attached. I
tested this on my local computer with 6 cores 12 threads. Any
suggestion on this?</p>
<p>Thanks,<br>
</p>
<p>Danyang<br>
</p><div><div class="h5">
<br>
<div class="m_9041936809186897311moz-cite-prefix">On 17-05-24 12:28 PM, Danyang Su wrote:<br>
</div>
<blockquote type="cite">
<p>Hi Hong,</p>
<p>Awesome. Thanks for testing the case. I will try your options
for the code and get back to you later.</p>
<p>Regards,</p>
<p>Danyang<br>
</p>
<br>
<div class="m_9041936809186897311moz-cite-prefix">On 17-05-24 12:21 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote">I tested your data.</div>
<div class="gmail_quote">Your matrices encountered zero
pivots, e.g.</div>
<div class="gmail_quote">
<div class="gmail_quote">petsc/src/ksp/ksp/examples/<wbr>tutorials
(master)</div>
<div class="gmail_quote">$ mpiexec -n 24 ./ex10 -f0
a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">[15]PETSC ERROR: Zero pivot in
LU factorization: <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>zeropivot</a></div>
<div class="gmail_quote">[15]PETSC ERROR: Zero pivot row
1249 value 2.05808e-14 tolerance 2.22045e-14</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Adding option
'-sub_pc_factor_shift_type nonzero', I got</div>
<div class="gmail_quote">
<div class="gmail_quote">mpiexec -n 24 ./ex10 -f0
a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged
-sub_pc_factor_shift_type nonzero -mat_view
ascii::ascii_info</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Mat Object: 24 MPI processes</div>
<div class="gmail_quote"> type: mpiaij</div>
<div class="gmail_quote"> rows=450000, cols=450000</div>
<div class="gmail_quote"> total: nonzeros=6991400,
allocated nonzeros=6991400</div>
<div class="gmail_quote"> total number of mallocs
used during MatSetValues calls =0</div>
<div class="gmail_quote"> not using I-node (on
process 0) routines</div>
<div class="gmail_quote"> 0 KSP Residual norm
5.849777711755e+01</div>
<div class="gmail_quote"> 1 KSP Residual norm
6.824179430230e-01</div>
<div class="gmail_quote"> 2 KSP Residual norm
3.994483555787e-02</div>
<div class="gmail_quote"> 3 KSP Residual norm
6.085841461433e-03</div>
<div class="gmail_quote"> 4 KSP Residual norm
8.876162583511e-04</div>
<div class="gmail_quote"> 5 KSP Residual norm
9.407780665278e-05</div>
<div class="gmail_quote">Number of iterations = 5</div>
<div class="gmail_quote">Residual norm 0.00542891</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>Yes. The matrix is 450000x450000 sparse. The hypre
takes hundreds of iterates, not for all but in most
of the timesteps. The matrix is not well
conditioned, with nonzero entries range from 1.0e-29
to 1.0e2. I also made double check if there is
anything wrong in the parallel version, however, the
matrix is the same with sequential version except
some round error which is relatively very small.
Usually for those not well conditioned matrix,
direct solver should be faster than iterative
solver, right? But when I use the sequential
iterative solver with ILU prec developed almost 20
years go by others, the solver converge fast with
appropriate factorization level. In other words,
when I use 24 processor using hypre, the speed is
almost the same as as the old sequential iterative
solver using 1 processor.</p>
<p>I use most of the default configuration for the
general case with pretty good speedup. And I am not
sure if I miss something for this problem.</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
<div>
<div class="m_9041936809186897311gmail-h5"> <br>
<div class="m_9041936809186897311gmail-m_1680376947984306833moz-cite-prefix">On
17-05-24 11:12 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, May 24,
2017 at 12:50 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew and Barry,</p>
<p>Thanks for the quick response. <br>
</p>
<p>I also tried superlu and mumps,
both work but it is about four times
slower than ILU(dt) prec through
hypre, with 24 processors I have
tested.</p>
</div>
</blockquote>
<div>You mean the total time is 4x? And
you are taking hundreds of iterates?
That seems hard to believe, unless you
are dropping</div>
<div>a huge number of elements. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> When I look into the convergence
information, the method using
ILU(dt) still takes 200 to 3000
linear iterations for each newton
iteration. One reason is this
equation is hard to solve. As for
the general cases, the same method
works awesome and get very good
speedup.</p>
</div>
</blockquote>
<div>I do not understand what you mean
here. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> I also doubt if I use hypre
correctly for this case. Is there
anyway to check this problem, or is
it possible to increase the
factorization level through hypre?</p>
</div>
</blockquote>
<div>I don't know.</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Danyang<br>
</p>
<br>
<div class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
17-05-24 04:59 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed,
May 24, 2017 at 2:21 AM,
Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear All,<br>
<br>
I use PCFactorSetLevels for
ILU and PCFactorSetFill for
other preconditioning in my
code to help solve the
problems that the default
option is hard to solve.
However, I found the latter
one, PCFactorSetFill does
not take effect for my
problem. The matrices and
rhs as well as the solutions
are attached from the link
below. I obtain the solution
using hypre preconditioner
and it takes 7 and 38
iterations for matrix 1 and
matrix 2. However, if I use
other preconditioner, the
solver just failed at the
first matrix. I have tested
this matrix using the native
sequential solver (not
PETSc) with ILU
preconditioning. If I set
the incomplete factorization
level to 0, this sequential
solver will take more than
100 iterations. If I
increase the factorization
level to 1 or more, it just
takes several iterations.
This remind me that the PC
factor for this matrices
should be increased.
However, when I tried it in
PETSc, it just does not
work.<br>
<br>
Matrix and rhs can be
obtained from the link
below.<br>
<br>
<a href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R" rel="noreferrer" target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
<br>
Would anyone help to check
if you can make this work by
increasing the PC factor
level or fill?<br>
</blockquote>
<div><br>
</div>
<div>We have ILU(k) supported
in serial. However ILU(dt)
which takes a tolerance only
works through Hypre</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html" target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
<div><br>
</div>
<div>I recommend you try
SuperLU or MUMPS, which can
both be downloaded
automatically by configure,
and</div>
<div>do a full sparse LU.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks and regards,<br>
<br>
Danyang<br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<span class="m_9041936809186897311gmail-m_1680376947984306833HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their
experiments is
infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_9041936809186897311gmail-m_1680376947984306833gmail_signature">
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>