<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Dear Hong and Barry,</p>
<p>I have implemented this option in the code, as we also need to
use configuration from file for convenience. When I run the code
using options, it works fine, however, when I run the code using
configuration file, it does not work. The code has two set of
equations, flow and reactive, with prefix been set to "flow_" and
"react_". When I run the code using</p>
<p>mpiexec -n 4 ../executable -flow_sub_pc_factor_shift_type nonzero
-react_sub_pc_factor_shift_type nonzero</p>
<p>it works. However, if I run using</p>
<p>mpiexec -n 4 ../executable <br>
</p>
<p>and let the executable file read the options from file, it just
does not work at "call
PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr) or none,
positive_definite ...". Do I miss something here?<br>
</p>
<p>Below is the pseudo code I have used for flow equations, similar
for reactive equations. <br>
</p>
<p> call
MatCreateAIJ(Petsc_Comm_World,nndof,nndof,nngbldof, &
<br>
nngbldof,d_nz,PETSC_NULL_INTEGER,o_nz, &
<br>
PETSC_NULL_INTEGER,a_flow,ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call MatSetFromOptions(a_flow,ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPCreate(Petsc_Comm_World, ksp_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPAppendOptionsPrefix(ksp_flow,"flow_",ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call
KSPSetInitialGuessNonzero(ksp_flow, &
<br>
b_initial_guess_nonzero_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call
KSPSetInitialGuessNonzero(ksp_flow, &
<br>
b_initial_guess_nonzero_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPSetDM(ksp_flow,dmda_flow%da,ierr)
<br>
CHKERRQ(ierr)
<br>
call KSPSetDMActive(ksp_flow,PETSC_FALSE,ierr)
<br>
CHKERRQ(ierr)<br>
</p>
<p> !!!!*********CHECK IF READ OPTION FROM FILE*********!!!!<br>
if (read_option_from_file) then<br>
<br>
call KSPSetType(ksp_flow, KSPGMRES, ierr) !or KSPBCGS
or others...
<br>
CHKERRQ(ierr)<br>
<br>
call KSPGetPC(ksp_flow, pc_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call PCSetType(pc_flow,PCBJACOBI, ierr) !or PCILU
or PCJACOBI or PCHYPRE ...
<br>
CHKERRQ(ierr)<br>
<br>
call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO,
ierr) or none, positive_definite ...
<br>
CHKERRQ(ierr)<br>
<br>
end if<br>
<br>
call
PCFactorGetMatSolverPackage(pc_flow,solver_pkg_flow,ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call
compute_jacobian(rank,dmda_flow%da, &
<br>
a_flow,a_in,ia_in,ja_in,nngl_in, &
<br>
row_idx_l2pg,col_idx_l2pg, &
<br>
b_non_interlaced)<br>
call KSPSetFromOptions(ksp_flow,ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSetUp(ksp_flow,ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSetUpOnBlocks(ksp_flow,ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSolve(ksp_flow,b_flow,x_flow,ierr)
<br>
CHKERRQ(ierr) <br>
</p>
<p> <br>
Thanks and Regards,<br>
</p>
<p>Danyang</p>
<div class="moz-cite-prefix">On 17-05-24 06:32 PM, Hong wrote:<br>
</div>
<blockquote
cite="mid:CAGCphBvmU1jyK60QKsE2=K_5TEe_8+U+=wHcqB3V1ssbZGwQ+w@mail.gmail.com"
type="cite">
<div dir="ltr"><span style="font-size:12.8px">Remove your option '</span><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">-vecload_block_size 10'.</span><br>
<div><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">
</span></div>
<div><font face="Courier New, Courier, monospace, arial,
sans-serif" color="#000000"><span style="font-size:14px;white-space:pre-wrap">Hong</span></font></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 24, 2017 at 3:06 PM,
Danyang Su <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Dear Hong,</p>
<p>I just tested with different number of processors for
the same matrix. It sometimes got "ERROR: Arguments are
incompatible" for different number of processors. It
works fine using 4, 8, or 24 processors, but failed with
"ERROR: Arguments are incompatible" using 16 or 48
processors. The error information is attached. I tested
this on my local computer with 6 cores 12 threads. Any
suggestion on this?</p>
<p>Thanks,<br>
</p>
<p>Danyang<br>
</p>
<div>
<div class="h5"> <br>
<div class="m_9041936809186897311moz-cite-prefix">On
17-05-24 12:28 PM, Danyang Su wrote:<br>
</div>
<blockquote type="cite">
<p>Hi Hong,</p>
<p>Awesome. Thanks for testing the case. I will try
your options for the code and get back to you
later.</p>
<p>Regards,</p>
<p>Danyang<br>
</p>
<br>
<div class="m_9041936809186897311moz-cite-prefix">On
17-05-24 12:21 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote">I tested your data.</div>
<div class="gmail_quote">Your matrices
encountered zero pivots, e.g.</div>
<div class="gmail_quote">
<div class="gmail_quote">petsc/src/ksp/ksp/examples/<wbr>tutorials
(master)</div>
<div class="gmail_quote">$ mpiexec -n 24
./ex10 -f0 a_react_in_2.bin -rhs
b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">[15]PETSC ERROR:
Zero pivot in LU factorization: <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot"
target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>zeropivot</a></div>
<div class="gmail_quote">[15]PETSC ERROR:
Zero pivot row 1249 value 2.05808e-14
tolerance 2.22045e-14</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Adding option
'-sub_pc_factor_shift_type nonzero', I
got</div>
<div class="gmail_quote">
<div class="gmail_quote">mpiexec -n 24
./ex10 -f0 a_react_in_2.bin -rhs
b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged
-sub_pc_factor_shift_type nonzero
-mat_view ascii::ascii_info</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Mat Object: 24
MPI processes</div>
<div class="gmail_quote"> type: mpiaij</div>
<div class="gmail_quote"> rows=450000,
cols=450000</div>
<div class="gmail_quote"> total:
nonzeros=6991400, allocated
nonzeros=6991400</div>
<div class="gmail_quote"> total number
of mallocs used during MatSetValues
calls =0</div>
<div class="gmail_quote"> not using
I-node (on process 0) routines</div>
<div class="gmail_quote"> 0 KSP
Residual norm 5.849777711755e+01</div>
<div class="gmail_quote"> 1 KSP
Residual norm 6.824179430230e-01</div>
<div class="gmail_quote"> 2 KSP
Residual norm 3.994483555787e-02</div>
<div class="gmail_quote"> 3 KSP
Residual norm 6.085841461433e-03</div>
<div class="gmail_quote"> 4 KSP
Residual norm 8.876162583511e-04</div>
<div class="gmail_quote"> 5 KSP
Residual norm 9.407780665278e-05</div>
<div class="gmail_quote">Number of
iterations = 5</div>
<div class="gmail_quote">Residual norm
0.00542891</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>Yes. The matrix is 450000x450000
sparse. The hypre takes hundreds of
iterates, not for all but in most of
the timesteps. The matrix is not well
conditioned, with nonzero entries
range from 1.0e-29 to 1.0e2. I also
made double check if there is anything
wrong in the parallel version,
however, the matrix is the same with
sequential version except some round
error which is relatively very small.
Usually for those not well conditioned
matrix, direct solver should be faster
than iterative solver, right? But when
I use the sequential iterative solver
with ILU prec developed almost 20
years go by others, the solver
converge fast with appropriate
factorization level. In other words,
when I use 24 processor using hypre,
the speed is almost the same as as the
old sequential iterative solver using
1 processor.</p>
<p>I use most of the default
configuration for the general case
with pretty good speedup. And I am not
sure if I miss something for this
problem.</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
<div>
<div
class="m_9041936809186897311gmail-h5">
<br>
<div
class="m_9041936809186897311gmail-m_1680376947984306833moz-cite-prefix">On
17-05-24 11:12 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Wed, May 24, 2017 at 12:50
PM, Danyang Su <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew and Barry,</p>
<p>Thanks for the quick
response. <br>
</p>
<p>I also tried superlu
and mumps, both work
but it is about four
times slower than
ILU(dt) prec through
hypre, with 24
processors I have
tested.</p>
</div>
</blockquote>
<div>You mean the total time
is 4x? And you are taking
hundreds of iterates? That
seems hard to believe,
unless you are dropping</div>
<div>a huge number of
elements. <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> When I look into the
convergence
information, the
method using ILU(dt)
still takes 200 to
3000 linear iterations
for each newton
iteration. One reason
is this equation is
hard to solve. As for
the general cases, the
same method works
awesome and get very
good speedup.</p>
</div>
</blockquote>
<div>I do not understand
what you mean here. <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> I also doubt if I
use hypre correctly
for this case. Is
there anyway to check
this problem, or is it
possible to increase
the factorization
level through hypre?</p>
</div>
</blockquote>
<div>I don't know.</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Danyang<br>
</p>
<br>
<div
class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
17-05-24 04:59 AM,
Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Wed, May 24,
2017 at 2:21 AM,
Danyang Su <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Dear
All,<br>
<br>
I use
PCFactorSetLevels
for ILU and
PCFactorSetFill
for other
preconditioning
in my code to
help solve the
problems that
the default
option is hard
to solve.
However, I
found the
latter one,
PCFactorSetFill
does not take
effect for my
problem. The
matrices and
rhs as well as
the solutions
are attached
from the link
below. I
obtain the
solution using
hypre
preconditioner
and it takes 7
and 38
iterations for
matrix 1 and
matrix 2.
However, if I
use other
preconditioner,
the solver
just failed at
the first
matrix. I have
tested this
matrix using
the native
sequential
solver (not
PETSc) with
ILU
preconditioning.
If I set the
incomplete
factorization
level to 0,
this
sequential
solver will
take more than
100
iterations. If
I increase the
factorization
level to 1 or
more, it just
takes several
iterations.
This remind me
that the PC
factor for
this matrices
should be
increased.
However, when
I tried it in
PETSc, it just
does not work.<br>
<br>
Matrix and rhs
can be
obtained from
the link
below.<br>
<br>
<a
moz-do-not-send="true"
href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R"
rel="noreferrer"
target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
<br>
Would anyone
help to check
if you can
make this work
by increasing
the PC factor
level or fill?<br>
</blockquote>
<div><br>
</div>
<div>We have
ILU(k)
supported in
serial.
However
ILU(dt) which
takes a
tolerance only
works through
Hypre</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html"
target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
<div><br>
</div>
<div>I recommend
you try
SuperLU or
MUMPS, which
can both be
downloaded
automatically
by configure,
and</div>
<div>do a full
sparse LU.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Thanks and
regards,<br>
<br>
Danyang<br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<span
class="m_9041936809186897311gmail-m_1680376947984306833HOEnZb"><font
color="#888888">
<div><br>
</div>
-- <br>
<div
class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div
class="m_9041936809186897311gmail-m_1680376947984306833gmail_signature">
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their experiments
is infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>