<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Hong,</p>
<p>It works like a charm. I really appreciate your help.<br>
</p>
<p>Regards,</p>
<p>Danyang<br>
</p>
<br>
<div class="moz-cite-prefix">On 17-05-25 07:49 AM, Hong wrote:<br>
</div>
<blockquote
cite="mid:CAGCphBu7jAaqSA_j8wreS_XRoPpD+vJYt-hiua1Tkp4q=Bf8Jg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang:</div>
<div class="gmail_quote">You must access inner pc, then set
shift. See </div>
<div class="gmail_quote">petsc/src/ksp/ksp/examples/tutorials/ex7.c</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">For example, I add following to
petsc/src/ksp/ksp/examples/tutorials/ex2.c, line 191:</div>
<div class="gmail_quote">
<div class="gmail_quote"> PetscBool isbjacobi;</div>
<div class="gmail_quote"> PC pc;</div>
<div class="gmail_quote"> ierr =
KSPGetPC(ksp,&pc);CHKERRQ(ierr);</div>
<div class="gmail_quote"> ierr =
PetscObjectTypeCompare((PetscObject)pc,PCBJACOBI,&isbjacobi);CHKERRQ(ierr);</div>
<div class="gmail_quote"> if (isbjacobi) {</div>
<div class="gmail_quote"> PetscInt nlocal;</div>
<div class="gmail_quote"> KSP *subksp;</div>
<div class="gmail_quote"> PC subpc;</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> ierr =
KSPSetUp(ksp);CHKERRQ(ierr);</div>
<div class="gmail_quote"> ierr =
KSPGetPC(ksp,&pc);CHKERRQ(ierr);</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> /* Extract the array of KSP
contexts for the local blocks */</div>
<div class="gmail_quote"> ierr =
PCBJacobiGetSubKSP(pc,&nlocal,NULL,&subksp);CHKERRQ(ierr);</div>
<div class="gmail_quote"> printf("isbjacobi, nlocal %D,
set option to subpc...\n",nlocal);</div>
<div class="gmail_quote"> for (i=0; i<nlocal; i++) {</div>
<div class="gmail_quote"> ierr =
KSPGetPC(subksp[i],&subpc);CHKERRQ(ierr);</div>
<div class="gmail_quote"> ierr =
PCFactorSetShiftType(subpc,MAT_SHIFT_NONZERO);CHKERRQ(ierr);</div>
<div class="gmail_quote"> }</div>
<div class="gmail_quote"> }</div>
</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Dear Hong and Barry,</p>
<p>I have implemented this option in the code, as we
also need to use configuration from file for
convenience. When I run the code using options, it
works fine, however, when I run the code using
configuration file, it does not work. The code has two
set of equations, flow and reactive, with prefix been
set to "flow_" and "react_". When I run the code using</p>
<p>mpiexec -n 4 ../executable
-flow_sub_pc_factor_shift_type nonzero
-react_sub_pc_factor_shift_<wbr>type nonzero</p>
<p>it works. However, if I run using</p>
<p>mpiexec -n 4 ../executable <br>
</p>
<p>and let the executable file read the options from
file, it just does not work at "call
PCFactorSetShiftType(pc_flow,<wbr>MAT_SHIFT_NONZERO,
ierr) or none, positive_definite ...". Do I miss
something here?<br>
</p>
<p>Below is the pseudo code I have used for flow
equations, similar for reactive equations. <br>
</p>
<p> call MatCreateAIJ(Petsc_Comm_World,<wbr>nndof,nndof,nngbldof,
&
<br>
nngbldof,d_nz,PETSC_NULL_<wbr>INTEGER,o_nz,
&
<br>
PETSC_NULL_INTEGER,a_flow,<wbr>ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call MatSetFromOptions(a_flow,ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPCreate(Petsc_Comm_World, ksp_flow,
ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPAppendOptionsPrefix(ksp_<wbr>flow,"flow_",ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPSetInitialGuessNonzero(ksp_<wbr>flow,
&
<br>
b_initial_guess_nonzero_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPSetInitialGuessNonzero(ksp_<wbr>flow,
&
<br>
b_initial_guess_nonzero_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call KSPSetDM(ksp_flow,dmda_flow%<wbr>da,ierr)
<br>
CHKERRQ(ierr)
<br>
call KSPSetDMActive(ksp_flow,PETSC_<wbr>FALSE,ierr)
<br>
CHKERRQ(ierr)<br>
</p>
<p> !!!!*********CHECK IF READ OPTION FROM
FILE*********!!!!<br>
if (read_option_from_file) then<br>
<br>
call KSPSetType(ksp_flow, KSPGMRES, ierr)
!or KSPBCGS or others...
<br>
CHKERRQ(ierr)<br>
<br>
call KSPGetPC(ksp_flow, pc_flow, ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call PCSetType(pc_flow,PCBJACOBI,
ierr) !or PCILU or PCJACOBI or PCHYPRE ...
<br>
CHKERRQ(ierr)<br>
<br>
call PCFactorSetShiftType(pc_flow,<wbr>MAT_SHIFT_NONZERO,
ierr) or none, positive_definite ...
<br>
CHKERRQ(ierr)<br>
<br>
end if<br>
<br>
call PCFactorGetMatSolverPackage(<wbr>pc_flow,solver_pkg_flow,ierr)
<br>
CHKERRQ(ierr)<br>
<br>
call compute_jacobian(rank,dmda_<wbr>flow%da,
&
<br>
a_flow,a_in,ia_in,ja_in,nngl_<wbr>in, &
<br>
row_idx_l2pg,col_idx_l2pg, <wbr> &
<br>
b_non_interlaced)<br>
call KSPSetFromOptions(ksp_flow,<wbr>ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSetUp(ksp_flow,ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSetUpOnBlocks(ksp_flow,<wbr>ierr)
<br>
CHKERRQ(ierr)
<br>
<br>
call KSPSolve(ksp_flow,b_flow,x_<wbr>flow,ierr)
<br>
CHKERRQ(ierr) <br>
</p>
<p> <br>
Thanks and Regards,<br>
</p>
<p>Danyang</p>
<div class="gmail-m_-8252057521010818115moz-cite-prefix">On
17-05-24 06:32 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><span style="font-size:12.8px">Remove
your option '</span><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">-vecload_block_size 10'.</span><br>
<div><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">
</span></div>
<div><font color="#000000" face="Courier New,
Courier, monospace, arial, sans-serif"><span style="font-size:14px;white-space:pre-wrap">Hong</span></font></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 24, 2017 at
3:06 PM, Danyang Su <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Dear Hong,</p>
<p>I just tested with different number of
processors for the same matrix. It sometimes
got "ERROR: Arguments are incompatible" for
different number of processors. It works
fine using 4, 8, or 24 processors, but
failed with "ERROR: Arguments are
incompatible" using 16 or 48 processors. The
error information is attached. I tested this
on my local computer with 6 cores 12
threads. Any suggestion on this?</p>
<p>Thanks,<br>
</p>
<p>Danyang<br>
</p>
<div>
<div class="gmail-m_-8252057521010818115h5">
<br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311moz-cite-prefix">On
17-05-24 12:28 PM, Danyang Su wrote:<br>
</div>
<blockquote type="cite">
<p>Hi Hong,</p>
<p>Awesome. Thanks for testing the case.
I will try your options for the code
and get back to you later.</p>
<p>Regards,</p>
<p>Danyang<br>
</p>
<br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311moz-cite-prefix">On
17-05-24 12:21 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote">I tested
your data.</div>
<div class="gmail_quote">Your
matrices encountered zero
pivots, e.g.</div>
<div class="gmail_quote">
<div class="gmail_quote">petsc/src/ksp/ksp/examples/tut<wbr>orials
(master)</div>
<div class="gmail_quote">$
mpiexec -n 24 ./ex10 -f0
a_react_in_2.bin -rhs
b_react_in_2.bin -ksp_monitor
-ksp_error_if_not_converged</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">[15]PETSC
ERROR: Zero pivot in LU
factorization: <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot"
target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#zeropivo<wbr>t</a></div>
<div class="gmail_quote">[15]PETSC
ERROR: Zero pivot row 1249
value 2.05808e-14 tolerance
2.22045e-14</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Adding
option
'-sub_pc_factor_shift_type
nonzero', I got</div>
<div class="gmail_quote">
<div class="gmail_quote">mpiexec
-n 24 ./ex10 -f0
a_react_in_2.bin -rhs
b_react_in_2.bin
-ksp_monitor
-ksp_error_if_not_converged
-sub_pc_factor_shift_type
nonzero -mat_view
ascii::ascii_info</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Mat
Object: 24 MPI processes</div>
<div class="gmail_quote">
type: mpiaij</div>
<div class="gmail_quote">
rows=450000, cols=450000</div>
<div class="gmail_quote">
total: nonzeros=6991400,
allocated nonzeros=6991400</div>
<div class="gmail_quote">
total number of mallocs
used during MatSetValues
calls =0</div>
<div class="gmail_quote">
not using I-node (on
process 0) routines</div>
<div class="gmail_quote"> 0
KSP Residual norm
5.849777711755e+01</div>
<div class="gmail_quote"> 1
KSP Residual norm
6.824179430230e-01</div>
<div class="gmail_quote"> 2
KSP Residual norm
3.994483555787e-02</div>
<div class="gmail_quote"> 3
KSP Residual norm
6.085841461433e-03</div>
<div class="gmail_quote"> 4
KSP Residual norm
8.876162583511e-04</div>
<div class="gmail_quote"> 5
KSP Residual norm
9.407780665278e-05</div>
<div class="gmail_quote">Number
of iterations = 5</div>
<div class="gmail_quote">Residual
norm 0.00542891</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>Yes. The matrix is
450000x450000 sparse. The
hypre takes hundreds of
iterates, not for all but
in most of the timesteps.
The matrix is not well
conditioned, with nonzero
entries range from 1.0e-29
to 1.0e2. I also made
double check if there is
anything wrong in the
parallel version, however,
the matrix is the same
with sequential version
except some round error
which is relatively very
small. Usually for those
not well conditioned
matrix, direct solver
should be faster than
iterative solver, right?
But when I use the
sequential iterative
solver with ILU prec
developed almost 20 years
go by others, the solver
converge fast with
appropriate factorization
level. In other words,
when I use 24 processor
using hypre, the speed is
almost the same as as the
old sequential iterative
solver using 1 processor.</p>
<p>I use most of the default
configuration for the
general case with pretty
good speedup. And I am not
sure if I miss something
for this problem.</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
<div>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-h5">
<br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-m_1680376947984306833moz-cite-prefix">On
17-05-24 11:12 AM,
Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Wed, May 24,
2017 at 12:50
PM, Danyang Su <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>Hi Matthew
and Barry,</p>
<p>Thanks for
the quick
response. <br>
</p>
<p>I also
tried superlu
and mumps,
both work but
it is about
four times
slower than
ILU(dt) prec
through hypre,
with 24
processors I
have tested.</p>
</div>
</blockquote>
<div>You mean
the total time
is 4x? And you
are taking
hundreds of
iterates? That
seems hard to
believe,
unless you are
dropping</div>
<div>a huge
number of
elements. <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p> When I
look into the
convergence
information,
the method
using ILU(dt)
still takes
200 to 3000
linear
iterations for
each newton
iteration. One
reason is this
equation is
hard to solve.
As for the
general cases,
the same
method works
awesome and
get very good
speedup.</p>
</div>
</blockquote>
<div>I do not
understand
what you mean
here. <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p> I also
doubt if I use
hypre
correctly for
this case. Is
there anyway
to check this
problem, or is
it possible to
increase the
factorization
level through
hypre?</p>
</div>
</blockquote>
<div>I don't
know.</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Danyang<br>
</p>
<br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
17-05-24 04:59
AM, Matthew
Knepley wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Wed, May 24,
2017 at 2:21
AM, Danyang Su
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:danyang.su@gmail.com"
target="_blank">danyang.su@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Dear
All,<br>
<br>
I use
PCFactorSetLevels
for ILU and
PCFactorSetFill
for other
preconditioning
in my code to
help solve the
problems that
the default
option is hard
to solve.
However, I
found the
latter one,
PCFactorSetFill
does not take
effect for my
problem. The
matrices and
rhs as well as
the solutions
are attached
from the link
below. I
obtain the
solution using
hypre
preconditioner
and it takes 7
and 38
iterations for
matrix 1 and
matrix 2.
However, if I
use other
preconditioner,
the solver
just failed at
the first
matrix. I have
tested this
matrix using
the native
sequential
solver (not
PETSc) with
ILU
preconditioning.
If I set the
incomplete
factorization
level to 0,
this
sequential
solver will
take more than
100
iterations. If
I increase the
factorization
level to 1 or
more, it just
takes several
iterations.
This remind me
that the PC
factor for
this matrices
should be
increased.
However, when
I tried it in
PETSc, it just
does not work.<br>
<br>
Matrix and rhs
can be
obtained from
the link
below.<br>
<br>
<a
moz-do-not-send="true"
href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R"
rel="noreferrer"
target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
<br>
Would anyone
help to check
if you can
make this work
by increasing
the PC factor
level or fill?<br>
</blockquote>
<div><br>
</div>
<div>We have
ILU(k)
supported in
serial.
However
ILU(dt) which
takes a
tolerance only
works through
Hypre</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html"
target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
<div><br>
</div>
<div>I
recommend you
try SuperLU or
MUMPS, which
can both be
downloaded
automatically
by configure,
and</div>
<div>do a full
sparse LU.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Thanks and
regards,<br>
<br>
Danyang<br>
<br>
<br>
</blockquote>
</div>
<br>
<br
clear="all">
<span
class="gmail-HOEnZb"><font
color="#888888"> <span
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-m_1680376947984306833HOEnZb"><font
color="#888888">
<div><br>
</div>
-- <br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888"> </font></span></blockquote>
<span
class="gmail-HOEnZb"><font
color="#888888"> <br>
</font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888"> </font></span></blockquote>
<span
class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888"> <br>
<br
clear="all">
<div><br>
</div>
-- <br>
<div
class="gmail-m_-8252057521010818115m_9041936809186897311gmail-m_1680376947984306833gmail_signature">
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
</div>
</div>
</div>
</font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888">
</font></span></div>
<span
class="gmail-HOEnZb"><font
color="#888888"> </font></span></blockquote>
<span
class="gmail-HOEnZb"><font
color="#888888"> <br>
</font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></blockquote>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> <br>
</font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></blockquote>
<span class="gmail-HOEnZb"><font
color="#888888"> <br>
</font></span></blockquote>
<span class="gmail-HOEnZb"><font
color="#888888"> <br>
</font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font
color="#888888"> </font></span></div>
<span class="gmail-HOEnZb"><font color="#888888">
</font></span></blockquote>
<span class="gmail-HOEnZb"><font color="#888888">
</font></span></div>
<span class="gmail-HOEnZb"><font color="#888888"> <br>
</font></span></div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>