<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Thank. The inserted options works now. I didn't put
PetscOptionsInsertString in the right place before.<br>
<br>
Danyang<br>
<br>
<div class="moz-cite-prefix">On 15-12-07 12:01 PM, Hong wrote:<br>
</div>
<blockquote
cite="mid:CAGCphBtp0TxsPj5HvKdkFaN44ZiU7CShc=TWVme_HVFt0EtRLA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang:</div>
<div class="gmail_quote">Add 'call MatSetFromOptions(A,ierr)'
to your code.</div>
<div class="gmail_quote">Attached below is ex52f.F modified
from your ex52f.F to be compatible with petsc-dev.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello Hong,<br>
<br>
Thanks for the quick reply and the option
"-mat_superlu_dist_fact SamePattern" works like a charm,
if I use this option from the command line. <br>
<br>
How can I add this option as the default. I tried using
PetscOptionsInsertString("-mat_superlu_dist_fact
SamePattern",ierr) in my code but this does not work. <br>
<br>
Thanks,<br>
<br>
Danyang
<div>
<div class="h5"><br>
<br>
<div>On 15-12-07 10:42 AM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Adding
'-mat_superlu_dist_fact SamePattern' fixed
the problem. Below is how I figured it out.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">1. Reading ex52f.F,
I see '-superlu_default' =
'-pc_factor_mat_solver_package
superlu_dist', the later enables runtime
options for other packages. I use
superlu_dist-4.2 and superlu-4.1 for the
tests below.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">2. Use the Matrix
168 to setup KSP solver and factorization,
all packages, petsc, superlu_dist and
mumps give same correct results:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package petsc</div>
<div class="gmail_quote"> -->loac matrix
a</div>
<div class="gmail_quote"> -->load rhs b</div>
<div class="gmail_quote"> size l,m,n,mm
90000 90000 90000
90000</div>
<div class="gmail_quote">Norm of error
7.7308E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">..</div>
<div class="gmail_quote"> -->Test for
matrix 172</div>
<div class="gmail_quote">Norm of error
3.8461E-11 iterations 1</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package superlu_dist</div>
<div class="gmail_quote">Norm of error
9.4073E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"> -->Test for
matrix 172</div>
<div class="gmail_quote">Norm of error
3.8187E-11 iterations 1</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">3. Use superlu, I
get</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package superlu</div>
<div class="gmail_quote">Norm of error
1.0191E-06 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"> -->Test for
matrix 172</div>
<div class="gmail_quote">Norm of error
9.7858E-07 iterations 1</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Replacing default
DiagPivotThresh: 1. to 0.0, I get same
solutions as other packages:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package superlu
-mat_superlu_diagpivotthresh 0.0</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Norm of error
8.3614E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">...</div>
<div class="gmail_quote"> -->Test for
matrix 172</div>
<div class="gmail_quote">Norm of error
3.7098E-11 iterations 1</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">4.</div>
<div class="gmail_quote">using '-mat_view
ascii::ascii_info', I found that
a_flow_check_1.bin and
a_flow_check_168.bin seem have same
structure:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> -->loac
matrix a</div>
<div class="gmail_quote">Mat Object: 1 MPI
processes</div>
<div class="gmail_quote"> type: seqaij</div>
<div class="gmail_quote"> rows=90000,
cols=90000</div>
<div class="gmail_quote"> total:
nonzeros=895600, allocated
nonzeros=895600</div>
<div class="gmail_quote"> total number of
mallocs used during MatSetValues calls
=0</div>
<div class="gmail_quote"> using I-node
routines: found 45000 nodes, limit used
is 5</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">
<div class="gmail_quote">5.</div>
<div class="gmail_quote">Using
a_flow_check_1.bin, I am able to
reproduce the error you reported: all
packages give correct results except
superlu_dist:</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_1.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package
superlu_dist</div>
<div class="gmail_quote">Norm of error
2.5970E-12 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">Norm of error
1.3936E-01 iterations 34</div>
<div class="gmail_quote"> -->Test for
matrix 169</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">I guess the
error might come from reuse of matrix
factor. Replacing default</div>
<div class="gmail_quote">-mat_superlu_dist_fact
<SamePattern_SameRowPerm> with</div>
<div class="gmail_quote">-mat_superlu_dist_fact
SamePattern, I get</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">./ex52f -f0
matrix_and_rhs_bin/a_flow_check_1.bin
-rhs
matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
matrix_and_rhs_bin -pc_type lu
-pc_factor_mat_solver_package
superlu_dist -mat_superlu_dist_fact
SamePattern</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Norm of error
2.5970E-12 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 168</div>
<div class="gmail_quote">Norm of error
9.4073E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 169</div>
<div class="gmail_quote">Norm of error
6.4303E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 170</div>
<div class="gmail_quote">Norm of error
7.4327E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 171</div>
<div class="gmail_quote">Norm of error
5.4162E-11 iterations 1</div>
<div class="gmail_quote"> -->Test for
matrix 172</div>
<div class="gmail_quote">Norm of error
3.4440E-11 iterations 1</div>
<div class="gmail_quote"> --> End of
test, bye</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Sherry may tell
you why SamePattern_SameRowPerm cause
the difference here.</div>
<div class="gmail_quote">Best on the
above experiments, I would set
following as default</div>
<div class="gmail_quote">'-mat_superlu_diagpivotthresh
0.0' in petsc/superlu interface.</div>
<div class="gmail_quote">'-mat_superlu_dist_fact
SamePattern' in petsc/superlu_dist
interface.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
</div>
</div>
</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi
Hong,<br>
<br>
I did more test today and finally found
that the solution accuracy depends on
the initial (first) matrix quality. I
modified the ex52f.F to do the test.
There are 6 matrices and right-hand-side
vectors. All these matrices and rhs are
from my reactive transport simulation.
Results will be quite different
depending on which one you use to do
factorization. Results will also be
different if you run with different
options. My code is similar to the First
or the Second test below. When the
matrix is well conditioned, it works
fine. But if the initial matrix is well
conditioned, it likely to crash when the
matrix become ill-conditioned. Since
most of my case are well conditioned so
I didn't detect the problem before. This
case is a special one. <br>
<br>
<br>
How can I avoid this problem? Shall I
redo factorization? Can PETSc
automatically detect this prolbem or is
there any option available to do this?<br>
<br>
All the data and test code (modified
ex52f) can be found via the dropbox link
below. <br>
<u><br>
</u><u><a moz-do-not-send="true"
href="https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0"
target="_blank">https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0</a></u><br>
<br>
<br>
Summary of my test is shown below. <br>
<br>
First, use the Matrix 1 to setup KSP
solver and factorization, then solve 168
to 172<br>
<br>
mpiexec.hydra -n 1 ./ex52f -f0
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin
-rhs
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin
-loop_matrices flow_check -loop_folder
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
-pc_type lu
-pc_factor_mat_solver_package
superlu_dist<br>
<br>
Norm of error 3.8815E-11 iterations
1<br>
-->Test for matrix 168<br>
Norm of error 4.2307E-01 iterations
32<br>
-->Test for matrix 169<br>
Norm of error 3.0528E-01 iterations
32<br>
-->Test for matrix 170<br>
Norm of error 3.1177E-01 iterations
32<br>
-->Test for matrix 171<br>
Norm of error 3.2793E-01 iterations
32<br>
-->Test for matrix 172<br>
Norm of error 3.1251E-01 iterations
31<br>
<br>
Second, use the Matrix 1 to setup KSP
solver and factorization using the
implemented SuperLU relative codes. I
thought this will generate the same
results as the First test, but it
actually not.<br>
<br>
mpiexec.hydra -n 1 ./ex52f -f0
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin
-rhs
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin
-loop_matrices flow_check -loop_folder
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
-superlu_default<br>
<br>
Norm of error 2.2632E-12 iterations
1<br>
-->Test for matrix 168<br>
Norm of error 1.0817E+04 iterations
1<br>
-->Test for matrix 169<br>
Norm of error 1.0786E+04 iterations
1<br>
-->Test for matrix 170<br>
Norm of error 1.0792E+04 iterations
1<br>
-->Test for matrix 171<br>
Norm of error 1.0792E+04 iterations
1<br>
-->Test for matrix 172<br>
Norm of error 1.0792E+04 iterations
1<br>
<br>
<br>
Third, use the Matrix 168 to setup KSP
solver and factorization, then solve 168
to 172<br>
<br>
mpiexec.hydra -n 1 ./ex52f -f0
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
-pc_type lu
-pc_factor_mat_solver_package
superlu_dist<br>
<br>
Norm of error 9.5528E-10 iterations
1<br>
-->Test for matrix 168<br>
Norm of error 9.4945E-10 iterations
1<br>
-->Test for matrix 169<br>
Norm of error 6.4279E-10 iterations
1<br>
-->Test for matrix 170<br>
Norm of error 7.4633E-10 iterations
1<br>
-->Test for matrix 171<br>
Norm of error 7.4863E-10 iterations
1<br>
-->Test for matrix 172<br>
Norm of error 8.9701E-10 iterations
1<br>
<br>
Fourth, use the Matrix 168 to setup KSP
solver and factorization using the
implemented SuperLU relative codes. I
thought this will generate the same
results as the Third test, but it
actually not.<br>
<br>
mpiexec.hydra -n 1 ./ex52f -f0
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin
-loop_matrices flow_check -loop_folder
/home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
-superlu_default<br>
<br>
Norm of error 3.7017E-11 iterations
1<br>
-->Test for matrix 168<br>
Norm of error 3.6420E-11 iterations
1<br>
-->Test for matrix 169<br>
Norm of error 3.7184E-11 iterations
1<br>
-->Test for matrix 170<br>
Norm of error 3.6847E-11 iterations
1<br>
-->Test for matrix 171<br>
Norm of error 3.7883E-11 iterations
1<br>
-->Test for matrix 172<br>
Norm of error 3.8805E-11 iterations
1<br>
<br>
Thanks very much,<br>
<br>
Danyang<br>
<br>
<div>On 15-12-03 01:59 PM, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Danyang :</div>
<div class="gmail_quote">Further
testing a_flow_check_168.bin,</div>
<div class="gmail_quote">
<div class="gmail_quote">./ex10
-f0
/Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin
-rhs
/Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin
-pc_type lu
-pc_factor_mat_solver_package
superlu
-ksp_monitor_true_residual
-mat_superlu_conditionnumber</div>
<div class="gmail_quote">
Recip. condition number =
1.610480e-12</div>
<div class="gmail_quote"> 0 KSP
preconditioned resid norm
6.873340313547e+09 true resid
norm 7.295020990196e+03
||r(i)||/||b||
1.000000000000e+00</div>
<div class="gmail_quote"> 1 KSP
preconditioned resid norm
2.051833296449e-02 true resid
norm 2.976859070118e-02
||r(i)||/||b||
4.080672384793e-06</div>
<div class="gmail_quote">Number
of iterations = 1</div>
<div class="gmail_quote">Residual
norm 0.0297686</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">condition
number of this matrix =
1/1.610480e-12 = 1.e+12,</div>
<div class="gmail_quote">i.e.,
this matrix is
ill-conditioned. </div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Hong</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF"> Hi Hong,<br>
<br>
The binary format of matrix,
rhs and solution can be
downloaded via the link
below.<br>
<br>
<a moz-do-not-send="true"
href="https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0"
target="_blank">https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0</a><br>
<br>
Thanks,<br>
<br>
Danyang
<div>
<div><br>
<br>
On 15-12-03 10:50 AM,
Hong wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">Danyang:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF"><br>
<br>
To my
surprising,
solutions from
SuperLU at
timestep 29
seems not
correct for
the first 4
Newton
iterations,
but the
solutions from
iteration
solver and
MUMPS are
correct. <br>
<br>
Please find
all the
matrices, rhs
and solutions
at timestep 29
via the link
below. The
data is a bit
large so that
I just share
it through
Dropbox. A
piece of
matlab code to
read these
data and then
computer the
norm has also
been attached.
<br>
<u><a
moz-do-not-send="true"
href="https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0"
target="_blank"><a class="moz-txt-link-freetext" href="https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0">https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0</a></a></u></div>
</blockquote>
<div><br>
</div>
<div>Can you
send us matrix
in petsc
binary format?</div>
<div><br>
</div>
<div>e.g., call
MatView(M,
PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD))</div>
<div>or
'-ksp_view_mat
binary'</div>
<div><br>
</div>
<div>Hong</div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF"><br>
<br>
Below is a
summary of the
norm from the
three solvers
at timestep
29, newton
iteration 1 to
5.<br>
<br>
Timestep 29<br>
Norm of
residual seq
1.661321e-09,
superlu
1.657103e+04,
mumps
3.731225e-11 <br>
Norm of
residual seq
1.753079e-09,
superlu
6.675467e+02,
mumps
1.509919e-13 <br>
Norm of
residual seq
4.914971e-10,
superlu
1.236362e-01,
mumps
2.139303e-17 <br>
Norm of
residual seq
3.532769e-10,
superlu
1.304670e-04,
mumps
5.387000e-20 <br>
Norm of
residual seq
3.885629e-10,
superlu
2.754876e-07,
mumps
4.108675e-21 <br>
<br>
Would anybody
please check
if SuperLU can
solve these
matrices?
Another
possibility is
that something
is wrong in my
own code. But
so far, I
cannot find
any problem in
my code since
the same code
works fine if
I using
iterative
solver or
direct solver
MUMPS. But for
other cases I
have tested,
all these
solvers work
fine.<br>
<br>
Please let me
know if I did
not write down
the problem
clearly.<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<br>
<br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>