<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Hello Hong,<br>
    <br>
    Thanks for the quick reply and the option "-mat_superlu_dist_fact
    SamePattern" works like a charm, if I use this option from the
    command line. <br>
    <br>
    How can I add this option as the default. I tried using
    PetscOptionsInsertString("-mat_superlu_dist_fact SamePattern",ierr)
    in my code but this does not work. <br>
    <br>
    Thanks,<br>
    <br>
    Danyang<br>
    <br>
    <div class="moz-cite-prefix">On 15-12-07 10:42 AM, Hong wrote:<br>
    </div>
    <blockquote
cite="mid:CAGCphBv6ahBz-cBfyXLkuwFsMTMmJpL8RBW7DRGNstr0vL=Hww@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">Danyang :</div>
          <div class="gmail_quote"><br>
          </div>
          <div class="gmail_quote">Adding '-mat_superlu_dist_fact
            SamePattern' fixed the problem. Below is how I figured it
            out.</div>
          <div class="gmail_quote"><br>
          </div>
          <div class="gmail_quote">
            <div class="gmail_quote">1. Reading ex52f.F, I see
              '-superlu_default' = '-pc_factor_mat_solver_package
              superlu_dist', the later enables runtime options for other
              packages. I use superlu_dist-4.2 and superlu-4.1 for the
              tests below.</div>
            <div class="gmail_quote"><br>
            </div>
            <div class="gmail_quote">2. Use the Matrix 168 to setup KSP
              solver and factorization, all packages, petsc,
              superlu_dist and mumps give same correct results:</div>
            <div class="gmail_quote"><br>
            </div>
            <div class="gmail_quote">./ex52f -f0
              matrix_and_rhs_bin/a_flow_check_168.bin -rhs
              matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
              flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
              -pc_factor_mat_solver_package petsc</div>
            <div class="gmail_quote"> -->loac matrix a</div>
            <div class="gmail_quote"> -->load rhs b</div>
            <div class="gmail_quote"> size l,m,n,mm       90000      
              90000       90000       90000</div>
            <div class="gmail_quote">Norm of error  7.7308E-11
              iterations     1</div>
            <div class="gmail_quote"> -->Test for matrix          168</div>
            <div class="gmail_quote">..</div>
            <div class="gmail_quote"> -->Test for matrix          172</div>
            <div class="gmail_quote">Norm of error  3.8461E-11
              iterations     1</div>
            <div class="gmail_quote"><br>
            </div>
            <div class="gmail_quote">./ex52f -f0
              matrix_and_rhs_bin/a_flow_check_168.bin -rhs
              matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
              flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
              -pc_factor_mat_solver_package superlu_dist</div>
            <div class="gmail_quote">Norm of error  9.4073E-11
              iterations     1</div>
            <div class="gmail_quote"> -->Test for matrix          168</div>
            <div class="gmail_quote">...</div>
            <div class="gmail_quote"> -->Test for matrix          172</div>
            <div class="gmail_quote">Norm of error  3.8187E-11
              iterations     1</div>
            <div class="gmail_quote"><br>
            </div>
            <div class="gmail_quote">
              <div class="gmail_quote">3. Use superlu, I get</div>
              <div class="gmail_quote">./ex52f -f0
                matrix_and_rhs_bin/a_flow_check_168.bin -rhs
                matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
                flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
                -pc_factor_mat_solver_package superlu</div>
              <div class="gmail_quote">Norm of error  1.0191E-06
                iterations     1</div>
              <div class="gmail_quote"> -->Test for matrix        
                 168</div>
              <div class="gmail_quote">...</div>
              <div class="gmail_quote"> -->Test for matrix        
                 172</div>
              <div class="gmail_quote">Norm of error  9.7858E-07
                iterations     1</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">Replacing default
                DiagPivotThresh: 1. to 0.0, I get same solutions as
                other packages:</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">./ex52f -f0
                matrix_and_rhs_bin/a_flow_check_168.bin -rhs
                matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
                flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
                -pc_factor_mat_solver_package superlu
                -mat_superlu_diagpivotthresh 0.0</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">Norm of error  8.3614E-11
                iterations     1</div>
              <div class="gmail_quote"> -->Test for matrix        
                 168</div>
              <div class="gmail_quote">...</div>
              <div class="gmail_quote"> -->Test for matrix        
                 172</div>
              <div class="gmail_quote">Norm of error  3.7098E-11
                iterations     1</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">4.</div>
              <div class="gmail_quote">using '-mat_view
                ascii::ascii_info', I found that a_flow_check_1.bin and
                a_flow_check_168.bin seem have same structure:</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote"> -->loac matrix a</div>
              <div class="gmail_quote">Mat Object: 1 MPI processes</div>
              <div class="gmail_quote">  type: seqaij</div>
              <div class="gmail_quote">  rows=90000, cols=90000</div>
              <div class="gmail_quote">  total: nonzeros=895600,
                allocated nonzeros=895600</div>
              <div class="gmail_quote">  total number of mallocs used
                during MatSetValues calls =0</div>
              <div class="gmail_quote">    using I-node routines: found
                45000 nodes, limit used is 5</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">
                <div class="gmail_quote">5.</div>
                <div class="gmail_quote">Using a_flow_check_1.bin, I am
                  able to reproduce the error you reported: all packages
                  give correct results except superlu_dist:</div>
                <div class="gmail_quote">./ex52f -f0
                  matrix_and_rhs_bin/a_flow_check_1.bin -rhs
                  matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
                  flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
                  -pc_factor_mat_solver_package superlu_dist</div>
                <div class="gmail_quote">Norm of error  2.5970E-12
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   168</div>
                <div class="gmail_quote">Norm of error  1.3936E-01
                  iterations    34</div>
                <div class="gmail_quote"> -->Test for matrix        
                   169</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">I guess the error might come
                  from reuse of matrix factor. Replacing default</div>
                <div class="gmail_quote">-mat_superlu_dist_fact
                  <SamePattern_SameRowPerm> with</div>
                <div class="gmail_quote">-mat_superlu_dist_fact
                  SamePattern, I get</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">./ex52f -f0
                  matrix_and_rhs_bin/a_flow_check_1.bin -rhs
                  matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices
                  flow_check -loop_folder matrix_and_rhs_bin -pc_type lu
                  -pc_factor_mat_solver_package superlu_dist
                  -mat_superlu_dist_fact SamePattern</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">Norm of error  2.5970E-12
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   168</div>
                <div class="gmail_quote">Norm of error  9.4073E-11
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   169</div>
                <div class="gmail_quote">Norm of error  6.4303E-11
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   170</div>
                <div class="gmail_quote">Norm of error  7.4327E-11
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   171</div>
                <div class="gmail_quote">Norm of error  5.4162E-11
                  iterations     1</div>
                <div class="gmail_quote"> -->Test for matrix        
                   172</div>
                <div class="gmail_quote">Norm of error  3.4440E-11
                  iterations     1</div>
                <div class="gmail_quote"> --> End of test, bye</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">Sherry may tell you why
                  SamePattern_SameRowPerm cause the difference here.</div>
                <div class="gmail_quote">Best on the above experiments,
                  I would set following as default</div>
                <div class="gmail_quote">'-mat_superlu_diagpivotthresh
                  0.0' in petsc/superlu interface.</div>
                <div class="gmail_quote">'-mat_superlu_dist_fact
                  SamePattern' in petsc/superlu_dist interface.</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">Hong</div>
              </div>
            </div>
          </div>
          <div class="gmail_quote"><br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000"> Hi Hong,<br>
                <br>
                I did more test today and finally found that the
                solution accuracy depends on the initial (first) matrix
                quality. I modified the ex52f.F to do the test. There
                are 6 matrices and right-hand-side vectors. All these
                matrices and rhs are from my reactive transport
                simulation. Results will be quite different depending on
                which one you use to do factorization. Results will also
                be different if you run with different options. My code
                is similar to the First or the Second test below. When
                the matrix is well conditioned, it works fine. But if
                the initial matrix is well conditioned, it likely to
                crash when the matrix become ill-conditioned. Since most
                of my case are well conditioned so I didn't detect the
                problem before. This case is a special one. <br>
                <br>
                <br>
                How can I avoid this problem? Shall I redo
                factorization? Can PETSc automatically detect this
                prolbem or is there any option available to do this?<br>
                <br>
                All the data and test code (modified ex52f) can be found
                via the dropbox link below. <br>
                <u><br>
                </u><u><a moz-do-not-send="true"
href="https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0"
                    target="_blank">https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0</a></u><br>
                <br>
                <br>
                Summary of my test is shown below. <br>
                <br>
                First, use the Matrix 1 to setup KSP solver and
                factorization, then solve 168 to 172<br>
                <br>
                mpiexec.hydra -n 1 ./ex52f -f0
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin
                -rhs
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin
                -loop_matrices flow_check -loop_folder
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
                -pc_type lu -pc_factor_mat_solver_package superlu_dist<br>
                <br>
                Norm of error  3.8815E-11 iterations     1<br>
                 -->Test for matrix          168<br>
                Norm of error  4.2307E-01 iterations    32<br>
                 -->Test for matrix          169<br>
                Norm of error  3.0528E-01 iterations    32<br>
                 -->Test for matrix          170<br>
                Norm of error  3.1177E-01 iterations    32<br>
                 -->Test for matrix          171<br>
                Norm of error  3.2793E-01 iterations    32<br>
                 -->Test for matrix          172<br>
                Norm of error  3.1251E-01 iterations    31<br>
                <br>
                Second, use the Matrix 1 to setup KSP solver and
                factorization using the implemented SuperLU relative
                codes. I thought this will generate the same results as
                the First test, but it actually not.<br>
                <br>
                mpiexec.hydra -n 1 ./ex52f -f0
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin
                -rhs
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin
                -loop_matrices flow_check -loop_folder
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
                -superlu_default<br>
                <br>
                Norm of error  2.2632E-12 iterations     1<br>
                 -->Test for matrix          168<br>
                Norm of error  1.0817E+04 iterations     1<br>
                 -->Test for matrix          169<br>
                Norm of error  1.0786E+04 iterations     1<br>
                 -->Test for matrix          170<br>
                Norm of error  1.0792E+04 iterations     1<br>
                 -->Test for matrix          171<br>
                Norm of error  1.0792E+04 iterations     1<br>
                 -->Test for matrix          172<br>
                Norm of error  1.0792E+04 iterations     1<br>
                <br>
                <br>
                Third, use the Matrix 168 to setup KSP solver and
                factorization, then solve 168 to 172<br>
                <br>
                mpiexec.hydra -n 1 ./ex52f -f0
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin

                -rhs
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin

                -loop_matrices flow_check -loop_folder
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
                -pc_type lu -pc_factor_mat_solver_package superlu_dist<br>
                <br>
                Norm of error  9.5528E-10 iterations     1<br>
                 -->Test for matrix          168<br>
                Norm of error  9.4945E-10 iterations     1<br>
                 -->Test for matrix          169<br>
                Norm of error  6.4279E-10 iterations     1<br>
                 -->Test for matrix          170<br>
                Norm of error  7.4633E-10 iterations     1<br>
                 -->Test for matrix          171<br>
                Norm of error  7.4863E-10 iterations     1<br>
                 -->Test for matrix          172<br>
                Norm of error  8.9701E-10 iterations     1<br>
                <br>
                Fourth, use the Matrix 168 to setup KSP solver and
                factorization using the implemented SuperLU relative
                codes. I thought this will generate the same results as
                the Third test, but it actually not.<br>
                <br>
                mpiexec.hydra -n 1 ./ex52f -f0
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin

                -rhs
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin

                -loop_matrices flow_check -loop_folder
                /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin
                -superlu_default<br>
                <br>
                Norm of error  3.7017E-11 iterations     1<br>
                 -->Test for matrix          168<br>
                Norm of error  3.6420E-11 iterations     1<br>
                 -->Test for matrix          169<br>
                Norm of error  3.7184E-11 iterations     1<br>
                 -->Test for matrix          170<br>
                Norm of error  3.6847E-11 iterations     1<br>
                 -->Test for matrix          171<br>
                Norm of error  3.7883E-11 iterations     1<br>
                 -->Test for matrix          172<br>
                Norm of error  3.8805E-11 iterations     1<br>
                <br>
                Thanks very much,<br>
                <br>
                Danyang<br>
                <br>
                <div>On 15-12-03 01:59 PM, Hong wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div class="gmail_extra">
                      <div class="gmail_quote">Danyang :</div>
                      <div class="gmail_quote">Further
                        testing a_flow_check_168.bin,</div>
                      <div class="gmail_quote">
                        <div class="gmail_quote">./ex10 -f0
                          /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin
                          -rhs
                          /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin
                          -pc_type lu -pc_factor_mat_solver_package
                          superlu -ksp_monitor_true_residual
                          -mat_superlu_conditionnumber</div>
                        <div class="gmail_quote">  Recip. condition
                          number = 1.610480e-12</div>
                        <div class="gmail_quote">  0 KSP preconditioned
                          resid norm 6.873340313547e+09 true resid norm
                          7.295020990196e+03 ||r(i)||/||b||
                          1.000000000000e+00</div>
                        <div class="gmail_quote">  1 KSP preconditioned
                          resid norm 2.051833296449e-02 true resid norm
                          2.976859070118e-02 ||r(i)||/||b||
                          4.080672384793e-06</div>
                        <div class="gmail_quote">Number of iterations =
                            1</div>
                        <div class="gmail_quote">Residual norm 0.0297686</div>
                        <div class="gmail_quote"><br>
                        </div>
                        <div class="gmail_quote">condition number of
                          this matrix = 1/1.610480e-12 = 1.e+12,</div>
                        <div class="gmail_quote">i.e., this matrix is
                          ill-conditioned. </div>
                        <div class="gmail_quote"><br>
                        </div>
                        <div class="gmail_quote">Hong</div>
                        <div class="gmail_quote"><br>
                        </div>
                        <div class="gmail_quote"><br>
                        </div>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> Hi
                            Hong,<br>
                            <br>
                            The binary format of matrix, rhs and
                            solution can be downloaded via the link
                            below.<br>
                            <br>
                            <a moz-do-not-send="true"
href="https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0"
                              target="_blank">https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0</a><br>
                            <br>
                            Thanks,<br>
                            <br>
                            Danyang
                            <div>
                              <div><br>
                                <br>
                                On 15-12-03 10:50 AM, Hong wrote:<br>
                                <blockquote type="cite">
                                  <div dir="ltr">
                                    <div class="gmail_extra">
                                      <div class="gmail_quote">Danyang:<br>
                                        <blockquote class="gmail_quote"
                                          style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                          <div text="#000000"
                                            bgcolor="#FFFFFF"><br>
                                            <br>
                                            To my surprising, solutions
                                            from SuperLU at timestep 29
                                            seems not correct for the
                                            first 4 Newton iterations,
                                            but the solutions from
                                            iteration solver and MUMPS
                                            are correct. <br>
                                            <br>
                                            Please find all the
                                            matrices, rhs and solutions
                                            at timestep 29 via the link
                                            below. The data is a bit
                                            large so that I just share
                                            it through Dropbox. A piece
                                            of matlab code to read these
                                            data and then computer the
                                            norm has also been attached.
                                            <br>
                                            <u><a moz-do-not-send="true"
href="https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0"
                                                target="_blank">https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0</a></u></div>
                                        </blockquote>
                                        <div><br>
                                        </div>
                                        <div>Can you send us matrix in
                                          petsc binary format?</div>
                                        <div><br>
                                        </div>
                                        <div>e.g., call MatView(M,
                                          PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD))</div>
                                        <div>or '-ksp_view_mat binary'</div>
                                        <div><br>
                                        </div>
                                        <div>Hong</div>
                                        <blockquote class="gmail_quote"
                                          style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                          <div text="#000000"
                                            bgcolor="#FFFFFF"><br>
                                            <br>
                                            Below is a summary of the
                                            norm from the three solvers
                                            at timestep 29, newton
                                            iteration 1 to 5.<br>
                                            <br>
                                            Timestep 29<br>
                                            Norm of residual seq
                                            1.661321e-09, superlu
                                            1.657103e+04, mumps
                                            3.731225e-11 <br>
                                            Norm of residual seq
                                            1.753079e-09, superlu
                                            6.675467e+02, mumps
                                            1.509919e-13 <br>
                                            Norm of residual seq
                                            4.914971e-10, superlu
                                            1.236362e-01, mumps
                                            2.139303e-17 <br>
                                            Norm of residual seq
                                            3.532769e-10, superlu
                                            1.304670e-04, mumps
                                            5.387000e-20 <br>
                                            Norm of residual seq
                                            3.885629e-10, superlu
                                            2.754876e-07, mumps
                                            4.108675e-21 <br>
                                            <br>
                                            Would anybody please check
                                            if SuperLU can solve these
                                            matrices? Another
                                            possibility is that
                                            something is wrong in my own
                                            code. But so far, I cannot
                                            find any problem in my code
                                            since the same code works
                                            fine if I using iterative
                                            solver or direct solver
                                            MUMPS. But for other cases I
                                            have tested,  all these
                                            solvers work fine.<br>
                                            <br>
                                            Please let me know if I did
                                            not write down the problem
                                            clearly.<br>
                                            <br>
                                            Thanks,<br>
                                            <br>
                                            Danyang<br>
                                            <br>
                                            <br>
                                            <br>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br>
                                    </div>
                                  </div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </div>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>