<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Dear Hong and Barry,</p>
    <p>I have implemented this option in the code, as we also need to
      use configuration from file for convenience. When I run the code
      using options, it works fine, however, when I run the code using
      configuration file, it does not work. The code has two set of
      equations, flow and reactive, with prefix been set to "flow_" and
      "react_". When I run the code using</p>
    <p>mpiexec -n 4 ../executable -flow_sub_pc_factor_shift_type nonzero
      -react_sub_pc_factor_shift_type nonzero</p>
    <p>it works. However, if I run using</p>
    <p>mpiexec -n 4 ../executable <br>
    </p>
    <p>and let the executable file read the options from file, it just
      does not work at "call
      PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr)  or none,
      positive_definite ...". Do I miss something here?<br>
    </p>
    <p>Below is the pseudo code I have used for flow equations, similar
      for reactive equations. <br>
    </p>
    <p>          call
      MatCreateAIJ(Petsc_Comm_World,nndof,nndof,nngbldof,     &
<br>
                                 
      nngbldof,d_nz,PETSC_NULL_INTEGER,o_nz,     &
<br>
                                  PETSC_NULL_INTEGER,a_flow,ierr)
<br>
                CHKERRQ(ierr)<br>
      <br>
              call MatSetFromOptions(a_flow,ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call KSPCreate(Petsc_Comm_World, ksp_flow, ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call KSPAppendOptionsPrefix(ksp_flow,"flow_",ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call
      KSPSetInitialGuessNonzero(ksp_flow,                       &
<br>
                      b_initial_guess_nonzero_flow, ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call
      KSPSetInitialGuessNonzero(ksp_flow,                       &
<br>
                      b_initial_guess_nonzero_flow, ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call KSPSetDM(ksp_flow,dmda_flow%da,ierr)
<br>
              CHKERRQ(ierr)
<br>
              call KSPSetDMActive(ksp_flow,PETSC_FALSE,ierr)
<br>
              CHKERRQ(ierr)<br>
    </p>
    <p>        !!!!*********CHECK IF READ OPTION FROM FILE*********!!!!<br>
              if (read_option_from_file) then<br>
      <br>
                call KSPSetType(ksp_flow, KSPGMRES, ierr)    !or KSPBCGS
      or others...
<br>
                CHKERRQ(ierr)<br>
      <br>
                call KSPGetPC(ksp_flow, pc_flow, ierr)
<br>
                CHKERRQ(ierr)<br>
      <br>
                call PCSetType(pc_flow,PCBJACOBI, ierr)       !or PCILU
      or PCJACOBI or PCHYPRE ...
<br>
                CHKERRQ(ierr)<br>
      <br>
                call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO,
      ierr)  or none, positive_definite ...
<br>
                CHKERRQ(ierr)<br>
      <br>
              end if<br>
       <br>
              call
      PCFactorGetMatSolverPackage(pc_flow,solver_pkg_flow,ierr)
<br>
              CHKERRQ(ierr)<br>
      <br>
              call
      compute_jacobian(rank,dmda_flow%da,                       &
<br>
                                   
      a_flow,a_in,ia_in,ja_in,nngl_in,         &
<br>
                                   
      row_idx_l2pg,col_idx_l2pg,               &
<br>
                                    b_non_interlaced)<br>
              call KSPSetFromOptions(ksp_flow,ierr)
<br>
              CHKERRQ(ierr)
<br>
       <br>
              call KSPSetUp(ksp_flow,ierr)
<br>
              CHKERRQ(ierr)
<br>
       <br>
              call KSPSetUpOnBlocks(ksp_flow,ierr)
<br>
              CHKERRQ(ierr)
<br>
       <br>
              call KSPSolve(ksp_flow,b_flow,x_flow,ierr)
<br>
              CHKERRQ(ierr) <br>
    </p>
    <p>        <br>
      Thanks and Regards,<br>
    </p>
    <p>Danyang</p>
    <div class="moz-cite-prefix">On 17-05-24 06:32 PM, Hong wrote:<br>
    </div>
    <blockquote
cite="mid:CAGCphBvmU1jyK60QKsE2=K_5TEe_8+U+=wHcqB3V1ssbZGwQ+w@mail.gmail.com"
      type="cite">
      <div dir="ltr"><span style="font-size:12.8px">Remove your option '</span><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">-vecload_block_size 10'.</span><br>
        <div><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">
</span></div>
        <div><font face="Courier New, Courier, monospace, arial,
            sans-serif" color="#000000"><span style="font-size:14px;white-space:pre-wrap">Hong</span></font></div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, May 24, 2017 at 3:06 PM,
          Danyang Su <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000">
              <p>Dear Hong,</p>
              <p>I just tested with different number of processors for
                the same matrix. It sometimes got "ERROR: Arguments are
                incompatible" for different number of processors. It
                works fine using 4, 8, or 24 processors, but failed with
                "ERROR: Arguments are incompatible" using 16 or 48
                processors. The error information is attached. I tested
                this on my local computer with 6 cores 12 threads. Any
                suggestion on this?</p>
              <p>Thanks,<br>
              </p>
              <p>Danyang<br>
              </p>
              <div>
                <div class="h5"> <br>
                  <div class="m_9041936809186897311moz-cite-prefix">On
                    17-05-24 12:28 PM, Danyang Su wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <p>Hi Hong,</p>
                    <p>Awesome. Thanks for testing the case. I will try
                      your options for the code and get back to you
                      later.</p>
                    <p>Regards,</p>
                    <p>Danyang<br>
                    </p>
                    <br>
                    <div class="m_9041936809186897311moz-cite-prefix">On
                      17-05-24 12:21 PM, Hong wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div class="gmail_extra">
                          <div class="gmail_quote">Danyang :</div>
                          <div class="gmail_quote">I tested your data.</div>
                          <div class="gmail_quote">Your matrices
                            encountered zero pivots, e.g.</div>
                          <div class="gmail_quote">
                            <div class="gmail_quote">petsc/src/ksp/ksp/examples/<wbr>tutorials
                              (master)</div>
                            <div class="gmail_quote">$ mpiexec -n 24
                              ./ex10 -f0 a_react_in_2.bin -rhs
                              b_react_in_2.bin -ksp_monitor
                              -ksp_error_if_not_converged</div>
                            <div class="gmail_quote"><br>
                            </div>
                            <div class="gmail_quote">
                              <div class="gmail_quote">[15]PETSC ERROR:
                                Zero pivot in LU factorization: <a
                                  moz-do-not-send="true"
                                  href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot"
                                  target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>zeropivot</a></div>
                              <div class="gmail_quote">[15]PETSC ERROR:
                                Zero pivot row 1249 value 2.05808e-14
                                tolerance 2.22045e-14</div>
                              <div class="gmail_quote">...</div>
                              <div class="gmail_quote"><br>
                              </div>
                              <div class="gmail_quote">Adding option
                                '-sub_pc_factor_shift_type nonzero', I
                                got</div>
                              <div class="gmail_quote">
                                <div class="gmail_quote">mpiexec -n 24
                                  ./ex10 -f0 a_react_in_2.bin -rhs
                                  b_react_in_2.bin -ksp_monitor
                                  -ksp_error_if_not_converged
                                  -sub_pc_factor_shift_type nonzero
                                  -mat_view ascii::ascii_info</div>
                                <div class="gmail_quote"><br>
                                </div>
                                <div class="gmail_quote">Mat Object: 24
                                  MPI processes</div>
                                <div class="gmail_quote">  type: mpiaij</div>
                                <div class="gmail_quote">  rows=450000,
                                  cols=450000</div>
                                <div class="gmail_quote">  total:
                                  nonzeros=6991400, allocated
                                  nonzeros=6991400</div>
                                <div class="gmail_quote">  total number
                                  of mallocs used during MatSetValues
                                  calls =0</div>
                                <div class="gmail_quote">    not using
                                  I-node (on process 0) routines</div>
                                <div class="gmail_quote">  0 KSP
                                  Residual norm 5.849777711755e+01</div>
                                <div class="gmail_quote">  1 KSP
                                  Residual norm 6.824179430230e-01</div>
                                <div class="gmail_quote">  2 KSP
                                  Residual norm 3.994483555787e-02</div>
                                <div class="gmail_quote">  3 KSP
                                  Residual norm 6.085841461433e-03</div>
                                <div class="gmail_quote">  4 KSP
                                  Residual norm 8.876162583511e-04</div>
                                <div class="gmail_quote">  5 KSP
                                  Residual norm 9.407780665278e-05</div>
                                <div class="gmail_quote">Number of
                                  iterations =   5</div>
                                <div class="gmail_quote">Residual norm
                                  0.00542891</div>
                                <div class="gmail_quote"><br>
                                </div>
                                <div class="gmail_quote">Hong</div>
                              </div>
                            </div>
                            <blockquote class="gmail_quote"
                              style="margin:0px 0px 0px
                              0.8ex;border-left:1px solid
                              rgb(204,204,204);padding-left:1ex">
                              <div bgcolor="#FFFFFF">
                                <p>Hi Matt,</p>
                                <p>Yes. The matrix is 450000x450000
                                  sparse. The hypre takes hundreds of
                                  iterates, not for all but in most of
                                  the timesteps. The matrix is not well
                                  conditioned, with nonzero entries
                                  range from 1.0e-29 to 1.0e2. I also
                                  made double check if there is anything
                                  wrong in the parallel version,
                                  however, the matrix is the same with
                                  sequential version except some round
                                  error which is relatively very small.
                                  Usually for those not well conditioned
                                  matrix, direct solver should be faster
                                  than iterative solver, right? But when
                                  I use the sequential iterative solver
                                  with ILU prec developed almost 20
                                  years go by others, the solver
                                  converge fast with appropriate
                                  factorization level. In other words,
                                  when I use 24 processor using hypre,
                                  the speed is almost the same as as the
                                  old sequential iterative solver using
                                  1 processor.</p>
                                <p>I use most of the default
                                  configuration for the general case
                                  with pretty good speedup. And I am not
                                  sure if I miss something for this
                                  problem.</p>
                                <p>Thanks,</p>
                                <p>Danyang<br>
                                </p>
                                <div>
                                  <div
                                    class="m_9041936809186897311gmail-h5">
                                    <br>
                                    <div
                                      class="m_9041936809186897311gmail-m_1680376947984306833moz-cite-prefix">On
                                      17-05-24 11:12 AM, Matthew Knepley
                                      wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr">
                                        <div class="gmail_extra">
                                          <div class="gmail_quote">On
                                            Wed, May 24, 2017 at 12:50
                                            PM, Danyang Su <span
                                              dir="ltr"><<a
                                                moz-do-not-send="true"
                                                href="mailto:danyang.su@gmail.com"
                                                target="_blank">danyang.su@gmail.com</a>></span>
                                            wrote:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0px 0px 0px
                                              0.8ex;border-left:1px
                                              solid
                                              rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF">
                                                <p>Hi Matthew and Barry,</p>
                                                <p>Thanks for the quick
                                                  response. <br>
                                                </p>
                                                <p>I also tried superlu
                                                  and mumps, both work
                                                  but it is about four
                                                  times slower than
                                                  ILU(dt) prec through
                                                  hypre, with 24
                                                  processors I have
                                                  tested.</p>
                                              </div>
                                            </blockquote>
                                            <div>You mean the total time
                                              is 4x? And you are taking
                                              hundreds of iterates? That
                                              seems hard to believe,
                                              unless you are dropping</div>
                                            <div>a huge number of
                                              elements. <br>
                                            </div>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0px 0px 0px
                                              0.8ex;border-left:1px
                                              solid
                                              rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF">
                                                <p> When I look into the
                                                  convergence
                                                  information, the
                                                  method using ILU(dt)
                                                  still takes 200 to
                                                  3000 linear iterations
                                                  for each newton
                                                  iteration. One reason
                                                  is this equation is
                                                  hard to solve. As for
                                                  the general cases, the
                                                  same method works
                                                  awesome and get very
                                                  good speedup.</p>
                                              </div>
                                            </blockquote>
                                            <div>I do not understand
                                              what you mean here. <br>
                                            </div>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0px 0px 0px
                                              0.8ex;border-left:1px
                                              solid
                                              rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF">
                                                <p> I also doubt if I
                                                  use hypre correctly
                                                  for this case. Is
                                                  there anyway to check
                                                  this problem, or is it
                                                  possible to increase
                                                  the factorization
                                                  level through hypre?</p>
                                              </div>
                                            </blockquote>
                                            <div>I don't know.</div>
                                            <div><br>
                                            </div>
                                            <div>  Matt <br>
                                            </div>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0px 0px 0px
                                              0.8ex;border-left:1px
                                              solid
                                              rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF">
                                                <p>Thanks,</p>
                                                <p>Danyang<br>
                                                </p>
                                                <br>
                                                <div
class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
                                                  17-05-24 04:59 AM,
                                                  Matthew Knepley wrote:<br>
                                                </div>
                                                <blockquote type="cite">
                                                  <div dir="ltr">
                                                    <div
                                                      class="gmail_extra">
                                                      <div
                                                        class="gmail_quote">On
                                                        Wed, May 24,
                                                        2017 at 2:21 AM,
                                                        Danyang Su <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:danyang.su@gmail.com"
                                                          target="_blank">danyang.su@gmail.com</a>></span>
                                                        wrote:<br>
                                                        <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">Dear
                                                          All,<br>
                                                          <br>
                                                          I use
                                                          PCFactorSetLevels
                                                          for ILU and
                                                          PCFactorSetFill
                                                          for other
                                                          preconditioning
                                                          in my code to
                                                          help solve the
                                                          problems that
                                                          the default
                                                          option is hard
                                                          to solve.
                                                          However, I
                                                          found the
                                                          latter one,
                                                          PCFactorSetFill
                                                          does not take
                                                          effect for my
                                                          problem. The
                                                          matrices and
                                                          rhs as well as
                                                          the solutions
                                                          are attached
                                                          from the link
                                                          below. I
                                                          obtain the
                                                          solution using
                                                          hypre
                                                          preconditioner
                                                          and it takes 7
                                                          and 38
                                                          iterations for
                                                          matrix 1 and
                                                          matrix 2.
                                                          However, if I
                                                          use other
                                                          preconditioner,
                                                          the solver
                                                          just failed at
                                                          the first
                                                          matrix. I have
                                                          tested this
                                                          matrix using
                                                          the native
                                                          sequential
                                                          solver (not
                                                          PETSc) with
                                                          ILU
                                                          preconditioning.
                                                          If I set the
                                                          incomplete
                                                          factorization
                                                          level to 0,
                                                          this
                                                          sequential
                                                          solver will
                                                          take more than
                                                          100
                                                          iterations. If
                                                          I increase the
                                                          factorization
                                                          level to 1 or
                                                          more, it just
                                                          takes several
                                                          iterations.
                                                          This remind me
                                                          that the PC
                                                          factor for
                                                          this matrices
                                                          should be
                                                          increased.
                                                          However, when
                                                          I tried it in
                                                          PETSc, it just
                                                          does not work.<br>
                                                          <br>
                                                          Matrix and rhs
                                                          can be
                                                          obtained from
                                                          the link
                                                          below.<br>
                                                          <br>
                                                          <a
                                                          moz-do-not-send="true"
href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R"
                                                          rel="noreferrer"
target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
                                                          <br>
                                                          Would anyone
                                                          help to check
                                                          if you can
                                                          make this work
                                                          by increasing
                                                          the PC factor
                                                          level or fill?<br>
                                                        </blockquote>
                                                        <div><br>
                                                        </div>
                                                        <div>We have
                                                          ILU(k)
                                                          supported in
                                                          serial.
                                                          However
                                                          ILU(dt) which
                                                          takes a
                                                          tolerance only
                                                          works through
                                                          Hypre</div>
                                                        <div><br>
                                                        </div>
                                                        <div>  <a
                                                          moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html"
target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
                                                        <div><br>
                                                        </div>
                                                        <div>I recommend
                                                          you try
                                                          SuperLU or
                                                          MUMPS, which
                                                          can both be
                                                          downloaded
                                                          automatically
                                                          by configure,
                                                          and</div>
                                                        <div>do a full
                                                          sparse LU.</div>
                                                        <div><br>
                                                        </div>
                                                        <div>  Thanks,</div>
                                                        <div><br>
                                                        </div>
                                                        <div>    Matt</div>
                                                        <div> </div>
                                                        <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          Thanks and
                                                          regards,<br>
                                                          <br>
                                                          Danyang<br>
                                                          <br>
                                                          <br>
                                                        </blockquote>
                                                      </div>
                                                      <br>
                                                      <br clear="all">
                                                      <span
                                                        class="m_9041936809186897311gmail-m_1680376947984306833HOEnZb"><font
color="#888888">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div
class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a
                                                          moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </font></span></div>
                                                  </div>
                                                </blockquote>
                                                <br>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                          <br clear="all">
                                          <div><br>
                                          </div>
                                          -- <br>
                                          <div
                                            class="m_9041936809186897311gmail-m_1680376947984306833gmail_signature">
                                            <div dir="ltr">
                                              <div>What most
                                                experimenters take for
                                                granted before they
                                                begin their experiments
                                                is infinitely more
                                                interesting than any
                                                results to which their
                                                experiments lead.<br>
                                                -- Norbert Wiener</div>
                                              <div><br>
                                              </div>
                                              <div><a
                                                  moz-do-not-send="true"
href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>