<div dir="ltr"><span style="font-size:12.8px">Remove your option '</span><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">-vecload_block_size 10'.</span><br><div><span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap"><br></span></div><div><font color="#000000" face="Courier New, Courier, monospace, arial, sans-serif"><span style="font-size:14px;white-space:pre-wrap">Hong</span></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 3:06 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <p>Dear Hong,</p>
    <p>I just tested with different number of processors for the same
      matrix. It sometimes got "ERROR: Arguments are incompatible" for
      different number of processors. It works fine using 4, 8, or 24
      processors, but failed with "ERROR: Arguments are incompatible"
      using 16 or 48 processors. The error information is attached. I
      tested this on my local computer with 6 cores 12 threads. Any
      suggestion on this?</p>
    <p>Thanks,<br>
    </p>
    <p>Danyang<br>
    </p><div><div class="h5">
    <br>
    <div class="m_9041936809186897311moz-cite-prefix">On 17-05-24 12:28 PM, Danyang Su wrote:<br>
    </div>
    <blockquote type="cite">
      
      <p>Hi Hong,</p>
      <p>Awesome. Thanks for testing the case. I will try your options
        for the code and get back to you later.</p>
      <p>Regards,</p>
      <p>Danyang<br>
      </p>
      <br>
      <div class="m_9041936809186897311moz-cite-prefix">On 17-05-24 12:21 PM, Hong wrote:<br>
      </div>
      <blockquote type="cite">
        <div dir="ltr">
          <div class="gmail_extra">
            <div class="gmail_quote">Danyang :</div>
            <div class="gmail_quote">I tested your data.</div>
            <div class="gmail_quote">Your matrices encountered zero
              pivots, e.g.</div>
            <div class="gmail_quote">
              <div class="gmail_quote">petsc/src/ksp/ksp/examples/<wbr>tutorials
                (master)</div>
              <div class="gmail_quote">$ mpiexec -n 24 ./ex10 -f0
                a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
                -ksp_error_if_not_converged</div>
              <div class="gmail_quote"><br>
              </div>
              <div class="gmail_quote">
                <div class="gmail_quote">[15]PETSC ERROR: Zero pivot in
                  LU factorization: <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>zeropivot</a></div>
                <div class="gmail_quote">[15]PETSC ERROR: Zero pivot row
                  1249 value 2.05808e-14 tolerance 2.22045e-14</div>
                <div class="gmail_quote">...</div>
                <div class="gmail_quote"><br>
                </div>
                <div class="gmail_quote">Adding option
                  '-sub_pc_factor_shift_type nonzero', I got</div>
                <div class="gmail_quote">
                  <div class="gmail_quote">mpiexec -n 24 ./ex10 -f0
                    a_react_in_2.bin -rhs b_react_in_2.bin -ksp_monitor
                    -ksp_error_if_not_converged
                    -sub_pc_factor_shift_type nonzero -mat_view
                    ascii::ascii_info</div>
                  <div class="gmail_quote"><br>
                  </div>
                  <div class="gmail_quote">Mat Object: 24 MPI processes</div>
                  <div class="gmail_quote">  type: mpiaij</div>
                  <div class="gmail_quote">  rows=450000, cols=450000</div>
                  <div class="gmail_quote">  total: nonzeros=6991400,
                    allocated nonzeros=6991400</div>
                  <div class="gmail_quote">  total number of mallocs
                    used during MatSetValues calls =0</div>
                  <div class="gmail_quote">    not using I-node (on
                    process 0) routines</div>
                  <div class="gmail_quote">  0 KSP Residual norm
                    5.849777711755e+01</div>
                  <div class="gmail_quote">  1 KSP Residual norm
                    6.824179430230e-01</div>
                  <div class="gmail_quote">  2 KSP Residual norm
                    3.994483555787e-02</div>
                  <div class="gmail_quote">  3 KSP Residual norm
                    6.085841461433e-03</div>
                  <div class="gmail_quote">  4 KSP Residual norm
                    8.876162583511e-04</div>
                  <div class="gmail_quote">  5 KSP Residual norm
                    9.407780665278e-05</div>
                  <div class="gmail_quote">Number of iterations =   5</div>
                  <div class="gmail_quote">Residual norm 0.00542891</div>
                  <div class="gmail_quote"><br>
                  </div>
                  <div class="gmail_quote">Hong</div>
                </div>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                <div bgcolor="#FFFFFF">
                  <p>Hi Matt,</p>
                  <p>Yes. The matrix is 450000x450000 sparse. The hypre
                    takes hundreds of iterates, not for all but in most
                    of the timesteps. The matrix is not well
                    conditioned, with nonzero entries range from 1.0e-29
                    to 1.0e2. I also made double check if there is
                    anything wrong in the parallel version, however, the
                    matrix is the same with sequential version except
                    some round error which is relatively very small.
                    Usually for those not well conditioned matrix,
                    direct solver should be faster than iterative
                    solver, right? But when I use the sequential
                    iterative solver with ILU prec developed almost 20
                    years go by others, the solver converge fast with
                    appropriate factorization level. In other words,
                    when I use 24 processor using hypre, the speed is
                    almost the same as as the old sequential iterative
                    solver using 1 processor.</p>
                  <p>I use most of the default configuration for the
                    general case with pretty good speedup. And I am not
                    sure if I miss something for this problem.</p>
                  <p>Thanks,</p>
                  <p>Danyang<br>
                  </p>
                  <div>
                    <div class="m_9041936809186897311gmail-h5"> <br>
                      <div class="m_9041936809186897311gmail-m_1680376947984306833moz-cite-prefix">On
                        17-05-24 11:12 AM, Matthew Knepley wrote:<br>
                      </div>
                      <blockquote type="cite">
                        <div dir="ltr">
                          <div class="gmail_extra">
                            <div class="gmail_quote">On Wed, May 24,
                              2017 at 12:50 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
                              wrote:<br>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p>Hi Matthew and Barry,</p>
                                  <p>Thanks for the quick response. <br>
                                  </p>
                                  <p>I also tried superlu and mumps,
                                    both work but it is about four times
                                    slower than ILU(dt) prec through
                                    hypre, with 24 processors I have
                                    tested.</p>
                                </div>
                              </blockquote>
                              <div>You mean the total time is 4x? And
                                you are taking hundreds of iterates?
                                That seems hard to believe, unless you
                                are dropping</div>
                              <div>a huge number of elements. <br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p> When I look into the convergence
                                    information, the method using
                                    ILU(dt) still takes 200 to 3000
                                    linear iterations for each newton
                                    iteration. One reason is this
                                    equation is hard to solve. As for
                                    the general cases, the same method
                                    works awesome and get very good
                                    speedup.</p>
                                </div>
                              </blockquote>
                              <div>I do not understand what you mean
                                here. <br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p> I also doubt if I use hypre
                                    correctly for this case. Is there
                                    anyway to check this problem, or is
                                    it possible to increase the
                                    factorization level through hypre?</p>
                                </div>
                              </blockquote>
                              <div>I don't know.</div>
                              <div><br>
                              </div>
                              <div>  Matt <br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p>Thanks,</p>
                                  <p>Danyang<br>
                                  </p>
                                  <br>
                                  <div class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121moz-cite-prefix">On
                                    17-05-24 04:59 AM, Matthew Knepley
                                    wrote:<br>
                                  </div>
                                  <blockquote type="cite">
                                    <div dir="ltr">
                                      <div class="gmail_extra">
                                        <div class="gmail_quote">On Wed,
                                          May 24, 2017 at 2:21 AM,
                                          Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span>
                                          wrote:<br>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear All,<br>
                                            <br>
                                            I use PCFactorSetLevels for
                                            ILU and PCFactorSetFill for
                                            other preconditioning in my
                                            code to help solve the
                                            problems that the default
                                            option is hard to solve.
                                            However, I found the latter
                                            one, PCFactorSetFill does
                                            not take effect for my
                                            problem. The matrices and
                                            rhs as well as the solutions
                                            are attached from the link
                                            below. I obtain the solution
                                            using hypre preconditioner
                                            and it takes 7 and 38
                                            iterations for matrix 1 and
                                            matrix 2. However, if I use
                                            other preconditioner, the
                                            solver just failed at the
                                            first matrix. I have tested
                                            this matrix using the native
                                            sequential solver (not
                                            PETSc) with ILU
                                            preconditioning. If I set
                                            the incomplete factorization
                                            level to 0, this sequential
                                            solver will take more than
                                            100 iterations. If I
                                            increase the factorization
                                            level to 1 or more, it just
                                            takes several iterations.
                                            This remind me that the PC
                                            factor for this matrices
                                            should be increased.
                                            However, when I tried it in
                                            PETSc, it just does not
                                            work.<br>
                                            <br>
                                            Matrix and rhs can be
                                            obtained from the link
                                            below.<br>
                                            <br>
                                            <a href="https://eilinator.eos.ubc.ca:8443/index.php/s/CalUcq9CMeblk4R" rel="noreferrer" target="_blank">https://eilinator.eos.ubc.ca:8<wbr>443/index.php/s/CalUcq9CMeblk4<wbr>R</a><br>
                                            <br>
                                            Would anyone help to check
                                            if you can make this work by
                                            increasing the PC factor
                                            level or fill?<br>
                                          </blockquote>
                                          <div><br>
                                          </div>
                                          <div>We have ILU(k) supported
                                            in serial. However ILU(dt)
                                            which takes a tolerance only
                                            works through Hypre</div>
                                          <div><br>
                                          </div>
                                          <div>  <a href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html" target="_blank">http://www.mcs.anl.gov/petsc<wbr>/documentation/linearsolvertab<wbr>le.html</a></div>
                                          <div><br>
                                          </div>
                                          <div>I recommend you try
                                            SuperLU or MUMPS, which can
                                            both be downloaded
                                            automatically by configure,
                                            and</div>
                                          <div>do a full sparse LU.</div>
                                          <div><br>
                                          </div>
                                          <div>  Thanks,</div>
                                          <div><br>
                                          </div>
                                          <div>    Matt</div>
                                          <div> </div>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks and regards,<br>
                                            <br>
                                            Danyang<br>
                                            <br>
                                            <br>
                                          </blockquote>
                                        </div>
                                        <br>
                                        <br clear="all">
                                        <span class="m_9041936809186897311gmail-m_1680376947984306833HOEnZb"><font color="#888888">
                                            <div><br>
                                            </div>
                                            -- <br>
                                            <div class="m_9041936809186897311gmail-m_1680376947984306833m_-1065376578025428121gmail_signature">
                                              <div dir="ltr">
                                                <div>What most
                                                  experimenters take for
                                                  granted before they
                                                  begin their
                                                  experiments is
                                                  infinitely more
                                                  interesting than any
                                                  results to which their
                                                  experiments lead.<br>
                                                  -- Norbert Wiener</div>
                                                <div><br>
                                                </div>
                                                <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
                                                </div>
                                              </div>
                                            </div>
                                          </font></span></div>
                                    </div>
                                  </blockquote>
                                  <br>
                                </div>
                              </blockquote>
                            </div>
                            <br>
                            <br clear="all">
                            <div><br>
                            </div>
                            -- <br>
                            <div class="m_9041936809186897311gmail-m_1680376947984306833gmail_signature">
                              <div dir="ltr">
                                <div>What most experimenters take for
                                  granted before they begin their
                                  experiments is infinitely more
                                  interesting than any results to which
                                  their experiments lead.<br>
                                  -- Norbert Wiener</div>
                                <div><br>
                                </div>
                                <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">http://www.caam.rice.edu/~mk51<wbr>/</a><br>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </blockquote>
                      <br>
                    </div>
                  </div>
                </div>
              </blockquote>
            </div>
            <br>
          </div>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br></div>