<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Matt<br>
      <br>
      With MUMPS it is indeed the same but not faster. E.g., see 1.txt
      and 2.txt<br>
      <br>
      With GAMG, cg takes forever (see 4.txt). For reference GAMG with
      preonly is attached again (3.txt)<br>
      <br>
      All logs include -fieldsplit_0_ksp_monitor.<br>
      <br>
      Also, the true residual shows small oscillations. Is that normal?<br>
      <br>
      Tabrez<br>
      <br>
      <br>
      On 10/28/2014 07:08 AM, Matthew Knepley wrote:<br>
    </div>
    <blockquote
cite="mid:CAMYG4Gm3bvb63k7WvH=B0fXuA5GZ-qvmFhvU3grXjg4ZC4Z0LA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">On Tue, Oct 28, 2014 at 5:39 AM,
            Tabrez Ali <span dir="ltr"><<a moz-do-not-send="true"
                href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div>Mark<br>
                  <br>
                  When I replace "-fieldsplit_0_ksp_type preonly" with
                  "-fieldsplit_0_ksp_type cg" then it becomes very slow
                  (had to kill it).<br>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>That means something in the setup is wrong. It should
              be about the same or faster. Run with
              -fieldsplit_0_ksp_monitor</div>
            <div>so we can see what is happening.</div>
            <div><br>
            </div>
            <div>   Matt</div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div> With MUMPS , i.e., with '-fieldsplit_0_pc_type lu
                  -fieldsplit_0_pc_factor_mat_solver_package mumps
                  -fieldsplit_0_ksp_type preonly' it works fine but
                  takes more time, and will be an issue for larger
                  problems. The output for this run is attached.<br>
                  <br>
                  I will work on passing rigid body modes (as Matt
                  mentioned) but short of that what is the best set of
                  options for solving the following problem (i.e.,
                  linear elasticity with constraints):<br>
                  <br>
                  <tt>|K cG'| | u | = |F|</tt><tt><br>
                  </tt><tt>|G  0 | |l/c|   |d|</tt><br>
                  <br>
                  where c is a scaling factor (so that cG' terms are
                  more or less of the same order as K)? The constraints
                  are used to impose slip between surfaces and so on.<br>
                  <br>
                  Tabrez<br>
                  <br>
                  On 10/27/2014 01:17 PM, Mark Adams wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">The null space for GAMG is not critical
                    but useful for elasticity.  If you in fact have an
                    indefinite operator (eg, not "pinned)  the you need
                    to use an iterative coarse grid solver.  You are
                    using '-fieldsplit_0_pc_type gamg
                    -fieldsplit_0_ksp_type preonly'.  And you have a
                    hard elasticity problem.  You are going to want to
                    start with a stronger solver. Use cg instead of
                    preonly.  As Matt said start with MUMPS, then go to
                    CG/GAMG, then you can see how far you can cut the
                    _0_ solver down.
                    <div>Mark</div>
                  </div>
                  <div class="gmail_extra"><br>
                    <div class="gmail_quote">On Thu, Oct 23, 2014 at
                      11:51 AM, Matthew Knepley <span dir="ltr"><<a
                          moz-do-not-send="true"
                          href="mailto:knepley@gmail.com"
                          target="_blank">knepley@gmail.com</a>></span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div dir="ltr">
                          <div class="gmail_extra">
                            <div class="gmail_quote"><span>On Thu, Oct
                                23, 2014 at 10:48 AM, Tabrez Ali <span
                                  dir="ltr"><<a
                                    moz-do-not-send="true"
                                    href="mailto:stali@geology.wisc.edu"
                                    target="_blank">stali@geology.wisc.edu</a>></span>
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div text="#000000" bgcolor="#ffffff">
                                    Matt<br>
                                    <br>
                                    On 10/23/2014 09:54 AM, Matthew
                                    Knepley wrote:
                                    <blockquote type="cite">
                                      <div dir="ltr">
                                        <div class="gmail_extra">
                                          <div class="gmail_quote">On
                                            Thu, Oct 23, 2014 at 9:27
                                            AM, Tabrez Ali <span
                                              dir="ltr"><<a
                                                moz-do-not-send="true"
                                                href="mailto:stali@geology.wisc.edu"
                                                target="_blank">stali@geology.wisc.edu</a>></span>
                                            wrote:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0px 0px 0px
                                              0.8ex;border-left:1px
                                              solid
                                              rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF"
                                                text="#000000">
                                                <div>Matt<br>
                                                  <br>
                                                  Sorry about that (I
                                                  always forget it). The
                                                  output for the
                                                  smallest problem is
                                                  now attached (see
                                                  log.txt). I am also
                                                  attaching some results
                                                  that compare results
                                                  obtained using FS/LSC
                                                  and the direct solver
                                                  (MUMPS), again for the
                                                  smallest problem. The
                                                  difference, as you can
                                                  see is insignificant
                                                  O(1E-6).<br>
                                                </div>
                                              </div>
                                            </blockquote>
                                            <div><br>
                                            </div>
                                            <div>1) How do you use MUMPS
                                              if you have a saddle point</div>
                                          </div>
                                        </div>
                                      </div>
                                    </blockquote>
                                    I simply used -pc_type lu
                                    -pc_factor_mat_solver_package mumps.<br>
                                    <br>
                                    <blockquote type="cite">
                                      <div dir="ltr">
                                        <div class="gmail_extra">
                                          <div class="gmail_quote">
                                            <div><br>
                                            </div>
                                            <div>2) You can see from the
                                              output that something is
                                              seriously wrong with the
                                              preconditioner. It looks
                                              like it has a null space.</div>
                                            <div>    Did you add the
                                              elastic null modes to
                                              GAMG? Without this, it is
                                              not going to work. We have
                                              helper functions for this:</div>
                                            <div><br>
                                            </div>
                                            <div>  <a
                                                moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html"
                                                target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
                                            <div><br>
                                            </div>
                                            <div>you could just copy
                                              that code. And then use</div>
                                            <div><br>
                                            </div>
                                            <div>  <a
                                                moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html"
                                                target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
                                            <div><br>
                                            </div>
                                            <div>I don't see it in the
                                              output, so I think this is
                                              your problem.</div>
                                            <div><br>
                                            </div>
                                            <div>In order to test, I
                                              would first use MUMPS as
                                              the A00 solver and get the
                                              Schur stuff worked out.
                                              Then I would</div>
                                            <div>replace MUMPS with GAMG
                                              and tune it until I get
                                              back my original
                                              convergence.</div>
                                          </div>
                                        </div>
                                      </div>
                                    </blockquote>
                                    I will try this with
                                    MatNullSpaceCreateRigidBody. Btw
                                    does it matter if some nodes are
                                    pinned?<br>
                                  </div>
                                </blockquote>
                                <div><br>
                                </div>
                              </span>
                              <div>No these are null modes of the
                                operator, not of the particular problem.</div>
                              <div><br>
                              </div>
                              <div>  Matt</div>
                              <div>
                                <div>
                                  <div> </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div text="#000000"
                                      bgcolor="#ffffff"> Tabrez<br>
                                      <br>
                                      <blockquote type="cite">
                                        <div dir="ltr">
                                          <div class="gmail_extra">
                                            <div class="gmail_quote">
                                              <div><br>
                                              </div>
                                              <div>  Thanks,</div>
                                              <div><br>
                                              </div>
                                              <div>    Matt</div>
                                              <div> </div>
                                              <blockquote
                                                class="gmail_quote"
                                                style="margin:0px 0px
                                                0px
                                                0.8ex;border-left:1px
                                                solid
                                                rgb(204,204,204);padding-left:1ex">
                                                <div bgcolor="#FFFFFF"
                                                  text="#000000">
                                                  <div> Also, I did pass
                                                    'upper' and 'full'
                                                    to
                                                    '-pc_fieldsplit_schur_factorization_type'
                                                    but the iteration
                                                    count doesn't
                                                    improve (in fact, it
                                                    increases slightly).
                                                    The attached log is
                                                    with 'upper'.<br>
                                                    <br>
                                                    Regards,<br>
                                                    <br>
                                                    Tabrez<br>
                                                    <br>
                                                    On 10/23/2014 07:46
                                                    AM, Matthew Knepley
                                                    wrote:<br>
                                                  </div>
                                                  <blockquote
                                                    type="cite">
                                                    <div dir="ltr">
                                                      <div
                                                        class="gmail_extra">
                                                        <div
                                                          class="gmail_quote">On

                                                          Thu, Oct 23,
                                                          2014 at 7:20
                                                          AM, Tabrez Ali
                                                          <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
                                                          target="_blank">stali@geology.wisc.edu</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0px
                                                          0px 0px
                                                          0.8ex;border-left:1px
                                                          solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          Hello<br>
                                                          <br>
                                                          I am using the
                                                          following
                                                          options
                                                          (below) for
                                                          solving linear
                                                          elasticity/poroelasticity

                                                          problems
                                                          involving slip
                                                          between two
                                                          surfaces
                                                          involving
                                                          non-trivial
                                                          geometries,
                                                          i.e., elements
                                                          with high
                                                          aspect ratios,
                                                          large
                                                          contrasts in
                                                          material
                                                          properties
                                                          etc. The
                                                          constraints
                                                          are imposed
                                                          using Lagrange
                                                          Multipliers. <br>
                                                          <br>
                                                          A picture
                                                          (shows
                                                          displacement
                                                          magnitude) is
                                                          attached. The
                                                          boundary
                                                          nodes, i.e.,
                                                          the base and
                                                          the four side
                                                          are pinned.<br>
                                                          <br>
                                                          The following
                                                          options appear
                                                          to work well
                                                          for the saddle
                                                          point problem:<br>
                                                          <br>
                                                          <tt>-pc_type
                                                          fieldsplit
                                                          -pc_fieldsplit_type
                                                          schur
                                                          -pc_fieldsplit_detect_saddle_point
                                                          -fieldsplit_0_pc_type

                                                          gamg
                                                          -fieldsplit_0_ksp_type
                                                          preonly
                                                          -fieldsplit_1_pc_type
                                                          lsc
                                                          -fieldsplit_1_ksp_type
                                                          preonly
                                                          -pc_fieldsplit_schur_fact_type
                                                          lower
                                                          -ksp_monitor</tt><br>
                                                          <br>
                                                          However, the
                                                          number of
                                                          iterations
                                                          keep on
                                                          increasing
                                                          with the
                                                          problems size
                                                          (see attached
                                                          plot), e.g.,<br>
                                                          <br>
                                                          <tt>120K
                                                          Tets        <b>507</b>
                                                          Iterations
                                                          (KSP Residual
                                                          norm
                                                          8.827362494659e-05)</tt><tt>
                                                          in  17 secs on
                                                            3 cores<br>
                                                          </tt><tt>1
                                                          Million Tets 
                                                          <b>1374</b>
                                                          Iterations
                                                          (KSP Residual
                                                          norm
                                                          7.164704416296e-05)</tt><tt>
                                                          in 117 secs
                                                          on  20 cores<br>
                                                          </tt><tt>8
                                                          Million Tets 
                                                          <b>2495</b>
                                                          Iterations
                                                          (KSP Residual
                                                          norm
                                                          9.101247550026e-05)
                                                          in 225 secs on
                                                          160 cores</tt><br>
                                                          <br>
                                                          So what other
                                                          options should
                                                          I try to
                                                          improve solver
                                                          performance?
                                                          Any
                                                          tips/insights
                                                          would be
                                                          appreciated as
                                                          preconditioning
                                                          is black magic
                                                          to me.<br>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>For
                                                          reports,
                                                          always run
                                                          with </div>
                                                          <div><br>
                                                          </div>
                                                          <div> 
                                                          -ksp_view
                                                          -ksp_monitor_true_residual
-ksp_converged_reason</div>
                                                          <div><br>
                                                          </div>
                                                          <div>so that
                                                          we can see
                                                          exactly what
                                                          you used.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I believe
                                                          the default is
                                                          a diagonal
                                                          factorization.
                                                          Since your
                                                          outer iterates
                                                          are
                                                          increasing, I
                                                          would
                                                          strengthen
                                                          this</div>
                                                          <div>to either
                                                          upper or full</div>
                                                          <div><br>
                                                          </div>
                                                          <div> 
                                                          -pc_fieldsplit_schur_factorization_type
                                                          <upper,
                                                          full></div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     
                                                          Matt </div>
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0px
                                                          0px 0px
                                                          0.8ex;border-left:1px
                                                          solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          Thanks in
                                                          advance.<span><font
color="#888888"><br>
                                                          <br>
                                                          Tabrez<br>
                                                          </font></span></div>
                                                          </blockquote>
                                                        </div>
                                                        <br>
                                                        <br clear="all">
                                                        <span><font
                                                          color="#888888">
                                                          <span><font
                                                          color="#888888">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener </font></span></font></span></div>
                                                      <span><font
                                                          color="#888888">
                                                        </font></span></div>
                                                    <span><font
                                                        color="#888888">
                                                      </font></span></blockquote>
                                                  <span><font
                                                      color="#888888"> <br>
                                                    </font></span></div>
                                                <span><font
                                                    color="#888888"> </font></span></blockquote>
                                              <span><font
                                                  color="#888888"> </font></span></div>
                                            <span><font color="#888888">
                                                <br>
                                                <br clear="all">
                                                <span class="HOEnZb"><font
                                                    color="#888888">
                                                    <div><br>
                                                    </div>
                                                    -- <br>
                                                    What most
                                                    experimenters take
                                                    for granted before
                                                    they begin their
                                                    experiments is
                                                    infinitely more
                                                    interesting than any
                                                    results to which
                                                    their experiments
                                                    lead.<br>
                                                    -- Norbert Wiener</font></span></font></span></div>
                                          <span class="HOEnZb"><font
                                              color="#888888"> </font></span></div>
                                        <span class="HOEnZb"><font
                                            color="#888888"> </font></span></blockquote>
                                      <span class="HOEnZb"><font
                                          color="#888888"> </font></span></div>
                                    <span class="HOEnZb"><font
                                        color="#888888"> </font></span></blockquote>
                                  <span class="HOEnZb"><font
                                      color="#888888"> </font></span></div>
                                <span class="HOEnZb"><font
                                    color="#888888"> </font></span></div>
                              <span class="HOEnZb"><font color="#888888">
                                </font></span></div>
                            <span class="HOEnZb"><font color="#888888">
                                <div>
                                  <div><br>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    What most experimenters take for
                                    granted before they begin their
                                    experiments is infinitely more
                                    interesting than any results to
                                    which their experiments lead.<br>
                                    -- Norbert Wiener </div>
                                </div>
                              </font></span></div>
                          <span class="HOEnZb"><font color="#888888"> </font></span></div>
                        <span class="HOEnZb"><font color="#888888"> </font></span></blockquote>
                      <span class="HOEnZb"><font color="#888888"> </font></span></div>
                    <span class="HOEnZb"><font color="#888888"> <br>
                      </font></span></div>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          What most experimenters take for granted before they begin
          their experiments is infinitely more interesting than any
          results to which their experiments lead.<br>
          -- Norbert Wiener
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>