<div dir="ltr"><div dir="ltr">On Tue, Mar 26, 2019 at 9:27 AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr">myriam.peyrounette@idris.fr</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>I checked with -ksp_view (attached) but no prefix is associated
      with the matrix. Some are associated to the KSP and PC, but none
      to the Mat</p></div></blockquote><div>Another thing that could prevent options being used is that *SetFromOptions() is not called for the object.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <div class="gmail-m_9164101223360987041moz-cite-prefix">Le 03/26/19 à 11:55, Dave May a écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr"><br>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at
              10:36, Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p>Oh you were right, the three options are unsused
                  (-matptap_via scalable, -inner_offdiag_matmatmult_via
                  scalable and -inner_diag_matmatmult_via scalable).
                  Does this mean I am not using the associated PtAP
                  functions?</p>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>No - not necessarily. All it means is the options were
              not parsed. </div>
            <div><br>
            </div>
            <div>If your matrices have an option prefix associated with
              them (e.g. abc) , then you need to provide the option as</div>
            <div>  -abc_matptap_via scalable</div>
            <div><br>
            </div>
            <div>If you are not sure if you matrices have a prefix, look
              at the result of -ksp_view (see below for an example)</div>
            <div><br>
            </div>
            <div>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">  </span>Mat
                  Object: 2 MPI processes</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>type:
                  mpiaij</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>rows=363,
                  cols=363, bs=3</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>total:
                  nonzeros=8649, allocated nonzeros=8649</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>total
                  number of mallocs used during MatSetValues calls =0</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">  </span>Mat
                  Object: (B_) 2 MPI processes</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>type:
                  mpiaij</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>rows=363,
                  cols=363, bs=3</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>total:
                  nonzeros=8649, allocated nonzeros=8649</span></p>
              <p class="gmail-m_9164101223360987041gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_9164101223360987041gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_9164101223360987041gmail-Apple-converted-space">    </span>total
                  number of mallocs used during MatSetValues calls =0</span></p>
            </div>
            <div><br>
            </div>
            <div>The first matrix has no options prefix, but the second
              does and it's called "B_".</div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div>  <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p>Myriam<br>
                </p>
                <br>
                <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615moz-cite-prefix">Le
                  03/26/19 à 11:10, Dave May a écrit :<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div dir="ltr"><br>
                      <div class="gmail_quote">
                        <div dir="ltr" class="gmail_attr">On Tue, 26 Mar
                          2019 at 09:52, Myriam Peyrounette via
                          petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                          wrote:<br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div bgcolor="#FFFFFF">
                            <p>How can I be sure they are indeed used?
                              Can I print this information in some log
                              file? <br>
                            </p>
                          </div>
                        </blockquote>
                        <div>Yes. Re-run the job with the command line
                          option</div>
                        <div><br>
                        </div>
                        <div>-options_left true<br>
                        </div>
                        <div><br>
                        </div>
                        <div>This will report all options parsed, and
                          importantly, will also indicate if any options
                          were unused.</div>
                        <div> </div>
                        <div><br>
                        </div>
                        <div>Thanks</div>
                        <div>Dave</div>
                        <div><br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div bgcolor="#FFFFFF">
                            <p> </p>
                            <p>Thanks in advance<br>
                            </p>
                            <p>Myriam<br>
                            </p>
                            <br>
                            <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-cite-prefix">Le
                              03/25/19 à 18:24, Matthew Knepley a
                              écrit :<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div dir="ltr">On Mon, Mar 25, 2019 at
                                  10:54 AM Myriam Peyrounette via
                                  petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                  wrote:<br>
                                </div>
                                <div class="gmail_quote">
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                    <div bgcolor="#FFFFFF">
                                      <p>Hi,</p>
                                      <p>thanks for the explanations. I
                                        tried the last PETSc version
                                        (commit
                                        fbc5705bc518d02a4999f188aad4ccff5f754cbf),
                                        which includes the patch you
                                        talked about. But the memory
                                        scaling shows no improvement
                                        (see scaling attached), even
                                        when using the "scalable"
                                        options :(</p>
                                      <p>I had a look at the PETSc
                                        functions
                                        MatPtAPNumeric_MPIAIJ_MPIAIJ and
                                        MatPtAPSymbolic_MPIAIJ_MPIAIJ
                                        (especially at the differences
                                        before and after the first "bad"
                                        commit), but I can't find what
                                        induced this memory issue.</p>
                                    </div>
                                  </blockquote>
                                  <div>Are you sure that the option was
                                    used? It just looks suspicious to me
                                    that they use exactly the same
                                    amount of memory. It should be
                                    different, even if it does not solve
                                    the problem.</div>
                                  <div><br>
                                  </div>
                                  <div>   Thanks,</div>
                                  <div><br>
                                  </div>
                                  <div>     Matt </div>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                    <div bgcolor="#FFFFFF">
                                      <p>Myriam<br>
                                      </p>
                                      <p><br>
                                      </p>
                                      <p><br>
                                      </p>
                                      <br>
                                      <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-cite-prefix">Le
                                        03/20/19 à 17:38, Fande Kong a
                                        écrit :<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div dir="ltr">
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div dir="ltr">
                                                    <div>Hi Myriam,</div>
                                                    <div><br>
                                                    </div>
                                                    <div>There are three
                                                      algorithms in
                                                      PETSc to do PtAP
                                                      ( const char     
                                                          *algTypes[3] =
{"scalable","nonscalable","hypre"};), and can be specified using the
                                                      petsc
                                                      options: -matptap_via
                                                      xxxx.</div>
                                                    <div><br>
                                                    </div>
                                                    <div>(1)
                                                      -matptap_via
                                                      hypre: This call
                                                      the hypre package
                                                      to do the PtAP
                                                      trough an
                                                      all-at-once triple
                                                      product. In our
                                                      experiences, it is
                                                      the most memory
                                                      efficient, but
                                                      could be slow.</div>
                                                    <div><br>
                                                    </div>
                                                    <div>(2) 
                                                      -matptap_via
                                                      scalable: This
                                                      involves a
                                                      row-wise algorithm
                                                      plus an outer
                                                      product.  This
                                                      will use more
                                                      memory than hypre,
                                                      but way faster.
                                                      This used to have
                                                      a bug that could
                                                      take all your
                                                      memory, and I have
                                                      a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>. 
                                                      When using this
                                                      option, we may
                                                      want to have extra
                                                      options such as 
                                                       -inner_offdiag_matmatmult_via
                                                      scalable
                                                      -inner_diag_matmatmult_via
                                                      scalable  to
                                                      select inner
                                                      scalable
                                                      algorithms.</div>
                                                    <div><br>
                                                    </div>
                                                    <div>(3)  -matptap_via
                                                      nonscalable: 
                                                      Suppose to be even
                                                      faster, but use
                                                      more memory. It
                                                      does dense matrix
                                                      operations.</div>
                                                    <div><br>
                                                    </div>
                                                    <div><br>
                                                    </div>
                                                    <div>Thanks,</div>
                                                    <div><br>
                                                    </div>
                                                    <div>Fande Kong</div>
                                                    <div><br>
                                                    </div>
                                                    <div><br>
                                                    </div>
                                                    <div><br>
                                                    </div>
                                                    <br>
                                                    <div class="gmail_quote">
                                                      <div dir="ltr" class="gmail_attr">On
                                                        Wed, Mar 20,
                                                        2019 at 10:06 AM
                                                        Myriam
                                                        Peyrounette via
                                                        petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                                        wrote:<br>
                                                      </div>
                                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                        <div bgcolor="#FFFFFF">
                                                          <p>More
                                                          precisely:
                                                          something
                                                          happens when
                                                          upgrading the
                                                          functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
                                                          </p>
                                                          <p>Unfortunately,
                                                          there are a
                                                          lot of
                                                          differences
                                                          between the
                                                          old and new
                                                          versions of
                                                          these
                                                          functions. I
                                                          keep
                                                          investigating
                                                          but if you
                                                          have any idea,
                                                          please let me
                                                          know.</p>
                                                          <p>Best,<br>
                                                          </p>
                                                          <p>Myriam<br>
                                                          </p>
                                                          <br>
                                                          <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                                                          03/20/19 à
                                                          13:48, Myriam
                                                          Peyrounette a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <p>Hi all,</p>
                                                          <p>I used git
                                                          bisect to
                                                          determine when
                                                          the memory
                                                          need
                                                          increased. I
                                                          found that the
                                                          first "bad"
                                                          commit is  
                                                          aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
                                                          <p>Barry was
                                                          right, this
                                                          commit seems
                                                          to be about an
                                                          evolution of <span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
                                                          You mentioned
                                                          the option
                                                          "-matptap_via
                                                          scalable" but
                                                          I can't find
                                                          any
                                                          information
                                                          about it. Can
                                                          you tell me
                                                          more?</span></span></p>
                                                          <p><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
                                                          <p><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
                                                          <p><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
                                                          <br>
                                                          <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                                                          03/11/19 à
                                                          14:40, Mark
                                                          Adams a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Is
                                                          there a
                                                          difference in
                                                          memory usage
                                                          on your tiny
                                                          problem? I
                                                          assume no.
                                                          <div><br>
                                                          </div>
                                                          <div>I don't
                                                          see anything
                                                          that could
                                                          come from GAMG
                                                          other than the
                                                          RAP stuff that
                                                          you have
                                                          discussed
                                                          already.</div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette
                                                          <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>The code I
                                                          am using here
                                                          is the example
                                                          42 of PETSc (<a class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
                                                          Indeed it
                                                          solves the
                                                          Stokes
                                                          equation. I
                                                          thought it was
                                                          a good idea to
                                                          use an example
                                                          you might know
                                                          (and didn't
                                                          find any that
                                                          uses GAMG
                                                          functions). I
                                                          just changed
                                                          the PCMG setup
                                                          so that the
                                                          memory problem
                                                          appears. And
                                                          it appears
                                                          when adding
                                                          PCGAMG.</p>
                                                          <p>I don't
                                                          care about the
                                                          performance or
                                                          even the
                                                          result
                                                          rightness
                                                          here, but only
                                                          about the
                                                          difference in
                                                          memory use
                                                          between 3.6
                                                          and 3.10. Do
                                                          you think
                                                          finding a more
                                                          adapted script
                                                          would help?<br>
                                                          </p>
                                                          <p>I used the
                                                          threshold of
                                                          0.1 only once,
                                                          at the
                                                          beginning, to
                                                          test its
                                                          influence. I
                                                          used the
                                                          default
                                                          threshold (of
                                                          0, I guess)
                                                          for all the
                                                          other runs.</p>
                                                          <p>Myriam<br>
                                                          </p>
                                                          <br>
                                                          <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
                                                          03/11/19 à
                                                          13:52, Mark
                                                          Adams a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">In
                                                          looking at
                                                          this larger
                                                          scale run ...
                                                          <div><br>
                                                          </div>
                                                          <div>* Your
                                                          eigen
                                                          estimates are
                                                          much lower
                                                          than your tiny
                                                          test problem. 
                                                          But this is
                                                          Stokes
                                                          apparently and
                                                          it should not
                                                          work anyway.
                                                          Maybe you have
                                                          a small time
                                                          step that adds
                                                          a lot of mass
                                                          that brings
                                                          the eigen
                                                          estimates
                                                          down. And your
                                                          min eigenvalue
                                                          (not used) is
                                                          positive. I
                                                          would expect
                                                          negative for
                                                          Stokes ...</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* You
                                                          seem to be
                                                          setting a
                                                          threshold
                                                          value of 0.1
                                                          -- that is
                                                          very high</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* v3.6
                                                          says "using
                                                          nonzero
                                                          initial guess"
                                                          but this is
                                                          not in v3.10.
                                                          Maybe we just
                                                          stopped
                                                          printing that.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* There
                                                          were some
                                                          changes to
                                                          coasening
                                                          parameters in
                                                          going from
                                                          v3.6 but it
                                                          does not look
                                                          like your
                                                          problem was
                                                          effected. (The
                                                          coarsening
                                                          algo is
                                                          non-deterministic
                                                          by default and
                                                          you can see
                                                          small
                                                          difference on
                                                          different
                                                          runs)</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* We may
                                                          have also
                                                          added a
                                                          "noisy" RHS
                                                          for eigen
                                                          estimates by
                                                          default from
                                                          v3.6.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* And for
                                                          non-symetric
                                                          problems you
                                                          can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built for Stokes anyway.</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
                                                          <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>I used
                                                          PCView to
                                                          display the
                                                          size of the
                                                          linear system
                                                          in each level
                                                          of the MG.
                                                          You'll find
                                                          the outputs
                                                          attached to
                                                          this mail (zip
                                                          file) for both
                                                          the default
                                                          threshold
                                                          value and a
                                                          value of 0.1,
                                                          and for both
                                                          3.6 and 3.10
                                                          PETSc
                                                          versions. <br>
                                                          </p>
                                                          <p>For
                                                          convenience, I
                                                          summarized the
                                                          information in
                                                          a graph, also
                                                          attached (png
                                                          file).</p>
                                                          <p>As you can
                                                          see, there are
                                                          slight
                                                          differences
                                                          between the
                                                          two versions
                                                          but none is
                                                          critical, in
                                                          my opinion. Do
                                                          you see
                                                          anything
                                                          suspicious in
                                                          the outputs?</p>
                                                          <p>+ I can't
                                                          find the
                                                          default
                                                          threshold
                                                          value. Do you
                                                          know where I
                                                          can find it?<br>
                                                          </p>
                                                          <p>Thanks for
                                                          the follow-up</p>
                                                          <p>Myriam<br>
                                                          </p>
                                                          <br>
                                                          <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
                                                          03/05/19 à
                                                          14:06, Matthew
                                                          Knepley a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Tue, Mar 5,
                                                          2019 at 7:14
                                                          AM Myriam
                                                          Peyrounette
                                                          <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>Hi Matt,</p>
                                                          <p>I plotted
                                                          the memory
                                                          scalings using
                                                          different
                                                          threshold
                                                          values. The
                                                          two scalings
                                                          are slightly
                                                          translated
                                                          (from -22 to
                                                          -88 mB) but
                                                          this gain is
                                                          neglectable.
                                                          The
                                                          3.6-scaling
                                                          keeps being
                                                          robust while
                                                          the
                                                          3.10-scaling
                                                          deteriorates.</p>
                                                          <p>Do you have
                                                          any other
                                                          suggestion?</p>
                                                          </div>
                                                          </blockquote>
                                                          <div>Mark,
                                                          what is the
                                                          option she can
                                                          give to output
                                                          all the GAMG
                                                          data?</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Also, run
                                                          using
                                                          -ksp_view.
                                                          GAMG will
                                                          report all the
                                                          sizes of its
                                                          grids, so it
                                                          should be easy
                                                          to see</div>
                                                          <div>if the
                                                          coarse grid
                                                          sizes are
                                                          increasing,
                                                          and also what
                                                          the effect of
                                                          the threshold
                                                          value is.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>   
                                                           Matt <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>Thanks<br>
                                                          </p>
                                                          Myriam <br>
                                                          <br>
                                                          <div class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
                                                          03/02/19 à
                                                          02:27, Matthew
                                                          Knepley a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Fri, Mar 1,
                                                          2019 at 10:53
                                                          AM Myriam
                                                          Peyrounette
                                                          via
                                                          petsc-users
                                                          <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                          I used to run
                                                          my code with
                                                          PETSc 3.6.
                                                          Since I
                                                          upgraded the
                                                          PETSc version<br>
                                                          to 3.10, this
                                                          code has a bad
                                                          memory
                                                          scaling.<br>
                                                          <br>
                                                          To report this
                                                          issue, I took
                                                          the PETSc
                                                          script ex42.c
                                                          and slightly<br>
                                                          modified it so
                                                          that the KSP
                                                          and PC
                                                          configurations
                                                          are the same
                                                          as in my<br>
                                                          code. In
                                                          particular, I
                                                          use a
                                                          "personnalised"
                                                          multi-grid
                                                          method. The<br>
                                                          modifications
                                                          are indicated
                                                          by the keyword
                                                          "TopBridge" in
                                                          the attached<br>
                                                          scripts.<br>
                                                          <br>
                                                          To plot the
                                                          memory (weak)
                                                          scaling, I ran
                                                          four
                                                          calculations
                                                          for each<br>
                                                          script with
                                                          increasing
                                                          problem sizes
                                                          and
                                                          computations
                                                          cores:<br>
                                                          <br>
                                                          1. 100,000
                                                          elts on 4
                                                          cores<br>
                                                          2. 1 million
                                                          elts on 40
                                                          cores<br>
                                                          3. 10 millions
                                                          elts on 400
                                                          cores<br>
                                                          4. 100
                                                          millions elts
                                                          on 4,000 cores<br>
                                                          <br>
                                                          The resulting
                                                          graph is also
                                                          attached. The
                                                          scaling using
                                                          PETSc 3.10<br>
                                                          clearly
                                                          deteriorates
                                                          for large
                                                          cases, while
                                                          the one using
                                                          PETSc 3.6 is<br>
                                                          robust.<br>
                                                          <br>
                                                          After a few
                                                          tests, I found
                                                          that the
                                                          scaling is
                                                          mostly
                                                          sensitive to
                                                          the<br>
                                                          use of the AMG
                                                          method for the
                                                          coarse grid
                                                          (line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
                                                          deteriorates
                                                          when
                                                          commenting
                                                          lines 1777 to
                                                          1790 (in
                                                          main_ex42_petsc36.cc).<br>
                                                          <br>
                                                          Do you have
                                                          any idea of
                                                          what changed
                                                          between
                                                          version 3.6
                                                          and version<br>
                                                          3.10 that may
                                                          imply such
                                                          degradation?<br>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>I believe
                                                          the default
                                                          values for
                                                          PCGAMG changed
                                                          between
                                                          versions. It
                                                          sounds like
                                                          the coarsening
                                                          rate</div>
                                                          <div>is not
                                                          great enough,
                                                          so that these
                                                          grids are too
                                                          large. This
                                                          can be set
                                                          using:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>There is
                                                          some
                                                          explanation of
                                                          this effect on
                                                          that page. Let
                                                          us know if
                                                          setting this
                                                          does not
                                                          correct the
                                                          situation.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     Matt</div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          Let me know if
                                                          you need
                                                          further
                                                          information.<br>
                                                          <br>
                                                          Best,<br>
                                                          <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          CNRS/IDRIS -
                                                          HLST<br>
                                                          --<br>
                                                          <br>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr" class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr" class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                        </div>
                                                      </blockquote>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </blockquote>
                                      <br>
                                      <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                    </div>
                                  </blockquote>
                                </div>
                                <br clear="all">
                                <div><br>
                                </div>
                                -- <br>
                                <div dir="ltr" class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail_signature">
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>What most experimenters
                                              take for granted before
                                              they begin their
                                              experiments is infinitely
                                              more interesting than any
                                              results to which their
                                              experiments lead.<br>
                                              -- Norbert Wiener</div>
                                            <div><br>
                                            </div>
                                            <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                            <br>
                            <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                </blockquote>
                <br>
                <pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
              </div>
            </blockquote>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="gmail-m_9164101223360987041moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>