<div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at 10:36, Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr">myriam.peyrounette@idris.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Oh you were right, the three options are unsused (-matptap_via
      scalable, -inner_offdiag_matmatmult_via scalable and
      -inner_diag_matmatmult_via scalable). Does this mean I am not
      using the associated PtAP functions?</p></div></blockquote><div><br></div><div>No - not necessarily. All it means is the options were not parsed. </div><div><br></div><div>If your matrices have an option prefix associated with them (e.g. abc) , then you need to provide the option as</div><div>  -abc_matptap_via scalable</div><div><br></div><div>If you are not sure if you matrices have a prefix, look at the result of -ksp_view (see below for an example)</div><div><br></div><div>







<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">  </span>Mat Object: 2 MPI processes</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>type: mpiaij</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>rows=363, cols=363, bs=3</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>total: nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>total number of mallocs used during MatSetValues calls =0</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">  </span>Mat Object: (B_) 2 MPI processes</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>type: mpiaij</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>rows=363, cols=363, bs=3</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>total: nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">    </span>total number of mallocs used during MatSetValues calls =0</span></p></div><div><br></div><div>The first matrix has no options prefix, but the second does and it's called "B_".</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <p>Myriam<br>
    </p>
    <br>
    <div class="gmail-m_-8509232780790875615moz-cite-prefix">Le 03/26/19 à 11:10, Dave May a écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr"><br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at
              09:52, Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p>How can I be sure they are indeed used? Can I print
                  this information in some log file? <br>
                </p>
              </div>
            </blockquote>
            <div>Yes. Re-run the job with the command line option</div>
            <div><br>
            </div>
            <div>-options_left true<br>
            </div>
            <div><br>
            </div>
            <div>This will report all options parsed, and importantly,
              will also indicate if any options were unused.</div>
            <div> </div>
            <div><br>
            </div>
            <div>Thanks</div>
            <div>Dave</div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p> </p>
                <p>Thanks in advance<br>
                </p>
                <p>Myriam<br>
                </p>
                <br>
                <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-cite-prefix">Le
                  03/25/19 à 18:24, Matthew Knepley a écrit :<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div dir="ltr">On Mon, Mar 25, 2019 at 10:54 AM
                      Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                      wrote:<br>
                    </div>
                    <div class="gmail_quote">
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                        <div bgcolor="#FFFFFF">
                          <p>Hi,</p>
                          <p>thanks for the explanations. I tried the
                            last PETSc version (commit
                            fbc5705bc518d02a4999f188aad4ccff5f754cbf),
                            which includes the patch you talked about.
                            But the memory scaling shows no improvement
                            (see scaling attached), even when using the
                            "scalable" options :(</p>
                          <p>I had a look at the PETSc functions
                            MatPtAPNumeric_MPIAIJ_MPIAIJ and
                            MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at
                            the differences before and after the first
                            "bad" commit), but I can't find what induced
                            this memory issue.</p>
                        </div>
                      </blockquote>
                      <div>Are you sure that the option was used? It
                        just looks suspicious to me that they use
                        exactly the same amount of memory. It should be
                        different, even if it does not solve the
                        problem.</div>
                      <div><br>
                      </div>
                      <div>   Thanks,</div>
                      <div><br>
                      </div>
                      <div>     Matt </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                        <div bgcolor="#FFFFFF">
                          <p>Myriam<br>
                          </p>
                          <p><br>
                          </p>
                          <p><br>
                          </p>
                          <br>
                          <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-cite-prefix">Le
                            03/20/19 à 17:38, Fande Kong a écrit :<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div>Hi Myriam,</div>
                                        <div><br>
                                        </div>
                                        <div>There are three algorithms
                                          in PETSc to do PtAP ( const
                                          char          *algTypes[3] =
                                          {"scalable","nonscalable","hypre"};),
                                          and can be specified using the
                                          petsc options: -matptap_via
                                          xxxx.</div>
                                        <div><br>
                                        </div>
                                        <div>(1) -matptap_via hypre:
                                          This call the hypre package to
                                          do the PtAP trough an
                                          all-at-once triple product. In
                                          our experiences, it is the
                                          most memory efficient, but
                                          could be slow.</div>
                                        <div><br>
                                        </div>
                                        <div>(2)  -matptap_via scalable:
                                          This involves a row-wise
                                          algorithm plus an outer
                                          product.  This will use more
                                          memory than hypre, but way
                                          faster. This used to have a
                                          bug that could take all your
                                          memory, and I have a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>. 
                                          When using this option, we may
                                          want to have extra options
                                          such as 
                                           -inner_offdiag_matmatmult_via
                                          scalable
                                          -inner_diag_matmatmult_via
                                          scalable  to select inner
                                          scalable algorithms.</div>
                                        <div><br>
                                        </div>
                                        <div>(3)  -matptap_via
                                          nonscalable:  Suppose to be
                                          even faster, but use more
                                          memory. It does dense matrix
                                          operations.</div>
                                        <div><br>
                                        </div>
                                        <div><br>
                                        </div>
                                        <div>Thanks,</div>
                                        <div><br>
                                        </div>
                                        <div>Fande Kong</div>
                                        <div><br>
                                        </div>
                                        <div><br>
                                        </div>
                                        <div><br>
                                        </div>
                                        <br>
                                        <div class="gmail_quote">
                                          <div dir="ltr" class="gmail_attr">On Wed,
                                            Mar 20, 2019 at 10:06 AM
                                            Myriam Peyrounette via
                                            petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                            wrote:<br>
                                          </div>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                            <div bgcolor="#FFFFFF">
                                              <p>More precisely:
                                                something happens when
                                                upgrading the functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
                                              </p>
                                              <p>Unfortunately, there
                                                are a lot of differences
                                                between the old and new
                                                versions of these
                                                functions. I keep
                                                investigating but if you
                                                have any idea, please
                                                let me know.</p>
                                              <p>Best,<br>
                                              </p>
                                              <p>Myriam<br>
                                              </p>
                                              <br>
                                              <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                                                03/20/19 à 13:48, Myriam
                                                Peyrounette a écrit :<br>
                                              </div>
                                              <blockquote type="cite">
                                                <p>Hi all,</p>
                                                <p>I used git bisect to
                                                  determine when the
                                                  memory need increased.
                                                  I found that the first
                                                  "bad" commit is  
                                                  aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
                                                <p>Barry was right, this
                                                  commit seems to be
                                                  about an evolution of
                                                  <span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
                                                      You mentioned the
                                                      option
                                                      "-matptap_via
                                                      scalable" but I
                                                      can't find any
                                                      information about
                                                      it. Can you tell
                                                      me more?</span></span></p>
                                                <p><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
                                                <p><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
                                                <p><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
                                                <br>
                                                <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                                                  03/11/19 à 14:40, Mark
                                                  Adams a écrit :<br>
                                                </div>
                                                <blockquote type="cite">
                                                  <div dir="ltr">Is
                                                    there a difference
                                                    in memory usage on
                                                    your tiny problem? I
                                                    assume no.
                                                    <div><br>
                                                    </div>
                                                    <div>I don't see
                                                      anything that
                                                      could come from
                                                      GAMG other than
                                                      the RAP stuff that
                                                      you have discussed
                                                      already.</div>
                                                  </div>
                                                  <br>
                                                  <div class="gmail_quote">
                                                    <div dir="ltr" class="gmail_attr">On
                                                      Mon, Mar 11, 2019
                                                      at 9:32 AM Myriam
                                                      Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                      wrote:<br>
                                                    </div>
                                                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                      <div bgcolor="#FFFFFF">
                                                        <p>The code I am
                                                          using here is
                                                          the example 42
                                                          of PETSc (<a class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
                                                          Indeed it
                                                          solves the
                                                          Stokes
                                                          equation. I
                                                          thought it was
                                                          a good idea to
                                                          use an example
                                                          you might know
                                                          (and didn't
                                                          find any that
                                                          uses GAMG
                                                          functions). I
                                                          just changed
                                                          the PCMG setup
                                                          so that the
                                                          memory problem
                                                          appears. And
                                                          it appears
                                                          when adding
                                                          PCGAMG.</p>
                                                        <p>I don't care
                                                          about the
                                                          performance or
                                                          even the
                                                          result
                                                          rightness
                                                          here, but only
                                                          about the
                                                          difference in
                                                          memory use
                                                          between 3.6
                                                          and 3.10. Do
                                                          you think
                                                          finding a more
                                                          adapted script
                                                          would help?<br>
                                                        </p>
                                                        <p>I used the
                                                          threshold of
                                                          0.1 only once,
                                                          at the
                                                          beginning, to
                                                          test its
                                                          influence. I
                                                          used the
                                                          default
                                                          threshold (of
                                                          0, I guess)
                                                          for all the
                                                          other runs.</p>
                                                        <p>Myriam<br>
                                                        </p>
                                                        <br>
                                                        <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
                                                          03/11/19 à
                                                          13:52, Mark
                                                          Adams a
                                                          écrit :<br>
                                                        </div>
                                                        <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">In
                                                          looking at
                                                          this larger
                                                          scale run ...
                                                          <div><br>
                                                          </div>
                                                          <div>* Your
                                                          eigen
                                                          estimates are
                                                          much lower
                                                          than your tiny
                                                          test problem. 
                                                          But this is
                                                          Stokes
                                                          apparently and
                                                          it should not
                                                          work anyway.
                                                          Maybe you have
                                                          a small time
                                                          step that adds
                                                          a lot of mass
                                                          that brings
                                                          the eigen
                                                          estimates
                                                          down. And your
                                                          min eigenvalue
                                                          (not used) is
                                                          positive. I
                                                          would expect
                                                          negative for
                                                          Stokes ...</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* You
                                                          seem to be
                                                          setting a
                                                          threshold
                                                          value of 0.1
                                                          -- that is
                                                          very high</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* v3.6
                                                          says "using
                                                          nonzero
                                                          initial guess"
                                                          but this is
                                                          not in v3.10.
                                                          Maybe we just
                                                          stopped
                                                          printing that.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* There
                                                          were some
                                                          changes to
                                                          coasening
                                                          parameters in
                                                          going from
                                                          v3.6 but it
                                                          does not look
                                                          like your
                                                          problem was
                                                          effected. (The
                                                          coarsening
                                                          algo is
                                                          non-deterministic
                                                          by default and
                                                          you can see
                                                          small
                                                          difference on
                                                          different
                                                          runs)</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* We may
                                                          have also
                                                          added a
                                                          "noisy" RHS
                                                          for eigen
                                                          estimates by
                                                          default from
                                                          v3.6.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>* And for
                                                          non-symetric
                                                          problems you
                                                          can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built for Stokes anyway.</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
                                                          <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>I used
                                                          PCView to
                                                          display the
                                                          size of the
                                                          linear system
                                                          in each level
                                                          of the MG.
                                                          You'll find
                                                          the outputs
                                                          attached to
                                                          this mail (zip
                                                          file) for both
                                                          the default
                                                          threshold
                                                          value and a
                                                          value of 0.1,
                                                          and for both
                                                          3.6 and 3.10
                                                          PETSc
                                                          versions. <br>
                                                          </p>
                                                          <p>For
                                                          convenience, I
                                                          summarized the
                                                          information in
                                                          a graph, also
                                                          attached (png
                                                          file).</p>
                                                          <p>As you can
                                                          see, there are
                                                          slight
                                                          differences
                                                          between the
                                                          two versions
                                                          but none is
                                                          critical, in
                                                          my opinion. Do
                                                          you see
                                                          anything
                                                          suspicious in
                                                          the outputs?</p>
                                                          <p>+ I can't
                                                          find the
                                                          default
                                                          threshold
                                                          value. Do you
                                                          know where I
                                                          can find it?<br>
                                                          </p>
                                                          <p>Thanks for
                                                          the follow-up</p>
                                                          <p>Myriam<br>
                                                          </p>
                                                          <br>
                                                          <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
                                                          03/05/19 à
                                                          14:06, Matthew
                                                          Knepley a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Tue, Mar 5,
                                                          2019 at 7:14
                                                          AM Myriam
                                                          Peyrounette
                                                          <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>Hi Matt,</p>
                                                          <p>I plotted
                                                          the memory
                                                          scalings using
                                                          different
                                                          threshold
                                                          values. The
                                                          two scalings
                                                          are slightly
                                                          translated
                                                          (from -22 to
                                                          -88 mB) but
                                                          this gain is
                                                          neglectable.
                                                          The
                                                          3.6-scaling
                                                          keeps being
                                                          robust while
                                                          the
                                                          3.10-scaling
                                                          deteriorates.</p>
                                                          <p>Do you have
                                                          any other
                                                          suggestion?</p>
                                                          </div>
                                                          </blockquote>
                                                          <div>Mark,
                                                          what is the
                                                          option she can
                                                          give to output
                                                          all the GAMG
                                                          data?</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Also, run
                                                          using
                                                          -ksp_view.
                                                          GAMG will
                                                          report all the
                                                          sizes of its
                                                          grids, so it
                                                          should be easy
                                                          to see</div>
                                                          <div>if the
                                                          coarse grid
                                                          sizes are
                                                          increasing,
                                                          and also what
                                                          the effect of
                                                          the threshold
                                                          value is.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>   
                                                           Matt <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <p>Thanks<br>
                                                          </p>
                                                          Myriam <br>
                                                          <br>
                                                          <div class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
                                                          03/02/19 à
                                                          02:27, Matthew
                                                          Knepley a
                                                          écrit :<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Fri, Mar 1,
                                                          2019 at 10:53
                                                          AM Myriam
                                                          Peyrounette
                                                          via
                                                          petsc-users
                                                          <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                          I used to run
                                                          my code with
                                                          PETSc 3.6.
                                                          Since I
                                                          upgraded the
                                                          PETSc version<br>
                                                          to 3.10, this
                                                          code has a bad
                                                          memory
                                                          scaling.<br>
                                                          <br>
                                                          To report this
                                                          issue, I took
                                                          the PETSc
                                                          script ex42.c
                                                          and slightly<br>
                                                          modified it so
                                                          that the KSP
                                                          and PC
                                                          configurations
                                                          are the same
                                                          as in my<br>
                                                          code. In
                                                          particular, I
                                                          use a
                                                          "personnalised"
                                                          multi-grid
                                                          method. The<br>
                                                          modifications
                                                          are indicated
                                                          by the keyword
                                                          "TopBridge" in
                                                          the attached<br>
                                                          scripts.<br>
                                                          <br>
                                                          To plot the
                                                          memory (weak)
                                                          scaling, I ran
                                                          four
                                                          calculations
                                                          for each<br>
                                                          script with
                                                          increasing
                                                          problem sizes
                                                          and
                                                          computations
                                                          cores:<br>
                                                          <br>
                                                          1. 100,000
                                                          elts on 4
                                                          cores<br>
                                                          2. 1 million
                                                          elts on 40
                                                          cores<br>
                                                          3. 10 millions
                                                          elts on 400
                                                          cores<br>
                                                          4. 100
                                                          millions elts
                                                          on 4,000 cores<br>
                                                          <br>
                                                          The resulting
                                                          graph is also
                                                          attached. The
                                                          scaling using
                                                          PETSc 3.10<br>
                                                          clearly
                                                          deteriorates
                                                          for large
                                                          cases, while
                                                          the one using
                                                          PETSc 3.6 is<br>
                                                          robust.<br>
                                                          <br>
                                                          After a few
                                                          tests, I found
                                                          that the
                                                          scaling is
                                                          mostly
                                                          sensitive to
                                                          the<br>
                                                          use of the AMG
                                                          method for the
                                                          coarse grid
                                                          (line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
                                                          deteriorates
                                                          when
                                                          commenting
                                                          lines 1777 to
                                                          1790 (in
                                                          main_ex42_petsc36.cc).<br>
                                                          <br>
                                                          Do you have
                                                          any idea of
                                                          what changed
                                                          between
                                                          version 3.6
                                                          and version<br>
                                                          3.10 that may
                                                          imply such
                                                          degradation?<br>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>I believe
                                                          the default
                                                          values for
                                                          PCGAMG changed
                                                          between
                                                          versions. It
                                                          sounds like
                                                          the coarsening
                                                          rate</div>
                                                          <div>is not
                                                          great enough,
                                                          so that these
                                                          grids are too
                                                          large. This
                                                          can be set
                                                          using:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>There is
                                                          some
                                                          explanation of
                                                          this effect on
                                                          that page. Let
                                                          us know if
                                                          setting this
                                                          does not
                                                          correct the
                                                          situation.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     Matt</div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          Let me know if
                                                          you need
                                                          further
                                                          information.<br>
                                                          <br>
                                                          Best,<br>
                                                          <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          CNRS/IDRIS -
                                                          HLST<br>
                                                          --<br>
                                                          <br>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr" class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr" class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                        </blockquote>
                                                        <br>
                                                        <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                </blockquote>
                                                <br>
                                                <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                              </blockquote>
                                              <br>
                                              <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                            </div>
                                          </blockquote>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                        </div>
                      </blockquote>
                    </div>
                    <br clear="all">
                    <div><br>
                    </div>
                    -- <br>
                    <div dir="ltr" class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail_signature">
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div>
                              <div dir="ltr">
                                <div>What most experimenters take for
                                  granted before they begin their
                                  experiments is infinitely more
                                  interesting than any results to which
                                  their experiments lead.<br>
                                  -- Norbert Wiener</div>
                                <div><br>
                                </div>
                                <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </blockquote>
                <br>
                <pre class="gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
              </div>
            </blockquote>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="gmail-m_-8509232780790875615moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
  </div>

</blockquote></div></div></div>