<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>There is a small difference in memory usage already (of 135mB).
      It is not a big deal but it will be for larger problems (as shown
      by the memory scaling). If we find the origin of this small gap
      for a small case, we probably find the reason why the memory
      scaling is so bad with 3.10.</p>
    <p>I am currently looking for the exact commit where the problem
      arises, using git bisect. I'll let you know about the result.<br>
    </p>
    <br>
    <div class="moz-cite-prefix">Le 03/11/19 à 14:40, Mark Adams a
      écrit :<br>
    </div>
    <blockquote type="cite"
cite="mid:CADOhEh5MHFQOoHF24hOWhoAiLnqYuZO29hwKzHmMndicF1xj0g@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      <div dir="ltr">Is there a difference in memory usage on your tiny
        problem? I assume no.
        <div><br>
        </div>
        <div>I don't see anything that could come from GAMG other than
          the RAP stuff that you have discussed already.</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32
          AM Myriam Peyrounette <<a
            href="mailto:myriam.peyrounette@idris.fr"
            moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div bgcolor="#FFFFFF">
            <p>The code I am using here is the example 42 of PETSc
              (<a
                class="gmail-m_4941328961016005032moz-txt-link-freetext"
href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html"
                target="_blank" moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
              Indeed it solves the Stokes equation. I thought it was a
              good idea to use an example you might know (and didn't
              find any that uses GAMG functions). I just changed the
              PCMG setup so that the memory problem appears. And it
              appears when adding PCGAMG.</p>
            <p>I don't care about the performance or even the result
              rightness here, but only about the difference in memory
              use between 3.6 and 3.10. Do you think finding a more
              adapted script would help?<br>
            </p>
            <p>I used the threshold of 0.1 only once, at the beginning,
              to test its influence. I used the default threshold (of 0,
              I guess) for all the other runs.</p>
            <p>Myriam<br>
            </p>
            <br>
            <div class="gmail-m_4941328961016005032moz-cite-prefix">Le
              03/11/19 à 13:52, Mark Adams a écrit :<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">
                <div dir="ltr">In looking at this larger scale run ...
                  <div><br>
                  </div>
                  <div>* Your eigen estimates are much lower than your
                    tiny test problem.  But this is Stokes apparently
                    and it should not work anyway. Maybe you have a
                    small time step that adds a lot of mass that brings
                    the eigen estimates down. And your min eigenvalue
                    (not used) is positive. I would expect negative for
                    Stokes ...</div>
                  <div><br>
                  </div>
                  <div>* You seem to be setting a threshold value of 0.1
                    -- that is very high</div>
                  <div><br>
                  </div>
                  <div>* v3.6 says "using nonzero initial guess" but
                    this is not in v3.10. Maybe we just stopped printing
                    that.</div>
                  <div><br>
                  </div>
                  <div>* There were some changes to coasening parameters
                    in going from v3.6 but it does not look like your
                    problem was effected. (The coarsening algo is
                    non-deterministic by default and you can see small
                    difference on different runs)</div>
                  <div><br>
                  </div>
                  <div>* We may have also added a "noisy" RHS for eigen
                    estimates by default from v3.6.</div>
                  <div><br>
                  </div>
                  <div>* And for non-symetric problems you can try
                    -pc_gamg_agg_nsmooths 0, but again GAMG is not built
                    for Stokes anyway.</div>
                  <div><br>
                  </div>
                </div>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at
                  11:53 AM Myriam Peyrounette <<a
                    href="mailto:myriam.peyrounette@idris.fr"
                    target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px 0.8ex;border-left:1px solid
                  rgb(204,204,204);padding-left:1ex">
                  <div bgcolor="#FFFFFF">
                    <p>I used PCView to display the size of the linear
                      system in each level of the MG. You'll find the
                      outputs attached to this mail (zip file) for both
                      the default threshold value and a value of 0.1,
                      and for both 3.6 and 3.10 PETSc versions. <br>
                    </p>
                    <p>For convenience, I summarized the information in
                      a graph, also attached (png file).</p>
                    <p>As you can see, there are slight differences
                      between the two versions but none is critical, in
                      my opinion. Do you see anything suspicious in the
                      outputs?</p>
                    <p>+ I can't find the default threshold value. Do
                      you know where I can find it?<br>
                    </p>
                    <p>Thanks for the follow-up</p>
                    <p>Myriam<br>
                    </p>
                    <br>
                    <div
class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
                      03/05/19 à 14:06, Matthew Knepley a écrit :<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div dir="ltr">On Tue, Mar 5, 2019 at 7:14 AM
                          Myriam Peyrounette <<a
                            href="mailto:myriam.peyrounette@idris.fr"
                            target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
                          wrote:<br>
                        </div>
                        <div class="gmail_quote">
                          <blockquote class="gmail_quote"
                            style="margin:0px 0px 0px
                            0.8ex;border-left:1px solid
                            rgb(204,204,204);padding-left:1ex">
                            <div bgcolor="#FFFFFF">
                              <p>Hi Matt,</p>
                              <p>I plotted the memory scalings using
                                different threshold values. The two
                                scalings are slightly translated (from
                                -22 to -88 mB) but this gain is
                                neglectable. The 3.6-scaling keeps being
                                robust while the 3.10-scaling
                                deteriorates.</p>
                              <p>Do you have any other suggestion?</p>
                            </div>
                          </blockquote>
                          <div>Mark, what is the option she can give to
                            output all the GAMG data?</div>
                          <div><br>
                          </div>
                          <div>Also, run using -ksp_view. GAMG will
                            report all the sizes of its grids, so it
                            should be easy to see</div>
                          <div>if the coarse grid sizes are increasing,
                            and also what the effect of the threshold
                            value is.</div>
                          <div><br>
                          </div>
                          <div>  Thanks,</div>
                          <div><br>
                          </div>
                          <div>     Matt <br>
                          </div>
                          <blockquote class="gmail_quote"
                            style="margin:0px 0px 0px
                            0.8ex;border-left:1px solid
                            rgb(204,204,204);padding-left:1ex">
                            <div bgcolor="#FFFFFF">
                              <p>Thanks<br>
                              </p>
                              Myriam <br>
                              <br>
                              <div
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
                                03/02/19 à 02:27, Matthew Knepley a
                                écrit :<br>
                              </div>
                              <blockquote type="cite">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">On Fri, Mar 1, 2019
                                      at 10:53 AM Myriam Peyrounette via
                                      petsc-users <<a
                                        href="mailto:petsc-users@mcs.anl.gov"
                                        target="_blank"
                                        moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
                                      wrote:<br>
                                    </div>
                                    <div class="gmail_quote">
                                      <blockquote class="gmail_quote"
                                        style="margin:0px 0px 0px
                                        0.8ex;border-left:1px solid
                                        rgb(204,204,204);padding-left:1ex">Hi,<br>
                                        <br>
                                        I used to run my code with PETSc
                                        3.6. Since I upgraded the PETSc
                                        version<br>
                                        to 3.10, this code has a bad
                                        memory scaling.<br>
                                        <br>
                                        To report this issue, I took the
                                        PETSc script ex42.c and slightly<br>
                                        modified it so that the KSP and
                                        PC configurations are the same
                                        as in my<br>
                                        code. In particular, I use a
                                        "personnalised" multi-grid
                                        method. The<br>
                                        modifications are indicated by
                                        the keyword "TopBridge" in the
                                        attached<br>
                                        scripts.<br>
                                        <br>
                                        To plot the memory (weak)
                                        scaling, I ran four calculations
                                        for each<br>
                                        script with increasing problem
                                        sizes and computations cores:<br>
                                        <br>
                                        1. 100,000 elts on 4 cores<br>
                                        2. 1 million elts on 40 cores<br>
                                        3. 10 millions elts on 400 cores<br>
                                        4. 100 millions elts on 4,000
                                        cores<br>
                                        <br>
                                        The resulting graph is also
                                        attached. The scaling using
                                        PETSc 3.10<br>
                                        clearly deteriorates for large
                                        cases, while the one using PETSc
                                        3.6 is<br>
                                        robust.<br>
                                        <br>
                                        After a few tests, I found that
                                        the scaling is mostly sensitive
                                        to the<br>
                                        use of the AMG method for the
                                        coarse grid (line 1780 in<br>
                                        main_ex42_petsc36.cc). In
                                        particular, the performance
                                        strongly<br>
                                        deteriorates when commenting
                                        lines 1777 to 1790 (in
                                        main_ex42_petsc36.cc).<br>
                                        <br>
                                        Do you have any idea of what
                                        changed between version 3.6 and
                                        version<br>
                                        3.10 that may imply such
                                        degradation?<br>
                                      </blockquote>
                                      <div><br>
                                      </div>
                                      <div>I believe the default values
                                        for PCGAMG changed between
                                        versions. It sounds like the
                                        coarsening rate</div>
                                      <div>is not great enough, so that
                                        these grids are too large. This
                                        can be set using:</div>
                                      <div><br>
                                      </div>
                                      <div>  <a
href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html"
                                          target="_blank"
                                          moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
                                      <div><br>
                                      </div>
                                      <div>There is some explanation of
                                        this effect on that page. Let us
                                        know if setting this does not
                                        correct the situation.</div>
                                      <div><br>
                                      </div>
                                      <div>  Thanks,</div>
                                      <div><br>
                                      </div>
                                      <div>     Matt</div>
                                      <div> </div>
                                      <blockquote class="gmail_quote"
                                        style="margin:0px 0px 0px
                                        0.8ex;border-left:1px solid
                                        rgb(204,204,204);padding-left:1ex">
                                        Let me know if you need further
                                        information.<br>
                                        <br>
                                        Best,<br>
                                        <br>
                                        Myriam Peyrounette<br>
                                        <br>
                                        <br>
                                        -- <br>
                                        Myriam Peyrounette<br>
                                        CNRS/IDRIS - HLST<br>
                                        --<br>
                                        <br>
                                      </blockquote>
                                    </div>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    <div dir="ltr"
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>
                                              <div dir="ltr">
                                                <div>What most
                                                  experimenters take for
                                                  granted before they
                                                  begin their
                                                  experiments is
                                                  infinitely more
                                                  interesting than any
                                                  results to which their
                                                  experiments lead.<br>
                                                  -- Norbert Wiener</div>
                                                <div><br>
                                                </div>
                                                <div><a
                                                    href="http://www.cse.buffalo.edu/%7Eknepley/"
                                                    target="_blank"
                                                    moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </blockquote>
                              <br>
                              <pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                            </div>
                          </blockquote>
                        </div>
                        <br clear="all">
                        <div><br>
                        </div>
                        -- <br>
                        <div dir="ltr"
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
                          <div dir="ltr">
                            <div>
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>What most experimenters take
                                      for granted before they begin
                                      their experiments is infinitely
                                      more interesting than any results
                                      to which their experiments lead.<br>
                                      -- Norbert Wiener</div>
                                    <div><br>
                                    </div>
                                    <div><a
                                        href="http://www.cse.buffalo.edu/%7Eknepley/"
                                        target="_blank"
                                        moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                  </div>
                </blockquote>
              </div>
            </blockquote>
            <br>
            <pre class="gmail-m_4941328961016005032moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
  </body>
</html>