<div dir="ltr">Is there a difference in memory usage on your tiny problem? I assume no.<div><br></div><div>I don't see anything that could come from GAMG other than the RAP stuff that you have discussed already.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr">myriam.peyrounette@idris.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>The code I am using here is the example 42 of PETSc
(<a class="gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
      Indeed it solves the Stokes equation. I thought it was a good idea
      to use an example you might know (and didn't find any that uses
      GAMG functions). I just changed the PCMG setup so that the memory
      problem appears. And it appears when adding PCGAMG.</p>
    <p>I don't care about the performance or even the result rightness
      here, but only about the difference in memory use between 3.6 and
      3.10. Do you think finding a more adapted script would help?<br>
    </p>
    <p>I used the threshold of 0.1 only once, at the beginning, to test
      its influence. I used the default threshold (of 0, I guess) for
      all the other runs.</p>
    <p>Myriam<br>
    </p>
    <br>
    <div class="gmail-m_4941328961016005032moz-cite-prefix">Le 03/11/19 à 13:52, Mark Adams a
      écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">In looking at this larger scale run ...
          <div><br>
          </div>
          <div>* Your eigen estimates are much lower than your tiny test
            problem.  But this is Stokes apparently and it should not
            work anyway. Maybe you have a small time step that adds a
            lot of mass that brings the eigen estimates down. And your
            min eigenvalue (not used) is positive. I would expect
            negative for Stokes ...</div>
          <div><br>
          </div>
          <div>* You seem to be setting a threshold value of 0.1 -- that
            is very high</div>
          <div><br>
          </div>
          <div>* v3.6 says "using nonzero initial guess" but this is not
            in v3.10. Maybe we just stopped printing that.</div>
          <div><br>
          </div>
          <div>* There were some changes to coasening parameters in
            going from v3.6 but it does not look like your problem was
            effected. (The coarsening algo is non-deterministic by
            default and you can see small difference on different runs)</div>
          <div><br>
          </div>
          <div>* We may have also added a "noisy" RHS for eigen
            estimates by default from v3.6.</div>
          <div><br>
          </div>
          <div>* And for non-symetric problems you can try
            -pc_gamg_agg_nsmooths 0, but again GAMG is not built for
            Stokes anyway.</div>
          <div><br>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 11:53
          AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div bgcolor="#FFFFFF">
            <p>I used PCView to display the size of the linear system in
              each level of the MG. You'll find the outputs attached to
              this mail (zip file) for both the default threshold value
              and a value of 0.1, and for both 3.6 and 3.10 PETSc
              versions. <br>
            </p>
            <p>For convenience, I summarized the information in a graph,
              also attached (png file).</p>
            <p>As you can see, there are slight differences between the
              two versions but none is critical, in my opinion. Do you
              see anything suspicious in the outputs?</p>
            <p>+ I can't find the default threshold value. Do you know
              where I can find it?<br>
            </p>
            <p>Thanks for the follow-up</p>
            <p>Myriam<br>
            </p>
            <br>
            <div class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
              03/05/19 à 14:06, Matthew Knepley a écrit :<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">
                <div dir="ltr">On Tue, Mar 5, 2019 at 7:14 AM Myriam
                  Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                  wrote:<br>
                </div>
                <div class="gmail_quote">
                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#FFFFFF">
                      <p>Hi Matt,</p>
                      <p>I plotted the memory scalings using different
                        threshold values. The two scalings are slightly
                        translated (from -22 to -88 mB) but this gain is
                        neglectable. The 3.6-scaling keeps being robust
                        while the 3.10-scaling deteriorates.</p>
                      <p>Do you have any other suggestion?</p>
                    </div>
                  </blockquote>
                  <div>Mark, what is the option she can give to output
                    all the GAMG data?</div>
                  <div><br>
                  </div>
                  <div>Also, run using -ksp_view. GAMG will report all
                    the sizes of its grids, so it should be easy to see</div>
                  <div>if the coarse grid sizes are increasing, and also
                    what the effect of the threshold value is.</div>
                  <div><br>
                  </div>
                  <div>  Thanks,</div>
                  <div><br>
                  </div>
                  <div>     Matt <br>
                  </div>
                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#FFFFFF">
                      <p>Thanks<br>
                      </p>
                      Myriam <br>
                      <br>
                      <div class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
                        03/02/19 à 02:27, Matthew Knepley a écrit :<br>
                      </div>
                      <blockquote type="cite">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">On Fri, Mar 1, 2019 at 10:53
                              AM Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                              wrote:<br>
                            </div>
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                <br>
                                I used to run my code with PETSc 3.6.
                                Since I upgraded the PETSc version<br>
                                to 3.10, this code has a bad memory
                                scaling.<br>
                                <br>
                                To report this issue, I took the PETSc
                                script ex42.c and slightly<br>
                                modified it so that the KSP and PC
                                configurations are the same as in my<br>
                                code. In particular, I use a
                                "personnalised" multi-grid method. The<br>
                                modifications are indicated by the
                                keyword "TopBridge" in the attached<br>
                                scripts.<br>
                                <br>
                                To plot the memory (weak) scaling, I ran
                                four calculations for each<br>
                                script with increasing problem sizes and
                                computations cores:<br>
                                <br>
                                1. 100,000 elts on 4 cores<br>
                                2. 1 million elts on 40 cores<br>
                                3. 10 millions elts on 400 cores<br>
                                4. 100 millions elts on 4,000 cores<br>
                                <br>
                                The resulting graph is also attached.
                                The scaling using PETSc 3.10<br>
                                clearly deteriorates for large cases,
                                while the one using PETSc 3.6 is<br>
                                robust.<br>
                                <br>
                                After a few tests, I found that the
                                scaling is mostly sensitive to the<br>
                                use of the AMG method for the coarse
                                grid (line 1780 in<br>
                                main_ex42_petsc36.cc). In particular,
                                the performance strongly<br>
                                deteriorates when commenting lines 1777
                                to 1790 (in main_ex42_petsc36.cc).<br>
                                <br>
                                Do you have any idea of what changed
                                between version 3.6 and version<br>
                                3.10 that may imply such degradation?<br>
                              </blockquote>
                              <div><br>
                              </div>
                              <div>I believe the default values for
                                PCGAMG changed between versions. It
                                sounds like the coarsening rate</div>
                              <div>is not great enough, so that these
                                grids are too large. This can be set
                                using:</div>
                              <div><br>
                              </div>
                              <div>  <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
                              <div><br>
                              </div>
                              <div>There is some explanation of this
                                effect on that page. Let us know if
                                setting this does not correct the
                                situation.</div>
                              <div><br>
                              </div>
                              <div>  Thanks,</div>
                              <div><br>
                              </div>
                              <div>     Matt</div>
                              <div> </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Let
                                me know if you need further information.<br>
                                <br>
                                Best,<br>
                                <br>
                                Myriam Peyrounette<br>
                                <br>
                                <br>
                                -- <br>
                                Myriam Peyrounette<br>
                                CNRS/IDRIS - HLST<br>
                                --<br>
                                <br>
                              </blockquote>
                            </div>
                            <br clear="all">
                            <div><br>
                            </div>
                            -- <br>
                            <div dir="ltr" class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr">
                                        <div>What most experimenters
                                          take for granted before they
                                          begin their experiments is
                                          infinitely more interesting
                                          than any results to which
                                          their experiments lead.<br>
                                          -- Norbert Wiener</div>
                                        <div><br>
                                        </div>
                                        <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </blockquote>
                      <br>
                      <pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                    </div>
                  </blockquote>
                </div>
                <br clear="all">
                <div><br>
                </div>
                -- <br>
                <div dir="ltr" class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
                  <div dir="ltr">
                    <div>
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div>What most experimenters take for
                              granted before they begin their
                              experiments is infinitely more interesting
                              than any results to which their
                              experiments lead.<br>
                              -- Norbert Wiener</div>
                            <div><br>
                            </div>
                            <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
            <br>
            <pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
    <pre class="gmail-m_4941328961016005032moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
  </div>

</blockquote></div>