<div dir="ltr"><div dir="ltr">On Mon, Mar 25, 2019 at 10:54 AM Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Hi,</p>
    <p>thanks for the explanations. I tried the last PETSc version
      (commit fbc5705bc518d02a4999f188aad4ccff5f754cbf), which includes
      the patch you talked about. But the memory scaling shows no
      improvement (see scaling attached), even when using the "scalable"
      options :(</p>
    <p>I had a look at the PETSc functions MatPtAPNumeric_MPIAIJ_MPIAIJ
      and MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the differences
      before and after the first "bad" commit), but I can't find what
      induced this memory issue.</p>
    <p></p></div></blockquote><div>Are you sure that the option was used? It just looks suspicious to me that they use exactly the same amount of memory. It should be different, even if it does not solve the problem.</div><div><br></div><div>   Thanks,</div><div><br></div><div>     Matt </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>Myriam<br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <br>
    <div class="gmail-m_7299408251829905453moz-cite-prefix">Le 03/20/19 à 17:38, Fande Kong a
      écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">
            <div dir="ltr">
              <div dir="ltr">
                <div dir="ltr">
                  <div>Hi Myriam,</div>
                  <div><br>
                  </div>
                  <div>There are three algorithms in PETSc to do PtAP
                    ( const char          *algTypes[3] =
                    {"scalable","nonscalable","hypre"};), and can be
                    specified using the petsc options: -matptap_via
                    xxxx.</div>
                  <div><br>
                  </div>
                  <div>(1) -matptap_via hypre: This call the hypre
                    package to do the PtAP trough an all-at-once triple
                    product. In our experiences, it is the most memory
                    efficient, but could be slow.</div>
                  <div><br>
                  </div>
                  <div>(2)  -matptap_via scalable: This involves a
                    row-wise algorithm plus an outer product.  This will
                    use more memory than hypre, but way faster. This
                    used to have a bug that could take all your memory,
                    and I have a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>. 
                    When using this option, we may want to have extra
                    options such as   -inner_offdiag_matmatmult_via
                    scalable -inner_diag_matmatmult_via scalable  to
                    select inner scalable algorithms.</div>
                  <div><br>
                  </div>
                  <div>(3)  -matptap_via nonscalable:  Suppose to be
                    even faster, but use more memory. It does dense
                    matrix operations.</div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div>Thanks,</div>
                  <div><br>
                  </div>
                  <div>Fande Kong</div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <br>
                  <div class="gmail_quote">
                    <div dir="ltr" class="gmail_attr">On Wed, Mar 20,
                      2019 at 10:06 AM Myriam Peyrounette via
                      petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                      wrote:<br>
                    </div>
                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                      <div bgcolor="#FFFFFF">
                        <p>More precisely: something happens when
                          upgrading the functions
                          MatPtAPNumeric_MPIAIJ_MPIAIJ and/or
                          MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
                        </p>
                        <p>Unfortunately, there are a lot of differences
                          between the old and new versions of these
                          functions. I keep investigating but if you
                          have any idea, please let me know.</p>
                        <p>Best,<br>
                        </p>
                        <p>Myriam<br>
                        </p>
                        <br>
                        <div class="gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                          03/20/19 à 13:48, Myriam Peyrounette a écrit :<br>
                        </div>
                        <blockquote type="cite">
                          <p>Hi all,</p>
                          <p>I used git bisect to determine when the
                            memory need increased. I found that the
                            first "bad" commit is  
                            aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
                          <p>Barry was right, this commit seems to be
                            about an evolution of <span class="gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
                                You mentioned the option "-matptap_via
                                scalable" but I can't find any
                                information about it. Can you tell me
                                more?</span></span></p>
                          <p><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
                          <p><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
                          <p><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
                          <br>
                          <div class="gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
                            03/11/19 à 14:40, Mark Adams a écrit :<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">Is there a difference in
                              memory usage on your tiny problem? I
                              assume no.
                              <div><br>
                              </div>
                              <div>I don't see anything that could come
                                from GAMG other than the RAP stuff that
                                you have discussed already.</div>
                            </div>
                            <br>
                            <div class="gmail_quote">
                              <div dir="ltr" class="gmail_attr">On Mon,
                                Mar 11, 2019 at 9:32 AM Myriam
                                Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                wrote:<br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#FFFFFF">
                                  <p>The code I am using here is the
                                    example 42 of PETSc (<a class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
                                    Indeed it solves the Stokes
                                    equation. I thought it was a good
                                    idea to use an example you might
                                    know (and didn't find any that uses
                                    GAMG functions). I just changed the
                                    PCMG setup so that the memory
                                    problem appears. And it appears when
                                    adding PCGAMG.</p>
                                  <p>I don't care about the performance
                                    or even the result rightness here,
                                    but only about the difference in
                                    memory use between 3.6 and 3.10. Do
                                    you think finding a more adapted
                                    script would help?<br>
                                  </p>
                                  <p>I used the threshold of 0.1 only
                                    once, at the beginning, to test its
                                    influence. I used the default
                                    threshold (of 0, I guess) for all
                                    the other runs.</p>
                                  <p>Myriam<br>
                                  </p>
                                  <br>
                                  <div class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
                                    03/11/19 à 13:52, Mark Adams a
                                    écrit :<br>
                                  </div>
                                  <blockquote type="cite">
                                    <div dir="ltr">
                                      <div dir="ltr">In looking at this
                                        larger scale run ...
                                        <div><br>
                                        </div>
                                        <div>* Your eigen estimates are
                                          much lower than your tiny test
                                          problem.  But this is Stokes
                                          apparently and it should not
                                          work anyway. Maybe you have a
                                          small time step that adds a
                                          lot of mass that brings the
                                          eigen estimates down. And your
                                          min eigenvalue (not used) is
                                          positive. I would expect
                                          negative for Stokes ...</div>
                                        <div><br>
                                        </div>
                                        <div>* You seem to be setting a
                                          threshold value of 0.1 -- that
                                          is very high</div>
                                        <div><br>
                                        </div>
                                        <div>* v3.6 says "using nonzero
                                          initial guess" but this is not
                                          in v3.10. Maybe we just
                                          stopped printing that.</div>
                                        <div><br>
                                        </div>
                                        <div>* There were some changes
                                          to coasening parameters in
                                          going from v3.6 but it does
                                          not look like your problem was
                                          effected. (The coarsening algo
                                          is non-deterministic by
                                          default and you can see small
                                          difference on different runs)</div>
                                        <div><br>
                                        </div>
                                        <div>* We may have also added a
                                          "noisy" RHS for eigen
                                          estimates by default from
                                          v3.6.</div>
                                        <div><br>
                                        </div>
                                        <div>* And for non-symetric
                                          problems you can try
                                          -pc_gamg_agg_nsmooths 0, but
                                          again GAMG is not built for
                                          Stokes anyway.</div>
                                        <div><br>
                                        </div>
                                      </div>
                                    </div>
                                    <br>
                                    <div class="gmail_quote">
                                      <div dir="ltr" class="gmail_attr">On
                                        Tue, Mar 5, 2019 at 11:53 AM
                                        Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                        wrote:<br>
                                      </div>
                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                        <div bgcolor="#FFFFFF">
                                          <p>I used PCView to display
                                            the size of the linear
                                            system in each level of the
                                            MG. You'll find the outputs
                                            attached to this mail (zip
                                            file) for both the default
                                            threshold value and a value
                                            of 0.1, and for both 3.6 and
                                            3.10 PETSc versions. <br>
                                          </p>
                                          <p>For convenience, I
                                            summarized the information
                                            in a graph, also attached
                                            (png file).</p>
                                          <p>As you can see, there are
                                            slight differences between
                                            the two versions but none is
                                            critical, in my opinion. Do
                                            you see anything suspicious
                                            in the outputs?</p>
                                          <p>+ I can't find the default
                                            threshold value. Do you know
                                            where I can find it?<br>
                                          </p>
                                          <p>Thanks for the follow-up</p>
                                          <p>Myriam<br>
                                          </p>
                                          <br>
                                          <div class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
                                            03/05/19 à 14:06, Matthew
                                            Knepley a écrit :<br>
                                          </div>
                                          <blockquote type="cite">
                                            <div dir="ltr">
                                              <div dir="ltr">On Tue, Mar
                                                5, 2019 at 7:14 AM
                                                Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
                                                wrote:<br>
                                              </div>
                                              <div class="gmail_quote">
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <p>Hi Matt,</p>
                                                    <p>I plotted the
                                                      memory scalings
                                                      using different
                                                      threshold values.
                                                      The two scalings
                                                      are slightly
                                                      translated (from
                                                      -22 to -88 mB) but
                                                      this gain is
                                                      neglectable. The
                                                      3.6-scaling keeps
                                                      being robust while
                                                      the 3.10-scaling
                                                      deteriorates.</p>
                                                    <p>Do you have any
                                                      other suggestion?</p>
                                                  </div>
                                                </blockquote>
                                                <div>Mark, what is the
                                                  option she can give to
                                                  output all the GAMG
                                                  data?</div>
                                                <div><br>
                                                </div>
                                                <div>Also, run using
                                                  -ksp_view. GAMG will
                                                  report all the sizes
                                                  of its grids, so it
                                                  should be easy to see</div>
                                                <div>if the coarse grid
                                                  sizes are increasing,
                                                  and also what the
                                                  effect of the
                                                  threshold value is.</div>
                                                <div><br>
                                                </div>
                                                <div>  Thanks,</div>
                                                <div><br>
                                                </div>
                                                <div>     Matt <br>
                                                </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <p>Thanks<br>
                                                    </p>
                                                    Myriam <br>
                                                    <br>
                                                    <div class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
                                                      03/02/19 à 02:27,
                                                      Matthew Knepley a
                                                      écrit :<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Fri, Mar 1,
                                                          2019 at 10:53
                                                          AM Myriam
                                                          Peyrounette
                                                          via
                                                          petsc-users
                                                          <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                          I used to run
                                                          my code with
                                                          PETSc 3.6.
                                                          Since I
                                                          upgraded the
                                                          PETSc version<br>
                                                          to 3.10, this
                                                          code has a bad
                                                          memory
                                                          scaling.<br>
                                                          <br>
                                                          To report this
                                                          issue, I took
                                                          the PETSc
                                                          script ex42.c
                                                          and slightly<br>
                                                          modified it so
                                                          that the KSP
                                                          and PC
                                                          configurations
                                                          are the same
                                                          as in my<br>
                                                          code. In
                                                          particular, I
                                                          use a
                                                          "personnalised"
                                                          multi-grid
                                                          method. The<br>
                                                          modifications
                                                          are indicated
                                                          by the keyword
                                                          "TopBridge" in
                                                          the attached<br>
                                                          scripts.<br>
                                                          <br>
                                                          To plot the
                                                          memory (weak)
                                                          scaling, I ran
                                                          four
                                                          calculations
                                                          for each<br>
                                                          script with
                                                          increasing
                                                          problem sizes
                                                          and
                                                          computations
                                                          cores:<br>
                                                          <br>
                                                          1. 100,000
                                                          elts on 4
                                                          cores<br>
                                                          2. 1 million
                                                          elts on 40
                                                          cores<br>
                                                          3. 10 millions
                                                          elts on 400
                                                          cores<br>
                                                          4. 100
                                                          millions elts
                                                          on 4,000 cores<br>
                                                          <br>
                                                          The resulting
                                                          graph is also
                                                          attached. The
                                                          scaling using
                                                          PETSc 3.10<br>
                                                          clearly
                                                          deteriorates
                                                          for large
                                                          cases, while
                                                          the one using
                                                          PETSc 3.6 is<br>
                                                          robust.<br>
                                                          <br>
                                                          After a few
                                                          tests, I found
                                                          that the
                                                          scaling is
                                                          mostly
                                                          sensitive to
                                                          the<br>
                                                          use of the AMG
                                                          method for the
                                                          coarse grid
                                                          (line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
                                                          deteriorates
                                                          when
                                                          commenting
                                                          lines 1777 to
                                                          1790 (in
                                                          main_ex42_petsc36.cc).<br>
                                                          <br>
                                                          Do you have
                                                          any idea of
                                                          what changed
                                                          between
                                                          version 3.6
                                                          and version<br>
                                                          3.10 that may
                                                          imply such
                                                          degradation?<br>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>I believe
                                                          the default
                                                          values for
                                                          PCGAMG changed
                                                          between
                                                          versions. It
                                                          sounds like
                                                          the coarsening
                                                          rate</div>
                                                          <div>is not
                                                          great enough,
                                                          so that these
                                                          grids are too
                                                          large. This
                                                          can be set
                                                          using:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>There is
                                                          some
                                                          explanation of
                                                          this effect on
                                                          that page. Let
                                                          us know if
                                                          setting this
                                                          does not
                                                          correct the
                                                          situation.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     Matt</div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          Let me know if
                                                          you need
                                                          further
                                                          information.<br>
                                                          <br>
                                                          Best,<br>
                                                          <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          Myriam
                                                          Peyrounette<br>
                                                          CNRS/IDRIS -
                                                          HLST<br>
                                                          --<br>
                                                          <br>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr" class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                    <pre class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                                  </div>
                                                </blockquote>
                                              </div>
                                              <br clear="all">
                                              <div><br>
                                              </div>
                                              -- <br>
                                              <div dir="ltr" class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
                                                <div dir="ltr">
                                                  <div>
                                                    <div dir="ltr">
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </blockquote>
                                          <br>
                                          <pre class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                        </div>
                                      </blockquote>
                                    </div>
                                  </blockquote>
                                  <br>
                                  <pre class="gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                                </div>
                              </blockquote>
                            </div>
                          </blockquote>
                          <br>
                          <pre class="gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                        </blockquote>
                        <br>
                        <pre class="gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
                      </div>
                    </blockquote>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="gmail-m_7299408251829905453moz-signature" cols="72">-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>