<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Mar 26, 2018 at 1:15 AM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <br>
    <div class="m_2941034901033641399moz-cite-prefix">On 24/3/2018 6:00 AM, Matthew Knepley
      wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">On Fri, Mar 23, 2018 at 5:23 AM, TAY
            wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                <div class="m_2941034901033641399m_746100086503134357moz-cite-prefix">On
                  21/3/2018 7:47 PM, Matthew Knepley wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div class="gmail_extra">
                      <div class="gmail_quote">On Wed, Mar 21, 2018 at
                        4:59 AM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> <br>
                            <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118moz-cite-prefix">On
                              19/3/2018 6:32 PM, Matthew Knepley wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div class="gmail_extra">
                                  <div class="gmail_quote">On Mon, Mar
                                    19, 2018 at 5:19 AM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
                                    wrote:<br>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF"> <br>
                                        <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354moz-cite-prefix">On
                                          17/3/2018 1:15 AM, Matthew
                                          Knepley wrote:<br>
                                        </div>
                                        <blockquote type="cite">
                                          <div dir="ltr">
                                            <div class="gmail_extra">
                                              <div class="gmail_quote">On
                                                Fri, Mar 16, 2018 at
                                                12:54 PM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
                                                wrote:<br>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <br>
                                                    <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354gmail-m_-1748057115269175174moz-cite-prefix">On
                                                      15/3/2018 6:21 PM,
                                                      Matthew Knepley
                                                      wrote:<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div class="gmail_extra">
                                                          <div class="gmail_quote">On
                                                          Thu, Mar 15,
                                                          2018 at 3:51
                                                          PM, TAY
                                                          wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                          I'm running a
                                                          CFD code which
                                                          solves the
                                                          momentum and
                                                          Poisson eqns.<br>
                                                          <br>
                                                          Due to poor
                                                          scaling with
                                                          HYPRE at
                                                          higher cpu
                                                          no., I decided
                                                          to try using
                                                          PETSc with
                                                          boomeramg and
                                                          gamg.<br>
                                                          <br>
                                                          I tested for
                                                          some small
                                                          cases and it
                                                          work well.
                                                          However, for
                                                          the large
                                                          problem which
                                                          has poor
                                                          scaling, it
                                                          gives an error
                                                          when I change
                                                          my Poisson
                                                          solver from
                                                          pure HYPRE to
                                                          PETSc with
                                                          boomeramg and
                                                          gamg.<br>
                                                          <br>
                                                          The error is :<br>
                                                          <br>
                                                          Caught signal
                                                          number 11
                                                          SEGV:
                                                          Segmentation
                                                          Violation,
                                                          probably
                                                          memory access
                                                          out of range<br>
                                                          <br>
                                                          I tried using:<br>
                                                          <br>
-poisson_ksp_type richardson -poisson_pc_type hypre
                                                          -poisson_pc_type_hypre
                                                          boomeramg<br>
                                                          <br>
-poisson_ksp_type gmres -poisson_pc_type hypre -poisson_pc_type_hypre
                                                          boomeramg<br>
                                                          <br>
-poisson_pc_type gamg
-poisson_pc_gamg_agg_nsmooths 1<br>
                                                          <br>
                                                          but they all
                                                          gave similar
                                                          error.<br>
                                                          <br>
                                                          So why is this
                                                          so? How should
                                                          I
                                                          troubleshoot?
                                                          I am now
                                                          running a
                                                          debug ver of
                                                          PETSc to check
                                                          the error msg.</blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>1) For
                                                          anything like
                                                          this, we would
                                                          like to see a
                                                          stack trace
                                                          from the
                                                          debugger or
                                                          valgrind
                                                          output.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>2) We do
                                                          have several
                                                          Poisson
                                                          examples. Does
                                                          it fail for
                                                          you on those?</div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                    Hi,<br>
                                                    <br>
                                                    Can you recommend me
                                                    some suitable egs?
                                                    Esp in Fortran?<br>
                                                  </div>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                                <div>Here is 2D Poisson</div>
                                                <div><br>
                                                </div>
                                                <div>  <a href="https://bitbucket.org/petsc/petsc/src/4b6141395f14f0c7d1415a2ff0158eec75a27d63/src/snes/examples/tutorials/ex5f.F90?at=master&fileviewer=file-view-default" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/src/4b6141395f14f0c7d141<wbr>5a2ff0158eec75a27d63/src/snes/<wbr>examples/tutorials/ex5f.F90?at<wbr>=master&fileviewer=file-view-d<wbr>efault</a></div>
                                                <div> </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div class="gmail_extra">
                                                          <div class="gmail_quote">
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                </blockquote>
                                              </div>
                                            </div>
                                          </div>
                                        </blockquote>
                                      </div>
                                    </blockquote>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                            Hi,<br>
                            <br>
                            I tried to use the different options like
                            ML, MG BoomerAMG and while they worked for
                            the small problem, it failed for the large
                            problem. I checked and compared the 2D
                            Poisson example with my 3D Poisson
                            subroutine. I found that my subroutine is
                            very similar to my momentum subroutine,
                            which is based on KSPSolve:<br>
                            <br>
                            0. DMDACreate, DMDACreate3d etc<br>
                            1. Assemble matrix<br>
                            2. KSPSetOperators, KSPSetType etc<br>
                            3. KSPSolve<br>
                            <br>
                            However, the 2D Poisson example uses
                            SNESSetDM, SNESSolve etc. So does it matter
                            if I use SNESSolve or KSPSolve?<br>
                          </div>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>Not if you are using Algebraic Multigrid.
                          "It failed for the large problem" is not
                          descriptive enough. Please send the output of</div>
                        <div><br>
                        </div>
                        <div>  -ksp_view -ksp_monitor_true_residual</div>
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> I also
                            didn't set any nullspace. Is this required
                            for a Poisson eqn solve?<br>
                          </div>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>If you only have the constant null space,
                          no. Most AMG packages do that one by default.</div>
                        <div><br>
                        </div>
                        <div>  Matt</div>
                      </div>
                    </div>
                  </div>
                </blockquote>
                Hi,<br>
                <br>
                I tried to use <br>
                <br>
                1. -ksp_view -ksp_monitor_true_residual<br>
                2. -poisson_ksp_view -poisson_ksp_monitor_true_resi<wbr>dual<br>
                <br>
                However, this did not give any output. May I know why?</div>
            </blockquote>
            <div><br>
            </div>
            <div>You need to call</div>
            <div><br>
            </div>
            <div>  KSPSetFromOptions()</div>
          </div>
        </div>
      </div>
    </blockquote>
    Hi Matt, <br>
    <br>
    But I thought I already called:<br>
    <br>
    call KSPSetOperators(ksp,A_mat,A_<wbr>mat,ierr)<br>
    <br>
        call KSPGetPC(ksp,pc,ierr)  <br>
        <br>
        call KSPSetOptionsPrefix(ksp,"<wbr>poisson_",ierr)<br>
    <br>
        call KSPSetType(ksp,ksptype,ierr)<br>
    <br>
        ksptype=KSPRICHARDSON <br>
        <br>
        call KSPSetType(ksp,ksptype,ierr)<br>
    <br>
        call PCSetType(pc,'hypre',ierr)<br>
    <br>
        call KSPSetFromOptions(ksp,ierr)<br>
    <br>
    ...<br>
    <br>
    Is my sequence of calling wrong?<br></div></blockquote><div><br></div><div>If its not viewing, something is wrong. Here is the easiest way to make things work.</div><div>Take an example, and make it do what you want, like view. Then change it until it</div><div>is your own thing.</div><div><br></div><div>The code snippet is right, so there must be something around it that is messing things up.</div><div><br></div><div>  Thanks,</div><div><br></div><div>    Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
    Thanks.<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div><br>
            </div>
            <div>after you create it. When you launch valgrind you need</div>
            <div><br>
            </div>
            <div>  valgrind --trace-children=yes <exec></div>
            <div><br>
            </div>
            <div>   Matt</div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> I also tried
                valgrind:<br>
                <br>
                ==32157== Memcheck, a memory error detector<br>
                ==32157== Copyright (C) 2002-2012, and GNU GPL'd, by
                Julian Seward et al.<br>
                ==32157== Using Valgrind-3.8.1 and LibVEX; rerun with -h
                for copyright info<br>
                ==32157== Command:
                /app/intel/xe2018/compilers_an<wbr>d_libraries_2018.0.128/linux/<wbr>mpi/intel64/bin/mpirun
                ./a.out -poisson_ksp_type richardson -poisson_pc_type
                hypre -poisson_pc_type_hypre boomeramg<br>
                ==32157==<br>
                ==32159==<br>
                ==32159== HEAP SUMMARY:<br>
                ==32159==     in use at exit: 49,981 bytes in 1,137
                blocks<br>
                ==32159==   total heap usage: 1,542 allocs, 405 frees,
                72,030 bytes allocated<br>
                ==32159==<br>
                ==32159== LEAK SUMMARY:<br>
                ==32159==    definitely lost: 0 bytes in 0 blocks<br>
                ==32159==    indirectly lost: 0 bytes in 0 blocks<br>
                ==32159==      possibly lost: 0 bytes in 0 blocks<br>
                ==32159==    still reachable: 49,981 bytes in 1,137
                blocks<br>
                ==32159==         suppressed: 0 bytes in 0 blocks<br>
                ==32159== Reachable blocks (those to which a pointer was
                found) are not shown.<br>
                ==32159== To see them, rerun with: --leak-check=full
                --show-reachable=yes<br>
                ==32159==<br>
                ==32159== For counts of detected and suppressed errors,
                rerun with: -v<br>
                ==32159== ERROR SUMMARY: 0 errors from 0 contexts
                (suppressed: 8 from 6)<br>
                [362]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
                [362]PETSC ERROR: Caught signal number 11 SEGV:
                Segmentation Violation, probably memory access out of
                range<br>
                [362]PETSC ERROR: Try option -start_in_debugger or
                -on_error_attach_debugger<br>
                [362]PETSC ERROR: or see <a class="m_2941034901033641399m_746100086503134357moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
                [362]PETSC ERROR: or try <a class="m_2941034901033641399m_746100086503134357moz-txt-link-freetext" href="http://valgrind.org" target="_blank">http://valgrind.org</a> on
                GNU/linux and Apple Mac OS X to find memory corruption
                errors<br>
                [362]PETSC ERROR: configure using --with-debugging=yes,
                recompile, link, and run<br>
                [362]PETSC ERROR: to get more information on the crash.<br>
                [362]PETSC ERROR: --------------------- Error Message
                ------------------------------<wbr>------------------------------<wbr>--<br>
                [362]PETSC ERROR: Signal received<br>
                [362]PETSC ERROR: See <a class="m_2941034901033641399m_746100086503134357moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a>
                for trouble shooting.<br>
                [362]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09,
                2017<br>
                [362]PETSC ERROR: ./a.out on a petsc-3.8.3_intel_rel
                named std1055 by tsltaywb Thu Mar 22 17:43:54 2018<br>
                [362]PETSC ERROR: Configure options
                --with-mpi-dir=/app/intel/xe20<wbr>18/compilers_and_libraries_<wbr>2018.0.128/linux/mpi/intel64
                --with-blaslapack-dir=/app/int<wbr>el/xe2018/compilers_and_librar<wbr>ies_2018.0.128/linux/mkl/lib/<wbr>intel64
                --download-hypre=/home/users/n<wbr>us/tsltaywb/source/git.hypre.t<wbr>ar.gz
                --with-debugging=0 --prefix=/home/users/nus/tslta<wbr>ywb/lib/petsc-3.8.3_intel_rel
                --with-shared-libraries=0 --known-mpi-shared-libraries=0
                --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2"
                --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2"<br>
                [362]PETSC ERROR: #1 User provided function() line 0 in 
                unknown file<br>
                application called MPI_Abort(MPI_COMM_WORLD, 59) -
                process 362<br>
                ==32409==<br>
                ==32409== HEAP SUMMARY:<br>
                ==32409==     in use at exit: 52,168 bytes in 1,223
                blocks<br>
                ==32409==   total heap usage: 3,055 allocs, 1,832 frees,
                122,188 bytes allocated<br>
                ==32409==<br>
                ==32409== 10 bytes in 1 blocks are definitely lost in
                loss record 89 of 319<br>
                ==32409==    at 0x4C28A2E: malloc
                (vg_replace_malloc.c:270)<br>
                ==32409==    by 0x466742: xmalloc (in /bin/bash)<br>
                ==32409==    by 0x42EC78: ??? (in /bin/bash)<br>
                ==32409==    by 0x430462: execute_command_internal (in
                /bin/bash)<br>
                ==32409==    by 0x43320D: ??? (in /bin/bash)<br>
                ==32409==    by 0x433629: ??? (in /bin/bash)<br>
                ==32409==    by 0x4303BC: execute_command_internal (in
                /bin/bash)<br>
                ==32409==    by 0x43110D: execute_command (in /bin/bash)<br>
                ==32409==    by 0x41D6D5: reader_loop (in /bin/bash)<br>
                ==32409==    by 0x41CEBB: main (in /bin/bash)<br>
                ==32409==<br>
                ==32409== LEAK SUMMARY:<br>
                ==32409==    definitely lost: 10 bytes in 1 blocks<br>
                ==32409==    indirectly lost: 0 bytes in 0 blocks<br>
                ==32409==      possibly lost: 0 bytes in 0 blocks<br>
                ==32409==    still reachable: 52,158 bytes in 1,222
                blocks<br>
                ==32409==         suppressed: 0 bytes in 0 blocks<br>
                ==32409== Reachable blocks (those to which a pointer was
                found) are not shown.<br>
                ==32409== To see them, rerun with: --leak-check=full
                --show-reachable=yes<br>
                ==32409== Reachable blocks (those to which a pointer was
                found) are not shown.<br>
                ==32409== To see them, rerun with: --leak-check=full
                --show-reachable=yes<br>
                ==32409==<br>
                ==32409== For counts of detected and suppressed errors,
                rerun with: -v<br>
                ==32409== ERROR SUMMARY: 1 errors from 1 contexts
                (suppressed: 8 from 6)<br>
                ==32157==<br>
                ==32157== HEAP SUMMARY:<br>
                ==32157==     in use at exit: 55,861 bytes in 1,202
                blocks<br>
                ==32157==   total heap usage: 3,249 allocs, 2,047 frees,
                125,857 bytes allocated<br>
                ==32157==<br>
                ==32157== LEAK SUMMARY:<br>
                ==32157==    definitely lost: 0 bytes in 0 blocks<br>
                ==32157==    indirectly lost: 0 bytes in 0 blocks<br>
                ==32157==      possibly lost: 0 bytes in 0 blocks<br>
                ==32157==    still reachable: 55,861 bytes in 1,202
                blocks<br>
                ==32157==         suppressed: 0 bytes in 0 blocks<br>
                ==32157== Reachable blocks (those to which a pointer was
                found) are not shown.<br>
                ==32157== To see them, rerun with: --leak-check=full
                --show-reachable=yes<br>
                ==32157==<br>
                ==32157== For counts of detected and suppressed errors,
                rerun with: -v<br>
                ==32157== ERROR SUMMARY: 0 errors from 0 contexts
                (suppressed: 8 from 6)<br>
                <br>
                Is this the output I'm supposed to get?<br>
                <br>
                Thanks!<br>
                <br>
                <br>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div class="gmail_extra">
                      <div class="gmail_quote">
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> Thanks<br>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div class="gmail_extra">
                                  <div class="gmail_quote">
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF">
                                        <blockquote type="cite">
                                          <div dir="ltr">
                                            <div class="gmail_extra">
                                              <div class="gmail_quote">
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div class="gmail_extra">
                                                          <div class="gmail_quote">
                                                          <div> </div>
                                                          <div>3) You
                                                          can also try
                                                          ML, which is
                                                          the same type
                                                          of MG as GAMG.
(--download-ml).</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                </blockquote>
                                              </div>
                                            </div>
                                          </div>
                                        </blockquote>
                                        I have recompiled PETSc with ML.
                                        Is there an example command line
                                        options which I can use for ML?<br>
                                      </div>
                                    </blockquote>
                                    <div><br>
                                    </div>
                                    <div>-pc_type ml</div>
                                    <div> </div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF"> Another
                                        question is generally speaking,
                                        is geometric multigrid (GMG)
                                        faster than algebraic?<br>
                                      </div>
                                    </blockquote>
                                    <div><br>
                                    </div>
                                    <div>No, only for the setup time.</div>
                                    <div> </div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF"> I tested on a
                                        small problem and the time taken
                                        varies from 1.15min (HYPRE,
                                        geometric) to 3.25 (GAMG).
                                        BoomerAMG is 1.45min.<br>
                                      </div>
                                    </blockquote>
                                    <div><br>
                                    </div>
                                    <div>I am not sure what you are
                                      running when you say Hypre
                                      geometric.</div>
                                    <div> </div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF"> Besides
                                        HYPRE, is there any other GMG I
                                        can use? <br>
                                      </div>
                                    </blockquote>
                                    <div><br>
                                    </div>
                                    <div>As I said above, it does not
                                      converge any faster and the solve
                                      is not faster, its all setup time,
                                      so small problems will look
                                      faster.</div>
                                    <div>You should be doing 10-20
                                      iterates. If you are doing more,
                                      MG is not working.<br>
                                    </div>
                                    <div><br>
                                    </div>
                                    <div>If you truly have a structured
                                      grid, then use DMDA in PETSc and
                                      you can use GMG with</div>
                                    <div><br>
                                    </div>
                                    <div>  -pc_type mg -pc_mg_nlevels
                                      <n></div>
                                    <div><br>
                                    </div>
                                    <div>   Matt</div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF">
                                        <blockquote type="cite">
                                          <div dir="ltr">
                                            <div class="gmail_extra">
                                              <div class="gmail_quote">
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div class="gmail_extra">
                                                          <div class="gmail_quote">
                                                          <div> </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                    My cluster can't
                                                    connect to the
                                                    internet. Where can
                                                    I 1st download it?<br>
                                                    <br>
                                                    Similarly, how can I
                                                    find out the
                                                    location of the ext
                                                    software by myself?<br>
                                                  </div>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                                <div>The locations are
                                                  all in the configure
                                                  Python modules:</div>
                                                <div><br>
                                                </div>
                                                <div>  <a href="https://bitbucket.org/petsc/petsc/src/4b6141395f14f0c7d1415a2ff0158eec75a27d63/config/BuildSystem/config/packages/ml.py?at=master&fileviewer=file-view-default" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/src/4b6141395f14f0c7d141<wbr>5a2ff0158eec75a27d63/config/Bu<wbr>ildSystem/config/packages/ml.p<wbr>y?at=master&fileviewer=file-vi<wbr>ew-default</a></div>
                                                <div><br>
                                                </div>
                                                <div>  Thanks,</div>
                                                <div><br>
                                                </div>
                                                <div>    Matt</div>
                                                <div> </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div class="gmail_extra">
                                                          <div class="gmail_quote">
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     Matt</div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354gmail-m_-1748057115269175174HOEnZb"><font color="#888888"><br>
                                                          -- <br>
                                                          Thank you very
                                                          much.<br>
                                                          <br>
                                                          Yours
                                                          sincerely,<br>
                                                          <br>
==============================<wbr>==================<br>
                                                          TAY Wee-Beng
                                                          (Zheng
                                                          Weiming) 郑伟明<br>
                                                          Personal
                                                          research
                                                          webpage: <a href="http://tayweebeng.wixsite.com/website" rel="noreferrer" target="_blank">http://tayweebeng.wixsite.com/<wbr>website</a><br>
                                                          Youtube
                                                          research
                                                          showcase: <a href="https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA" rel="noreferrer" target="_blank">https://www.youtube.com/channe<wbr>l/UC72ZHtvQNMpNs2uRTSToiLA</a><br>
                                                          linkedin: <a href="http://www.linkedin.com/in/tay-weebeng" rel="noreferrer" target="_blank">www.linkedin.com/in/tay-weeben<wbr>g</a><br>
==============================<wbr>==================<br>
                                                          <br>
                                                          </font></span></blockquote>
                                                          </div>
                                                          <br>
                                                          <br clear="all">
                                                          <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354gmail-HOEnZb"><font color="#888888">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354gmail-m_-1748057115269175174gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </font></span></font></span></div>
                                                        <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> </font></span></div>
                                                      <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> </font></span></blockquote>
                                                    <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888">
                                                        <br>
                                                      </font></span></div>
                                                  <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> </font></span></blockquote>
                                                <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> </font></span></div>
                                              <span class="m_2941034901033641399m_746100086503134357m_-619408701181955118HOEnZb"><font color="#888888"> <br>
                                                  <br clear="all">
                                                  <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888">
                                                      <div><br>
                                                      </div>
                                                      -- <br>
                                                      <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118m_4970404346621557354gmail_signature">
                                                        <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </font></span></font></span></div>
                                            <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> </font></span></div>
                                          <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> </font></span></blockquote>
                                        <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> <br>
                                          </font></span></div>
                                      <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> </font></span></blockquote>
                                    <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> </font></span></div>
                                  <span class="m_2941034901033641399m_746100086503134357HOEnZb"><font color="#888888"> <br>
                                      <br clear="all">
                                      <span class="m_2941034901033641399HOEnZb"><font color="#888888">
                                          <div><br>
                                          </div>
                                          -- <br>
                                          <div class="m_2941034901033641399m_746100086503134357m_-619408701181955118gmail_signature" data-smartmail="gmail_signature">
                                            <div dir="ltr">
                                              <div>
                                                <div dir="ltr">
                                                  <div>What most
                                                    experimenters take
                                                    for granted before
                                                    they begin their
                                                    experiments is
                                                    infinitely more
                                                    interesting than any
                                                    results to which
                                                    their experiments
                                                    lead.<br>
                                                    -- Norbert Wiener</div>
                                                  <div><br>
                                                  </div>
                                                  <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </font></span></font></span></div>
                                <span class="m_2941034901033641399HOEnZb"><font color="#888888"> </font></span></div>
                              <span class="m_2941034901033641399HOEnZb"><font color="#888888">
                                </font></span></blockquote>
                            <span class="m_2941034901033641399HOEnZb"><font color="#888888">
                                <br>
                              </font></span></div>
                          <span class="m_2941034901033641399HOEnZb"><font color="#888888"> </font></span></blockquote>
                        <span class="m_2941034901033641399HOEnZb"><font color="#888888"> </font></span></div>
                      <span class="m_2941034901033641399HOEnZb"><font color="#888888"> <br>
                          <br clear="all"><span class="HOEnZb"><font color="#888888">
                          <div><br>
                          </div>
                          -- <br>
                          <div class="m_2941034901033641399m_746100086503134357gmail_signature" data-smartmail="gmail_signature">
                            <div dir="ltr">
                              <div>
                                <div dir="ltr">
                                  <div>What most experimenters take for
                                    granted before they begin their
                                    experiments is infinitely more
                                    interesting than any results to
                                    which their experiments lead.<br>
                                    -- Norbert Wiener</div>
                                  <div><br>
                                  </div>
                                  <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </font></span></font></span></div><span class="HOEnZb"><font color="#888888">
                  </font></span></div><span class="HOEnZb"><font color="#888888">
                </font></span></blockquote><span class="HOEnZb"><font color="#888888">
                <br>
              </font></span></div><span class="HOEnZb"><font color="#888888">
            </font></span></blockquote><span class="HOEnZb"><font color="#888888">
          </font></span></div><span class="HOEnZb"><font color="#888888">
          <br>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          <div class="m_2941034901033641399gmail_signature" data-smartmail="gmail_signature">
            <div dir="ltr">
              <div>
                <div dir="ltr">
                  <div>What most experimenters take for granted before
                    they begin their experiments is infinitely more
                    interesting than any results to which their
                    experiments lead.<br>
                    -- Norbert Wiener</div>
                  <div><br>
                  </div>
                  <div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </font></span></div>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>