<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    I have done some more testing of the problem, continuing with
    src/ksp/ksp/examples/tutorials/ex2.c.<br>
    <br>
    The behavior I am seeing is that with smaller problems sizes
    superlu_dist is behaving properly<br>
    but with larger problem sizes things seem to go wrong and what goes
    wrong is apparently consistent; the error appears both with my intel
    build as well as with my gcc build.<br>
    <br>
    I have two run lines:<br>
    <br>
    runex2superlu:<br>
            -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100
    -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package
    superlu_dist<br>
    <br>
    runex2spooles:<br>
            -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 100 -n 100
    -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package spooles<br>
    <br>
    From my intel build, I get<br>
    <br>
    sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
    Norm of error 7.66145e-13 iterations 1<br>
    sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
    Norm of error 2.21422e-12 iterations 1<br>
    <br>
    From my GCC build, I get<br>
    sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
    Norm of error 7.66145e-13 iterations 1<br>
    sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
    Norm of error 2.21422e-12 iterations 1<br>
    <br>
    If I change the -m 100 -n 100 to -m 500 -n 500, I get for my intel
    build<br>
    <br>
    sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
    Norm of error 419.953 iterations 1<br>
    sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
    Norm of error 2.69468e-10 iterations 1<br>
    <br>
    From my GCC build with -m 500 -n 500, I get<br>
    <br>
    sg-macbook-prolocal:tutorials sg$ make runex2superlu<br>
    Norm of error 419.953 iterations 1<br>
    sg-macbook-prolocal:tutorials sg$ make runex2spooles<br>
    Norm of error 2.69468e-10 iterations 1<br>
    <br>
    <br>
    Any suggestions will be greatly appreciated.<br>
    <br>
    -sanjay<br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 12/23/12 6:42 PM, Matthew Knepley
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAMYG4GmW7FfEqvV6YEPygXFjpxjD2sDA9fkroAJ+DXXbjiW9nw@mail.gmail.com"
      type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra">On Sun, Dec 23, 2012 at 9:37 PM, Sanjay
          Govindjee <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>></span>
          wrote:<br>
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000"> I decided to go
                with ksp/ksp/exampeles/tutorials/ex2.c; I was unsure how
                to convert the run lines for snes/examples/ex5.c to work
                with a direct solver as I am not versed in SNES options.<br>
                <br>
                Notwithstanding something strange is happening only on
                select examples.  With ksp/ksp/exampeles/tutorials/ex2.c
                and the run line:<br>
                <br>
                -@${MPIEXEC} -n 2 ./ex2 -ksp_monitor_short -m 20 -n 20
                -ksp_type preonly -pc_type lu
                -pc_factor_mat_solver_package superlu_dist<br>
                    <br>
                I get good results (of the order):<br>
                <br>
                Norm of error 1.85464e-14 iterations 1<br>
                <br>
                using both superlu_dist and spooles.<br>
                <br>
                My BLAS/LAPACK: -llapack -lblas (so native to my
                machine).<br>
                <br>
                If you can guide me on a run line for the snes ex5.c I
                can try that too.  I'll also try to construct a GCC
                build later to see if that is an issue.<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div style="">Same line on ex5, but ex2 is good enough.
              However, it will not tell us anything new. Try another
              build.</div>
            <div style=""><br>
            </div>
            <div style="">   Matt</div>
            <div>
               </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000"> -sanjay<br>
                <br>
                <br>
                <div>On 12/23/12 5:58 PM, Matthew Knepley wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">On Sun, Dec 23, 2012 at 8:08 PM, Sanjay
                    Govindjee <span dir="ltr"><<a
                        moz-do-not-send="true"
                        href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>></span>
                    wrote:<br>
                    <div class="gmail_extra">
                      <div class="gmail_quote">
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div bgcolor="#FFFFFF" text="#000000"> Not
                            sure what you mean by where is your matrix? 
                            I am simply running ex6 in the
                            ksp/examples/tests directory.<br>
                            <br>
                            The reason I ran this test is because I was
                            seeing the same behavior with my finite
                            element code (on perfectly benign problems).<br>
                            <br>
                            Is there a built-in test that you use to
                            check that superlu_dist is working properly
                            with petsc?<br>
                            i.e. something you know that works with with
                            petsc 3.3-p5?<br>
                          </div>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>1) Run it on a SNES ex5 (or KSP ex2), which
                          is a nice Laplacian</div>
                        <div><br>
                        </div>
                        <div>2) Compare with MUMPS</div>
                        <div><br>
                        </div>
                        <div>   Matt</div>
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div bgcolor="#FFFFFF" text="#000000"> -sanjay
                            <div>
                              <div><br>
                                <br>
                                <br>
                                <div>On 12/23/12 4:56 PM, Jed Brown
                                  wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">Where is your matrix?
                                    It might be ending up with a very
                                    bad pivot. If the problem can be
                                    reproduced, it should be reported to
                                    the SuperLU_DIST developers to fix.
                                    (Note that we do not see this with
                                    other matrices.) You can also try
                                    MUMPS.</div>
                                  <div class="gmail_extra"><br>
                                    <br>
                                    <div class="gmail_quote">On Sun, Dec
                                      23, 2012 at 6:48 PM, Sanjay
                                      Govindjee <span dir="ltr"><<a
                                          moz-do-not-send="true"
                                          href="mailto:s_g@berkeley.edu"
                                          target="_blank">s_g@berkeley.edu</a>></span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote"
                                        style="margin:0 0 0
                                        .8ex;border-left:1px #ccc
                                        solid;padding-left:1ex">
                                        <div bgcolor="#FFFFFF"
                                          text="#000000"> I wanted to
                                          use SuperLU Dist to perform a
                                          direct solve but seem to be
                                          encountering<br>
                                          a problem.  I was wonder if
                                          this is a know issue and if
                                          there is a solution for it.<br>
                                          <br>
                                          The problem is easily observed
                                          using ex6.c in
                                          src/ksp/ksp/examples/tests.<br>
                                          <br>
                                          Out of the box: make runex6
                                          produces a residual error of
                                          O(1e-11), all is well.<br>
                                          <br>
                                          I then changed the run to run
                                          on two processors and add the
                                          flag<br>
                                          -pc_factor_mat_solver_package
                                          spooles  this produces a
                                          residual error of O(1e-11),
                                          all is still well.<br>
                                          <br>
                                          I then switch over to
                                          -pc_factor_mat_solver_package
                                          superlu_dist and the<br>
                                          residual error comes back as
                                          22.6637!  Something seems very
                                          wrong.<br>
                                          <br>
                                          My build is perfectly vanilla:<br>
                                          <br>
                                          export
                                          PETSC_DIR=/Users/sg/petsc-3.3-p5/<br>
                                          export PETSC_ARCH=intel<br>
                                          <br>
                                          ./configure --with-cc=icc
                                          --with-fc=ifort  \
-download-{spooles,parmetis,superlu_dist,prometheus,mpich,ml,hypre,metis}<br>
                                          <br>
                                          make
                                          PETSC_DIR=/Users/sg/petsc-3.3-p5/
                                          PETSC_ARCH=intel all<br>
                                          make
                                          PETSC_DIR=/Users/sg/petsc-3.3-p5/
                                          PETSC_ARCH=intel test<span><font
                                              color="#888888"><br>
                                              <br>
                                              -sanjay<br>
                                            </font></span></div>
                                      </blockquote>
                                    </div>
                                    <br>
                                  </div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                            <pre cols="72">-- 
-----------------------------------------------
Sanjay Govindjee, PhD, PE
Professor of Civil Engineering
Vice Chair for Academic Affairs

779 Davis Hall
Structural Engineering, Mechanics and Materials
Department of Civil Engineering
University of California
Berkeley, CA 94720-1710

Voice:  <a moz-do-not-send="true" href="tel:%2B1%20510%20642%206060" value="+15106426060" target="_blank">+1 510 642 6060</a>
FAX:    <a moz-do-not-send="true" href="tel:%2B1%20510%20643%205264" value="+15106435264" target="_blank">+1 510 643 5264</a>
<a moz-do-not-send="true" href="mailto:s_g@berkeley.edu" target="_blank">s_g@berkeley.edu</a>
<a moz-do-not-send="true" href="http://www.ce.berkeley.edu/%7Esanjay" target="_blank">http://www.ce.berkeley.edu/~sanjay</a>
-----------------------------------------------

New Books:  

Engineering Mechanics of Deformable 
Solids: A Presentation with Exercises
<a moz-do-not-send="true" href="http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641" target="_blank">http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641</a>
<a moz-do-not-send="true" href="http://ukcatalogue.oup.com/product/9780199651641.do" target="_blank">http://ukcatalogue.oup.com/product/9780199651641.do</a>
<a moz-do-not-send="true" href="http://amzn.com/0199651647" target="_blank">http://amzn.com/0199651647</a>


Engineering Mechanics 3 (Dynamics)
<a moz-do-not-send="true" href="http://www.springer.com/materials/mechanics/book/978-3-642-14018-1" target="_blank">http://www.springer.com/materials/mechanics/book/978-3-642-14018-1</a>
<a moz-do-not-send="true" href="http://amzn.com/3642140181" target="_blank">http://amzn.com/3642140181</a>

-----------------------------------------------
</pre>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <span class="HOEnZb"><font color="#888888">
                          <div><br>
                          </div>
                          -- <br>
                          What most experimenters take for granted
                          before they begin their experiments is
                          infinitely more interesting than any results
                          to which their experiments lead.<br>
                          -- Norbert Wiener </font></span></div>
                  </div>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          What most experimenters take for granted before they begin
          their experiments is infinitely more interesting than any
          results to which their experiments lead.<br>
          -- Norbert Wiener
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">
</pre>
  </body>
</html>