<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi Mark,<br>
      <br>
      I hope this back trace might be helpful.<br>
      <br>
      You may need to build petsc with <br>
      ./configure \
      <br>
      --with-x=0 \
      <br>
      --with-debugging=0 \
      <br>
      --with-log=0 \
      <br>
      <br>
      <br>
      env003> addr2line --exe=tpetsc_madams 0x4b76e3 0x4aa3d7
      0x484888 0x475798 0x548aa5 0x5e5df2 0x57a353 0x4a701e  0x44e9be
      0x44e1f4 <br>
      /autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:27<br>
      /autofs/na3_home1/adams/petsc/src/sys/utils/str.c:188<br>
      /autofs/na3_home1/adams/petsc/src/sys/logging/utils/eventlog.c:317<br>
      /autofs/na3_home1/adams/petsc/src/sys/logging/plog.c:747<br>
      /autofs/na3_home1/adams/petsc/src/mat/interface/dlregismat.c:145<br>
      /autofs/na3_home1/adams/petsc/src/mat/utils/gcreate.c:57<br>
      /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3576<br>
/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14<br>
      /autofs/na3_home1/efdazedo/test/PETSC/./tpetsc.F90:147<br>
      <br>
      <br>
      <br>
      <br>
      ======= Backtrace: =========<br>
      /lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
      /lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
      ./tpetsc_madams[0x4b7759]<br>
        /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
        /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
        /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
        /lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
        /lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
        /lib64/libc.so.6(__libc_malloc+0x77) [0x2aaaba6045e7]<br>
        ./tpetsc_madams() [0x4b76e3]<br>
        ./tpetsc_madams() [0x4aa3d7]<br>
        ./tpetsc_madams() [0x484888]<br>
        ./tpetsc_madams() [0x475798]<br>
        ./tpetsc_madams() [0x548aa5]<br>
        ./tpetsc_madams() [0x5e5df2]<br>
        ./tpetsc_madams() [0x57a353]<br>
        ./tpetsc_madams() [0x4a701e]<br>
        ./tpetsc_madams() [0x44e9be]<br>
        ./tpetsc_madams() [0x44e1f4]<br>
        /lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
        ./tpetsc_madams() [0x44e0e9]<br>
      <br>
      <br>
      <br>
      <br>
      <br>
      <br>
      On 04/16/2014 10:33 AM, Mark Adams wrote:<br>
    </div>
    <blockquote
cite="mid:CADOhEh769XLXz5i4_fqZsoftrMb-dzt4M=gCbPZt7Y8538RRNg@mail.gmail.com"
      type="cite">
      <div dir="ltr">cc'ing petsc-dev.
        <div><br>
        </div>
        <div>I will try it.</div>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Wed, Apr 16, 2014 at 10:21 AM, Ed
            D'Azevedo <span dir="ltr"><<a moz-do-not-send="true"
                href="mailto:dazevedoef@ornl.gov" target="_blank">dazevedoef@ornl.gov</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF">
                <div><br>
                  Hi Mark,<br>
                  <br>
                  I got an error when I tried the simple test code (see
                  attached) on Titan.<br>
                  <br>
                  Can you try to run the attached test case to see if it
                  will work for you?<br>
                  <br>
                  I have also sent the simple test code to Barry.<br>
                  <br>
                  <br>
                  <br>
                  The code seems to work with 1 thread<br>
                  <br>
                  env003> export OMP_NUM_THREADS=1<br>
                  env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
                   PETSC_VERSION_RELEASE               0<br>
                   PETSC_VERSION_MAJOR                 3<br>
                   PETSC_VERSION_MINOR                 4<br>
                   PETSC_VERSION_SUBMINOR              4<br>
                   PETSC_VERSION_PATCH                 0<br>
                   PETSC_VERSION_DATE      unknown<br>
                   petsc_version_lt(3,3,0) is false<br>
                   nthreads =             1  NCASES =           100<br>
                   nz =         88804<br>
                  Warning: ieee_inexact is signaling<br>
                  all done<br>
                   total time is     7.268438    <br>
                   maxval(err)    6.9650285539069046E-011<br>
                  Application 4895013 resources: utime ~8s, stime ~0s,
                  Rss ~22300, inblocks ~11428, outblocks ~35848<br>
                  <br>
                  <br>
                  The code seems to have trouble using more threads<br>
                  <br>
                  env003> export OMP_NUM_THREADS=16<br>
                  env003>  aprun -n 1 -d 16 ./tpetsc_madams<br>
                  *** glibc detected *** ./tpetsc_madams: double free or
                  corruption (!prev): 0x00000000015ee0b0 ***<br>
                   PETSC_VERSION_RELEASE               0<br>
                   PETSC_VERSION_MAJOR                 3<br>
                   PETSC_VERSION_MINOR                 4<br>
                   PETSC_VERSION_SUBMINOR              4<br>
                   PETSC_VERSION_PATCH                 0<br>
                   PETSC_VERSION_DATE      unknown<br>
                   petsc_version_lt(3,3,0) is false<br>
                   nthreads =            16  NCASES =           100<br>
                  tpetsc_madams: malloc.c:3091: sYSMALLOc: Assertion
                  `(old_top == (((mbinptr) (((char *)
                  &((av)->bins[((1) - 1) * 2])) -
                  __builtin_offsetof (struct malloc_chunk, fd))))
                  && old_size == 0) || ((unsigned long)
                  (old_size) >= (unsigned long)((((__builtin_offsetof
                  (struct malloc_chunk, fd_nextsize))+((2 *
                  (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t)))
                  - 1))) && ((old_top)->size & 0x1)
                  && ((unsigned long)old_end & pagemask) ==
                  0)' failed.<br>
                  Error: abort<br>
                     rax 0000000000000000, rbx 0000000000000fff, rcx
                  ffffffffffffffff<br>
                     rdx 0000000000000006, rsp 00002aaac4554f78, rbp
                  00002aaad0000098<br>
                     rsi 0000000000005200, rdi 00000000000051f1, r8 
                  00000000ffffffff<br>
                     r9  00002aaaba8f9e40, r10 0000000000000008, r11
                  0000000000000202<br>
                     r12 0000000000000000, r13 00002aaad0008e30, r14
                  0000000000000020<br>
                     r15 0000000000000000<br>
                  ======= Backtrace: =========<br>
                  /lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
                  /lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
                  ./tpetsc_madams[0x4b7759]<br>
                    /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
                    /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
                    /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
                    /lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
                    /lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
                    /lib64/libc.so.6(__libc_malloc+0x77)
                  [0x2aaaba6045e7]<br>
                    ./tpetsc_madams() [0x4b76e3]<br>
                    ./tpetsc_madams() [0x4aa3d7]<br>
                    ./tpetsc_madams() [0x484888]<br>
                    ./tpetsc_madams() [0x475798]<br>
                    ./tpetsc_madams() [0x548aa5]<br>
                    ./tpetsc_madams() [0x5e5df2]<br>
                    ./tpetsc_madams() [0x57a353]<br>
                    ./tpetsc_madams() [0x4a701e]<br>
                    ./tpetsc_madams() [0x44e9be]<br>
                    ./tpetsc_madams() [0x44e1f4]<br>
                    /lib64/libc.so.6(__libc_start_main+0xe6)
                  [0x2aaaba5a8c36]<br>
                    ./tpetsc_madams() [0x44e0e9]<br>
                  Application 4895021 exit codes: 127<br>
                  Application 4895021 resources: utime ~0s, stime ~0s,
                  Rss ~12544, inblocks ~11429, outblocks ~35849
                  <div>
                    <div class="h5"><br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                      On 04/11/2014 04:34 PM, Mark Adams wrote:<br>
                    </div>
                  </div>
                </div>
                <div>
                  <div class="h5">
                    <blockquote type="cite">
                      <div dir="ltr">PETSc dev 'master' now has Barry's
                        thread safe stuff so you should be able to use
                        that.  I have build it in:
                        <div><br>
                        </div>
                        <div>
                          <div>PETSC_DIR=/autofs/na3_home1/adams/petsc</div>
                          <div>PETSC_ARCH=arch-titan-opt</div>
                        </div>
                        <div><br>
                        </div>
                        <div>So try this version out.  And revert the
                          code to the repo version by doing:</div>
                        <div><br>
                        </div>
                        <div>> git checkout poisson.F90</div>
                        <div><br>
                        </div>
                        <div>and any other place where #if
                          PETSC_VERSION_GE(3,5,0) is used.  I only see
                          poisson.F90.</div>
                        <div><br>
                        </div>
                        <div>If this works I can install it wherever you
                          like as Ed did.</div>
                        <div><br>
                        </div>
                        <div>Mark</div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                      </div>
                      <div class="gmail_extra"><br>
                        <br>
                        <div class="gmail_quote"> On Thu, Apr 10, 2014
                          at 11:24 PM, Seung-Hoe Ku <span dir="ltr"><<a
                              moz-do-not-send="true"
                              href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                          wrote:<br>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div>
                                <div>
                                  <div>Hi Mark,<br>
                                    <br>
                                    I think it is not a big problem
                                    until we use petsc 3.5 other than
                                    you. Now I disabled #if #endif for
                                    other users. Sorry for the
                                    inconvenience. Could you uncomment
                                    it when you are working with 3.5,
                                    and does not commit to core_dev,
                                    please?<br>
                                  </div>
                                  We need Ed's thread safe petsc for
                                  performance issue, so we can find some
                                  way to resolve it when 3.5 is released
                                  and installed on titan.<br>
                                  <br>
                                </div>
                                Thanks,<br>
                              </div>
                              Seung-Hoe<br>
                              <br>
                            </div>
                            <div>
                              <div>
                                <div class="gmail_extra"> <br>
                                  <br>
                                  <div class="gmail_quote">On Thu, Apr
                                    10, 2014 at 11:38 AM, Mark Adams <span
                                      dir="ltr"><<a
                                        moz-do-not-send="true"
                                        href="mailto:mfadams@lbl.gov"
                                        target="_blank">mfadams@lbl.gov</a>></span>
                                    wrote:<br>
                                    <blockquote class="gmail_quote"
                                      style="margin:0 0 0
                                      .8ex;border-left:1px #ccc
                                      solid;padding-left:1ex">
                                      <div dir="ltr">We've figured out
                                        the problem (again).  It will be
                                        fixed in future versions.  
                                        <div><br>
                                        </div>
                                        <div>We can fix your
                                          installation.  I'm guessing Ed
                                          did this installation so it
                                          might not be worth fixing this
                                          code since you are up and
                                          running.  Let me know what you
                                          want to do.  </div>
                                        <div><br>
                                        </div>
                                        <div>The problem is that is a
                                          development version of the
                                          code, which is not as stable
                                          as the releases.  This version
                                          needs to be updated to get fix
                                          this problem.  I have to tell
                                          you how to do this update so
                                          let me know if you want to do
                                          it.</div>
                                        <span><font color="#888888">
                                            <div><br>
                                            </div>
                                            <div>Mark</div>
                                            <div>
                                              <div>
                                                <div><br>
                                                </div>
                                              </div>
                                            </div>
                                          </font></span></div>
                                      <div>
                                        <div>
                                          <div class="gmail_extra"><br>
                                            <br>
                                            <div class="gmail_quote">On
                                              Thu, Apr 10, 2014 at 8:58
                                              AM, Seung-Hoe Ku <span
                                                dir="ltr"><<a
                                                  moz-do-not-send="true"
href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                              wrote:<br>
                                              <blockquote
                                                class="gmail_quote"
                                                style="margin:0 0 0
                                                .8ex;border-left:1px
                                                #ccc
                                                solid;padding-left:1ex">
                                                <div dir="ltr">
                                                  <div>
                                                    <div>One question
                                                      is..<br>
                                                    </div>
                                                    If PETSC_VERSION_GE
                                                    is defined with
                                                    PETSC_VERSION_GT and
                                                    PETSC_VERSION_GT is
                                                    defined with
                                                    something in
                                                    petsc.h,<br>
                                                  </div>
                                                  redefinition of
                                                  PETSC_VERSION_GT after
                                                  petsc.h will change
                                                  PETSC_VERSION_GE?<br>
                                                  <br>
                                                  <br>
                                                </div>
                                                <div>
                                                  <div>
                                                    <div
                                                      class="gmail_extra"><br>
                                                      <br>
                                                      <div
                                                        class="gmail_quote">On
                                                        Thu, Apr 10,
                                                        2014 at 10:55
                                                        AM, Seung-Hoe Ku
                                                        <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                                        wrote:<br>
                                                        <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">This
                                                          is the code I
                                                          tried to
                                                          compile.<br>
                                                          <div><br>
                                                          #if
                                                          PETSC_VERSION_GE(5,5,0)<br>
                                                           214  
                                                          BBBBBBBBBBBBBBBBBBB=0<br>
                                                           215   call
                                                          KSPSetOperators(solver%ksp, 
                                                          solver%Amat, 
                                                          solver%Amat,
                                                          ierr )<br>
                                                           216 #else<br>
                                                           217  
                                                          AAAAAAAAAAAA=0<br>
                                                           218   call
                                                          KSPSetOperators(solver%ksp, 
                                                          solver%Amat, 
                                                          solver%Amat,
                                                          SAME_NONZERO_PATT    
                                                          ERN, ierr )<br>
                                                           219 #endif<br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">
                                                          On Thu, Apr
                                                          10, 2014 at
                                                          10:49 AM, Mark
                                                          Adams <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr"><br>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">
                                                          <div>On Thu,
                                                          Apr 10, 2014
                                                          at 8:25 AM,
                                                          Seung-Hoe Ku <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>I have
                                                          the same
                                                          problem. I
                                                          used
                                                          poisson.F90 in
                                                          /lustre/atlas2/env003/scratch/shku/XGC1_3_petsc_problem/

                                                          <br>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          It seems that
                                                          #undef should
                                                          be after
                                                          #include<finclude/petsc.h>.
                                                          Otherwise,
                                                          petsc.h seems
                                                          to try
                                                          redefine it.<br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div>Oh yes.</div>
                                                          <div>
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div> Anyway,
                                                          I got the same
                                                          error message:
                                                          <div><br>
                                                          PGF90-S-0038-Symbol,

                                                          bbbbbbbbbbbbbbbbbbb,

                                                          has not been
                                                          explicitly
                                                          declared
                                                          (poisson.F90)<br>
                                                            0 inform,  
                                                          0 warnings,  
                                                          1 severes, 0
                                                          fatal for
                                                          init_1field_solver<br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div>is this
                                                          in the source
                                                          file? </div>
                                                          <div>
                                                          <div>
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div> </div>
                                                          </div>
                                                          I tried 5.5.0
                                                          for the
                                                          arguments of
                                                          PETSC_VERSION_GE,
                                                          but still have
                                                          the same
                                                          problem.<br>
                                                          <br>
                                                          </div>
                                                          Or do you mean
                                                          VERSION_GE and
                                                          VERSION_LT
                                                          instead of
                                                          VERSION_GT and
                                                          VERSION_LE?<br>
                                                          <br>
                                                          </div>
                                                          <div>Thanks,<br>
                                                          </div>
                                                          Seung-Hoe<br>
                                                          <br>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, Apr 9,
                                                          2014 at 9:54
                                                          PM, Mark Adams
                                                          <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">We

                                                          might have a
                                                          fix.  It turns
                                                          out that some
                                                          fortran
                                                          compilers to
                                                          not do the
                                                          #define quite
                                                          right. Try
                                                          this:
                                                          <div> <br>
                                                          </div>
                                                          <div>> git
                                                          checkout
                                                          poisson.F90</div>
                                                          <div><br>
                                                          </div>
                                                          <div>This will
                                                          out the
                                                          original (bad)
                                                          file back.
                                                          Then add to
                                                          poisson.F90</div>
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          <div>#undef
                                                          PETSC_VERSION_GT</div>
                                                          <div>
                                                          <div>#define
                                                          PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR)
                                                          \</div>
                                                          <div> 
                                                          (0==PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR))</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <div>This is
                                                          the fix that
                                                          Jed thinks
                                                          will work.  If
                                                          it works we
                                                          can propagate
                                                          it.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Sorry
                                                          about the
                                                          confusion,</div>
                                                          <div>Mark</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, Apr 9,
                                                          2014 at 7:36
                                                          PM, Seung-Hoe
                                                          Ku <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">Great!

                                                          Thank you.<br>
                                                          <br>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">
                                                          On Wed, Apr 9,
                                                          2014 at 9:35
                                                          PM, Mark Adams
                                                          <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">Good

                                                          news.  I'm at
                                                          a meeting and
                                                          Jed Brown is
                                                          here and he
                                                          seems to have
                                                          been able to
                                                          reproduce the
                                                          error.
                                                           Preprocessors
                                                          can be a pain.
                                                           I'm going to
                                                          wait until Jed
                                                          has a chance
                                                          to look at
                                                          this and come
                                                          up with a
                                                          solution.</div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, Apr 9,
                                                          2014 at 7:28
                                                          PM, Seung-Hoe
                                                          Ku <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>Now I am
                                                          using
                                                          iterative
                                                          solver which
                                                          will be
                                                          replaced to
                                                          your 2 field
                                                          solver later.<br>
                                                          Sorry. The
                                                          line number
                                                          3003 is wrong.
                                                          It is  209 or
                                                          near it.<br>
                                                          </div>
                                                          <div>Yes. We
                                                          can take a
                                                          look at this
                                                          when you come.
                                                          It is not
                                                          urgent
                                                          problem.<br>
                                                          <br>
                                                          </div>
                                                          <div>Thanks,<br>
                                                          Seung-Hoe<br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, Apr 9,
                                                          2014 at 9:25
                                                          PM, Mark Adams
                                                          <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">I
                                                          noticed that
                                                          your makefile
                                                          was not using
                                                          the new petsc
                                                          solver.  Is
                                                          that
                                                          intensional?
                                                          <div><br>
                                                          </div>
                                                          <div> The line
                                                          you gave me
                                                          3003
                                                          poission.F90
                                                          seems to be
                                                          the last line
                                                          of the file.</div>
                                                          <div><br>
                                                          </div>
                                                          <div> I am not
                                                          getting Edison
                                                          or Titan to
                                                          build.  We can
                                                          take a look at
                                                          this next
                                                          week.</div>
                                                          <span><font
                                                          color="#888888">
                                                          <div><br>
                                                          </div>
                                                          </font></span>
                                                          <div><span><font
color="#888888">Mark</font></span>
                                                          <div>
                                                          <div><br>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Tue, Apr 8,
                                                          2014 at 12:38
                                                          PM, Seung-Hoe
                                                          Ku <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Hi Mark,<br>
                                                          <br>
                                                          </div>
                                                          It seems that
                                                          PETSC_VERSION_GE

                                                          is not working
                                                          line 3003 of
                                                          poisson.F90.
                                                          Is there any
                                                          reason of
                                                          using
                                                          PETSC_VERSION_GE
                                                          instead of
                                                          PETSC_VERSION_LE?
                                                          The previous
                                                          code worked, I
                                                          think.<br>
                                                          <br>
                                                          </div>
                                                          Thanks,<br>
                                                          Seung-Hoe<br>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                      </div>
                                                      <br>
                                                    </div>
                                                  </div>
                                                </div>
                                              </blockquote>
                                            </div>
                                            <br>
                                          </div>
                                        </div>
                                      </div>
                                    </blockquote>
                                  </div>
                                  <br>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                        <br>
                      </div>
                    </blockquote>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>