<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Dear all,</p>
    <p>Thank you so much for the advice. <br>
    </p>
    <blockquote
cite="mid:CAJ98EDrHNDXFduQWTGDzfzi6BNpM92vLGLf6LFE39v_s52Xg4g@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div dir="ltr">
            <div class="gmail_extra">
              <div class="gmail_quote"><span class="gmail-"></span>
                <div>All setup is done in the first solve.</div>
                <span class="gmail-">
                  <div> </div>
                  <blockquote class="gmail_quote" style="margin:0px 0px
                    0px 0.8ex;border-left:1px solid
                    rgb(204,204,204);padding-left:1ex">
                    <div> ** The time for 1st solve does not scale. <br>
                          In practice, I am solving a variable
                      coefficient  Poisson equation. I need to build the
                      matrix every time step. Therefore, each step is
                      similar to the 1st solve which does not scale. Is
                      there a way I can improve the performance? <br>
                    </div>
                  </blockquote>
                </span></div>
            </div>
          </div>
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div dir="ltr">
                <div class="gmail_extra">
                  <div class="gmail_quote"><span class="gmail-">
                      <div><br>
                      </div>
                    </span>
                    <div>You could use rediscretization instead of
                      Galerkin to produce the coarse operators.</div>
                  </div>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>
              <div>Yes I can think of one option for improved
                performance, but I cannot tell whether it will be
                beneficial because the logging isn't sufficiently fine
                grained (and there is no easy way to get the info out of
                petsc). <br>
                <br>
                I use PtAP to repartition the matrix, this could be
                consuming most of the setup time in Telescope with your
                run. Such a repartitioning could be avoid if you
                provided a method to create the operator on the coarse
                levels (what Matt is suggesting). However, this requires
                you to be able to define your coefficients on the coarse
                grid. This will most likely reduce setup time, but your
                coarse grid operators (now re-discretized) are likely to
                be less effective than those generated via Galerkin
                coarsening.<br>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Please correct me if I understand this incorrectly:   I can define
    my own restriction function and pass it to petsc instead of using
    PtAP.<br>
    If so,what's the interface to do that?<br>
     <br>
    <blockquote
cite="mid:CAJ98EDrHNDXFduQWTGDzfzi6BNpM92vLGLf6LFE39v_s52Xg4g@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div dir="ltr">
                <div class="gmail_extra"><span class="gmail-"></span>
                  <div class="gmail_quote">
                    <div>Also, you use CG/MG when FMG by itself would
                      probably be faster. Your smoother is likely not
                      strong enough, and you</div>
                    <div>should use something like V(2,2). There is a
                      lot of tuning that is possible, but difficult to
                      automate.</div>
                  </div>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Matt's completely correct. <br>
              If we could automate this in a meaningful manner, we would
              have done so.<br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    I am not as familiar with multigrid as you guys. It would be very
    kind if you could be more specific.<br>
    What does V(2,2) stand for? Is there some strong smoother build in
    petsc that I can try?<br>
    <br>
    <br>
    Another thing, the vector assemble and scatter take more time as I
    increased the cores#:<br>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title></title>
    <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
    <style type="text/css">
                body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
                a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em;  } 
                a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em;  } 
                comment { display:none;  } 
        </style><br>
     cores#                                       4096            
    8192          16384         32768          65536  <br>
    VecAssemblyBegin       298        2.91E+00    2.87E+00   
    8.59E+00    2.75E+01    2.21E+03<br>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title></title>
    <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
    <style type="text/css">
                body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
                a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em;  } 
                a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em;  } 
                comment { display:none;  } 
        </style>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title></title>
    <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
    <style type="text/css">
                body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
                a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em;  } 
                a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em;  } 
                comment { display:none;  } 
        </style>VecAssemblyEnd          298        3.37E-03    1.78E-03   
    1.78E-03       5.13E-03    1.99E-03<br>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title></title>
    <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
    <style type="text/css">
                body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
                a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em;  } 
                a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em;  } 
                comment { display:non</style>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title></title>
    <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
    <style type="text/css">
                body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
                a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em;  } 
                a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em;  } 
                comment { display:none;  } 
        </style>VecScatterBegin           76303    3.82E+00    3.01E+00   
    2.54E+00    4.40E+00    1.32E+00<br>
    VecScatterEnd              76303    3.09E+01    1.47E+01   
    2.23E+01    2.96E+01    2.10E+01<br>
    <br>
    The above data is produced by solving a constant coefficients
    Possoin equation with different rhs for 100 steps. <br>
    As you can see, the time of VecAssemblyBegin increase dramatically
    from 32K cores to 65K.  <br>
    With 65K cores, it took more time to assemble the rhs than solving
    the equation.   Is there a way to improve this?<br>
    <br>
    <br>
    Thank you.<br>
    <br>
    Regards,<br>
    Frank  <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <br>
    <blockquote
cite="mid:CAJ98EDrHNDXFduQWTGDzfzi6BNpM92vLGLf6LFE39v_s52Xg4g@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div dir="ltr">
                <div class="gmail_extra">
                  <div class="gmail_quote">
                    <div>
                      <div class="gmail-h5">
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
                          0.8ex;border-left:1px solid
                          rgb(204,204,204);padding-left:1ex">
                          <div bgcolor="#FFFFFF">
                            <div>
                              <div
                                class="gmail-m_-6757504983836048137h5"><br>
                                <br>
                                <br>
                                <br>
                                <br>
                                <div
class="gmail-m_-6757504983836048137m_-3012109709631955293moz-cite-prefix">On
                                  10/04/2016 12:56 PM, Dave May wrote:<br>
                                </div>
                                <blockquote type="cite"><br>
                                  <br>
                                  On Tuesday, 4 October 2016, frank <<a
                                    moz-do-not-send="true"
                                    href="mailto:hengjiew@uci.edu"
                                    target="_blank">hengjiew@uci.edu</a>>
                                  wrote:<br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div bgcolor="#FFFFFF">
                                      <p>Hi,</p>
                                      This question is follow-up of the
                                      thread "Question about memory
                                      usage in Multigrid
                                      preconditioner".<br>
                                      I used to have the "Out of
                                      Memory(OOM)" problem when using
                                      the CG+Telescope MG solver with
                                      32768 cores. Adding the "-matrap
                                      0; -matptap_scalable" option did
                                      solve that problem. <br>
                                      <br>
                                      Then I test the scalability by
                                      solving a 3d poisson eqn for 1
                                      step. I used one sub-communicator
                                      in all the tests. The difference
                                      between the petsc options in those
                                      tests are: 1 the
                                      pc_telescope_reduction_factor; 2
                                      the number of multigrid levels in
                                      the up/down solver. The function
                                      "ksp_solve" is timed. It is kind
                                      of slow and doesn't scale at all.
                                      <br>
                                      <br>
                                      Test1: 512^3 grid points<br>
                                      Core#       
                                      telescope_reduction_factor    <wbr>   
                                      MG levels# for up/down solver    
                                      Time for KSPSolve (s)<br>
                                      512            
                                      8                             <wbr>                   
                                      4 / 3                             <wbr>                
                                      6.2466<br>
                                      4096          
                                      64                            <wbr>                  
                                      5 / 3                             <wbr>                
                                      0.9361<br>
                                      32768        
                                      64                            <wbr>                  
                                      4 / 3                             <wbr>                
                                      4.8914<br>
                                      <br>
                                      Test2: 1024^3 grid points<br>
                                      Core#       
                                      telescope_reduction_factor    <wbr>   
                                      MG levels# for up/down solver    
                                      Time for KSPSolve (s)<br>
                                      4096          
                                      64                            <wbr>                  
                                      5 / 4
                                                                    <wbr>              
                                      3.4139<br>
                                      8192          
                                      128                           <wbr>                 
                                      5 / 4                             <wbr>                
                                      2.4196<br>
                                      16384         32         
                                                                    <wbr>      
                                      5 / 3
                                                                    <wbr>              
                                      5.4150<br>
                                      32768        
                                      64                            <wbr>                  
                                      5 / 3                             <wbr>                
                                      5.6067<br>
                                      65536        
                                      128                           <wbr>                 
                                      5 / 3                             <wbr>                
                                      6.5219</div>
                                  </blockquote>
                                  <div><br>
                                  </div>
                                  <div>You have to be very careful how
                                    you interpret these numbers. Your
                                    solver contains nested calls to
                                    KSPSolve, and unfortunately as a
                                    result the numbers you report
                                    include setup time. This will remain
                                    true even if you call KSPSetUp on
                                    the outermost KSP. </div>
                                  <div><br>
                                  </div>
                                  <div>Your email concerns scalability
                                    of the silver application, so let's
                                    focus on that issue.</div>
                                  <div><br>
                                  </div>
                                  <div>The only way to clearly separate
                                    setup from solve time is to perform
                                    two identical solves. The second
                                    solve will not require any setup.
                                    You should monitor the second solve
                                    via a new PetscStage.</div>
                                  <div><br>
                                  </div>
                                  <div>This was what I did in the
                                    telescope paper. It was the only way
                                    to understand the setup cost (and
                                    scaling) cf the solve time (and
                                    scaling).</div>
                                  <div><br>
                                  </div>
                                  <div>Thanks</div>
                                  <div>  Dave</div>
                                  <div>
                                    <div>
                                      <div><br>
                                      </div>
                                      <div> </div>
                                      <blockquote class="gmail_quote"
                                        style="margin:0px 0px 0px
                                        0.8ex;border-left:1px solid
                                        rgb(204,204,204);padding-left:1ex">
                                        <div bgcolor="#FFFFFF"> I guess
                                          I didn't set the MG levels
                                          properly. What would be the
                                          efficient way to arrange the
                                          MG levels?<br>
                                          Also which preconditionr at
                                          the coarse mesh of the 2nd
                                          communicator should I use to
                                          improve the performance? <br>
                                          <br>
                                          I attached the test code and
                                          the petsc options file for the
                                          1024^3 cube with 32768 cores.
                                          <br>
                                          <br>
                                          Thank you.<br>
                                          <br>
                                          Regards,<br>
                                          Frank<br>
                                          <br>
                                          <br>
                                          <br>
                                          <br>
                                          <br>
                                          <br>
                                          <div>On 09/15/2016 03:35 AM,
                                            Dave May wrote:<br>
                                          </div>
                                          <blockquote type="cite">
                                            <div dir="ltr">
                                              <div>
                                                <div>
                                                  <div>
                                                    <div>
                                                      <div>HI all,<br>
                                                        <br>
                                                      </div>
                                                      <div>I the only
                                                        unexpected
                                                        memory usage I
                                                        can see is
                                                        associated with
                                                        the call to
                                                        MatPtAP().<br>
                                                      </div>
                                                      <div>Here is
                                                        something you
                                                        can try
                                                        immediately.<br>
                                                      </div>
                                                    </div>
                                                    Run your code with
                                                    the additional
                                                    options<br>
                                                      -matrap 0
                                                    -matptap_scalable<br>
                                                    <br>
                                                  </div>
                                                  <div>I didn't realize
                                                    this before, but the
                                                    default behaviour of
                                                    MatPtAP in parallel
                                                    is actually to to
                                                    explicitly form the
                                                    transpose of P (e.g.
                                                    assemble R = P^T)
                                                    and then compute
                                                    R.A.P. <br>
                                                    You don't want to do
                                                    this. The option
                                                    -matrap 0 resolves
                                                    this issue.<br>
                                                  </div>
                                                  <div><br>
                                                  </div>
                                                  <div>The
                                                    implementation of
                                                    P^T.A.P has two
                                                    variants. <br>
                                                    The scalable
                                                    implementation (with
                                                    respect to memory
                                                    usage) is selected
                                                    via the second
                                                    option
                                                    -matptap_scalable.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Try it out - I
                                                    see a significant
                                                    memory reduction
                                                    using these options
                                                    for particular mesh
                                                    sizes / partitions.<br>
                                                  </div>
                                                  <div><br>
                                                  </div>
                                                  I've attached a
                                                  cleaned up version of
                                                  the code you sent me.<br>
                                                </div>
                                                There were a number of
                                                memory leaks and other
                                                issues.<br>
                                              </div>
                                              <div>The main points being<br>
                                              </div>
                                                * You should call
                                              DMDAVecGetArrayF90()
                                              before
                                              VecAssembly{Begin,End}<br>
                                                * You should call
                                              PetscFinalize(), otherwise
                                              the option -log_summary
                                              (-log_view) will not
                                              display anything once the
                                              program has completed.<br>
                                              <div>
                                                <div>
                                                  <div><br>
                                                    <br>
                                                  </div>
                                                  <div>Thanks,<br>
                                                  </div>
                                                  <div>  Dave<br>
                                                  </div>
                                                  <div>
                                                    <div>
                                                      <div><br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                            <div class="gmail_extra"><br>
                                              <div class="gmail_quote">On
                                                15 September 2016 at
                                                08:03, Hengjie Wang <span
                                                  dir="ltr"><<a
                                                    moz-do-not-send="true">hengjiew@uci.edu</a>></span>
                                                wrote:<br>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0px 0px
                                                  0px
                                                  0.8ex;border-left:1px
                                                  solid
                                                  rgb(204,204,204);padding-left:1ex">
                                                  <div bgcolor="#FFFFFF">
                                                    Hi Dave,<br>
                                                    <br>
                                                    Sorry, I should have
                                                    put more comment to
                                                    explain the code.  <br>
                                                    The number of
                                                    process in each
                                                    dimension is the
                                                    same: Px = Py=Pz=P.
                                                    So is the domain
                                                    size.<br>
                                                    So if the you want
                                                    to run the code for
                                                    a  512^3 grid points
                                                    on 16^3 cores, you
                                                    need to set "-N 512
                                                    -P 16" in the
                                                    command line.<br>
                                                    I add more comments
                                                    and also fix an
                                                    error in the
                                                    attached code. ( The
                                                    error only effects
                                                    the accuracy of
                                                    solution but not the
                                                    memory usage. ) <br>
                                                    <div><br>
                                                      Thank you.<span><font
color="#888888"><br>
                                                          Frank</font></span>
                                                      <div>
                                                        <div><br>
                                                          <br>
                                                          On 9/14/2016
                                                          9:05 PM, Dave
                                                          May wrote:<br>
                                                        </div>
                                                      </div>
                                                    </div>
                                                    <div>
                                                      <div>
                                                        <blockquote
                                                          type="cite"><br>
                                                          <br>
                                                          On Thursday,
                                                          15 September
                                                          2016, Dave May
                                                          <<a
                                                          moz-do-not-send="true">dave.mayhem23@gmail.com</a>>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex"><br>
                                                          <br>
                                                          On Thursday,
                                                          15 September
                                                          2016, frank
                                                          <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF">
                                                          Hi, <br>
                                                          <br>
                                                          I write a
                                                          simple code to
                                                          re-produce the
                                                          error. I hope
                                                          this can help
                                                          to diagnose
                                                          the problem.<br>
                                                          The code just
                                                          solves a 3d
                                                          poisson
                                                          equation. </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Why is
                                                          the stencil
                                                          width a
                                                          runtime
                                                          parameter??
                                                          And why is the
                                                          default value
                                                          2? For 7-pnt
                                                          FD Laplace,
                                                          you only need
                                                          a stencil
                                                          width of 1. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Was this
                                                          choice made to
                                                          mimic
                                                          something in
                                                          the
                                                          real application
                                                          code?</div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          Please ignore
                                                          - I
                                                          misunderstood your
                                                          usage of the
                                                          param set by
                                                          -P
                                                          <div>
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"><br>
                                                          I run the code
                                                          on a 1024^3
                                                          mesh. The
                                                          process
                                                          partition is
                                                          32 * 32 * 32.
                                                          That's when I
                                                          re-produce the
                                                          OOM error.
                                                          Each core has
                                                          about 2G
                                                          memory.<br>
                                                          I also run the
                                                          code on a
                                                          512^3 mesh
                                                          with 16 * 16 *
                                                          16 processes.
                                                          The ksp solver
                                                          works fine. <br>
                                                          I attached the
                                                          code,
                                                          ksp_view_pre's
                                                          output and my
                                                          petsc option
                                                          file.<br>
                                                          <br>
                                                          Thank you.<br>
                                                          Frank<br>
                                                          <div><br>
                                                          On 09/09/2016
                                                          06:38 PM,
                                                          Hengjie Wang
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">Hi
                                                          Barry, 
                                                          <div><br>
                                                          </div>
                                                          <div>I
                                                          checked. On
                                                          the
                                                          supercomputer,
                                                          I had the
                                                          option
                                                          "-ksp_view_pre"
                                                          but it is not
                                                          in file I sent
                                                          you. I am
                                                          sorry for the
                                                          confusion.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div>Frank<span></span><br>
                                                          <br>
                                                          On Friday,
                                                          September 9,
                                                          2016, Barry
                                                          Smith <<a
                                                          moz-do-not-send="true">bsmith@mcs.anl.gov</a>>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex"><br>
                                                          > On Sep 9,
                                                          2016, at 3:11
                                                          PM, frank <<a
moz-do-not-send="true">hengjiew@uci.edu</a>> wrote:<br>
                                                          ><br>
                                                          > Hi Barry,<br>
                                                          ><br>
                                                          > I think
                                                          the first KSP
                                                          view output is
                                                          from
                                                          -ksp_view_pre.
                                                          Before I
                                                          submitted the
                                                          test, I was
                                                          not sure
                                                          whether there
                                                          would be OOM
                                                          error or not.
                                                          So I added
                                                          both
                                                          -ksp_view_pre
                                                          and -ksp_view.<br>
                                                          <br>
                                                            But the
                                                          options file
                                                          you sent
                                                          specifically
                                                          does NOT list
                                                          the
                                                          -ksp_view_pre
                                                          so how could
                                                          it be from
                                                          that?<br>
                                                          <br>
                                                             Sorry to be
                                                          pedantic but
                                                          I've spent too
                                                          much time in
                                                          the past
                                                          trying to
                                                          debug from
                                                          incorrect
                                                          information
                                                          and want to
                                                          make sure that
                                                          the
                                                          information I
                                                          have is
                                                          correct before
                                                          thinking.
                                                          Please recheck
                                                          exactly what
                                                          happened.
                                                          Rerun with the
                                                          exact input
                                                          file you
                                                          emailed if
                                                          that is
                                                          needed.<br>
                                                          <br>
                                                             Barry<br>
                                                          <br>
                                                          ><br>
                                                          > Frank<br>
                                                          ><br>
                                                          ><br>
                                                          > On
                                                          09/09/2016
                                                          12:38 PM,
                                                          Barry Smith
                                                          wrote:<br>
                                                          >>   Why
                                                          does
                                                          ksp_view2.txt
                                                          have two KSP
                                                          views in it
                                                          while
                                                          ksp_view1.txt
                                                          has only one
                                                          KSPView in it?
                                                          Did you run
                                                          two different
                                                          solves in the
                                                          2 case but not
                                                          the one?<br>
                                                          >><br>
                                                          >> 
                                                           Barry<br>
                                                          >><br>
                                                          >><br>
                                                          >><br>
                                                          >>>
                                                          On Sep 9,
                                                          2016, at 10:56
                                                          AM, frank <<a
moz-do-not-send="true">hengjiew@uci.edu</a>> wrote:<br>
                                                          >>><br>
                                                          >>>
                                                          Hi,<br>
                                                          >>><br>
                                                          >>> I
                                                          want to
                                                          continue
                                                          digging into
                                                          the memory
                                                          problem here.<br>
                                                          >>> I
                                                          did find a
                                                          work around in
                                                          the past,
                                                          which is to
                                                          use less cores
                                                          per node so
                                                          that each core
                                                          has 8G memory.
                                                          However this
                                                          is deficient
                                                          and expensive.
                                                          I hope to
                                                          locate the
                                                          place that
                                                          uses the most
                                                          memory.<br>
                                                          >>><br>
                                                          >>>
                                                          Here is a
                                                          brief summary
                                                          of the tests I
                                                          did in past:<br>
>>>> Test1:   Mesh 1536*128*384  |  Process Mesh 48*4*12<br>
                                                          >>>
                                                          Maximum (over
                                                          computational
                                                          time) process
                                                          memory:       
                                                             total
                                                          7.0727e+08<br>
                                                          >>>
                                                          Current
                                                          process
                                                          memory:       
                                                                       
                                                                       
                                                                       
                                                                 total
                                                          7.0727e+08<br>
                                                          >>>
                                                          Maximum (over
                                                          computational
                                                          time) space
                                                          PetscMalloc()ed: 
                                                          total
                                                          6.3908e+11<br>
                                                          >>>
                                                          Current space
PetscMalloc()ed:                                                total
                                                          1.8275e+09<br>
                                                          >>><br>
>>>> Test2:    Mesh 1536*128*384  |  Process Mesh 96*8*24<br>
                                                          >>>
                                                          Maximum (over
                                                          computational
                                                          time) process
                                                          memory:       
                                                             total
                                                          5.9431e+09<br>
                                                          >>>
                                                          Current
                                                          process
                                                          memory:       
                                                                       
                                                                       
                                                                       
                                                                 total
                                                          5.9431e+09<br>
                                                          >>>
                                                          Maximum (over
                                                          computational
                                                          time) space
                                                          PetscMalloc()ed: 
                                                          total
                                                          5.3202e+12<br>
                                                          >>>
                                                          Current space
PetscMalloc()ed:                                                 total
                                                          5.4844e+09<br>
                                                          >>><br>
>>>> Test3:    Mesh 3072*256*768  |  Process Mesh 96*8*24<br>
                                                          >>> 
                                                             OOM( Out Of
                                                          Memory )
                                                          killer of the
                                                          supercomputer
                                                          terminated the
                                                          job during
                                                          "KSPSolve".<br>
                                                          >>><br>
                                                          >>> I
                                                          attached the
                                                          output of
                                                          ksp_view( the
                                                          third test's
                                                          output is from
                                                          ksp_view_pre
                                                          ), memory_view
                                                          and also the
                                                          petsc options.<br>
                                                          >>><br>
                                                          >>>
                                                          In all the
                                                          tests, each
                                                          core can
                                                          access about
                                                          2G memory. In
                                                          test3, there
                                                          are 4223139840
                                                          non-zeros in
                                                          the matrix.
                                                          This will
                                                          consume about
                                                          1.74M, using
                                                          double
                                                          precision.
                                                          Considering
                                                          some extra
                                                          memory used to
                                                          store integer
                                                          index, 2G
                                                          memory should
                                                          still be way
                                                          enough.<br>
                                                          >>><br>
                                                          >>>
                                                          Is there a way
                                                          to find out
                                                          which part of
                                                          KSPSolve uses
                                                          the most
                                                          memory?<br>
                                                          >>>
                                                          Thank you so
                                                          much.<br>
                                                          >>><br>
                                                          >>>
                                                          BTW, there are
                                                          4 options
                                                          remains unused
                                                          and I don't
                                                          understand why
                                                          they are
                                                          omitted:<br>
                                                          >>>
                                                          -mg_coarse_telescope_mg_coarse<wbr>_ksp_type
                                                          value: preonly<br>
                                                          >>>
                                                          -mg_coarse_telescope_mg_coarse<wbr>_pc_type
                                                          value: bjacobi<br>
                                                          >>>
                                                          -mg_coarse_telescope_mg_levels<wbr>_ksp_max_it
                                                          value: 1<br>
                                                          >>>
                                                          -mg_coarse_telescope_mg_levels<wbr>_ksp_type
                                                          value:
                                                          richardson<br>
                                                          >>><br>
                                                          >>><br>
                                                          >>>
                                                          Regards,<br>
                                                          >>>
                                                          Frank<br>
                                                          >>><br>
                                                          >>>
                                                          On 07/13/2016
                                                          05:47 PM, Dave
                                                          May wrote:<br>
>>>><br>
>>>> On 14 July 2016 at 01:07, frank <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
>>>> Hi Dave,<br>
>>>><br>
>>>> Sorry for the late reply.<br>
>>>> Thank you so much for your detailed reply.<br>
>>>><br>
>>>> I have a question about the estimation of the memory
                                                          usage. There
                                                          are 4223139840
                                                          allocated
                                                          non-zeros and
                                                          18432 MPI
                                                          processes.
                                                          Double
                                                          precision is
                                                          used. So the
                                                          memory per
                                                          process is:<br>
>>>>   4223139840 * 8bytes / 18432 / 1024 / 1024 = 1.74M ?<br>
>>>> Did I do sth wrong here? Because this seems too small.<br>
>>>><br>
>>>> No - I totally f***ed it up. You are correct. That'll
                                                          teach me for
                                                          fumbling
                                                          around with my
                                                          iphone
                                                          calculator and
                                                          not using my
                                                          brain. (Note
                                                          that to
                                                          convert to MB
                                                          just divide by
                                                          1e6, not
                                                          1024^2 -
                                                          although I
                                                          apparently
                                                          cannot convert
                                                          between units
                                                          correctly....)<br>
>>>><br>
>>>> From the PETSc objects associated with the solver, It
                                                          looks like it
                                                          _should_ run
                                                          with 2GB per
                                                          MPI rank.
                                                          Sorry for my
                                                          mistake.
                                                          Possibilities
                                                          are: somewhere
                                                          in your usage
                                                          of PETSc
                                                          you've
                                                          introduced a
                                                          memory leak;
                                                          PETSc is doing
                                                          a huge over
                                                          allocation
                                                          (e.g. as per
                                                          our discussion
                                                          of MatPtAP);
                                                          or in your
                                                          application
                                                          code there are
                                                          other objects
                                                          you have
                                                          forgotten to
                                                          log the memory
                                                          for.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> I am running this job on Bluewater<br>
>>>> I am using the 7 points FD stencil in 3D.<br>
>>>><br>
>>>> I thought so on both counts.<br>
>>>><br>
>>>> I apologize that I made a stupid mistake in computing
                                                          the memory per
                                                          core. My
                                                          settings
                                                          render each
                                                          core can
                                                          access only 2G
                                                          memory on
                                                          average
                                                          instead of 8G
                                                          which I
                                                          mentioned in
                                                          previous
                                                          email. I
                                                          re-run the job
                                                          with 8G memory
                                                          per core on
                                                          average and
                                                          there is no
                                                          "Out Of
                                                          Memory" error.
                                                          I would do
                                                          more test to
                                                          see if there
                                                          is still some
                                                          memory issue.<br>
>>>><br>
>>>> Ok. I'd still like to know where the memory was being
                                                          used since my
                                                          estimates were
                                                          off.<br>
>>>><br>
>>>><br>
>>>> Thanks,<br>
>>>>   Dave<br>
>>>><br>
>>>> Regards,<br>
>>>> Frank<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On 07/11/2016 01:18 PM, Dave May wrote:<br>
>>>>> Hi Frank,<br>
>>>>><br>
>>>>><br>
>>>>> On 11 July 2016 at 19:14, frank <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
>>>>> Hi Dave,<br>
>>>>><br>
>>>>> I re-run the test using bjacobi as the
                                                          preconditioner
                                                          on the coarse
                                                          mesh of
                                                          telescope. The
                                                          Grid is
                                                          3072*256*768
                                                          and process
                                                          mesh is
                                                          96*8*24. The
                                                          petsc option
                                                          file is
                                                          attached.<br>
>>>>> I still got the "Out Of Memory" error. The error
                                                          occurred
                                                          before the
                                                          linear solver
                                                          finished one
                                                          step. So I
                                                          don't have the
                                                          full info from
                                                          ksp_view. The
                                                          info from
                                                          ksp_view_pre
                                                          is attached.<br>
>>>>><br>
>>>>> Okay - that is essentially useless (sorry)<br>
>>>>><br>
>>>>> It seems to me that the error occurred when the
                                                          decomposition
                                                          was going to
                                                          be changed.<br>
>>>>><br>
>>>>> Based on what information?<br>
>>>>> Running with -info would give us more clues, but
                                                          will create a
                                                          ton of output.<br>
>>>>> Please try running the case which failed with -info<br>
>>>>>  I had another test with a grid of 1536*128*384 and
                                                          the same
                                                          process mesh
                                                          as above.
                                                          There was no
                                                          error. The
                                                          ksp_view info
                                                          is attached
                                                          for
                                                          comparison.<br>
>>>>> Thank you.<br>
>>>>><br>
>>>>><br>
>>>>> [3] Here is my crude estimate of your memory usage.<br>
>>>>> I'll target the biggest memory hogs only to get an
                                                          order of
                                                          magnitude
                                                          estimate<br>
>>>>><br>
>>>>> * The Fine grid operator contains 4223139840
                                                          non-zeros
                                                          --> 1.8 GB
                                                          per MPI rank
                                                          assuming
                                                          double
                                                          precision.<br>
>>>>> The indices for the AIJ could amount to another 0.3
                                                          GB (assuming
                                                          32 bit
                                                          integers)<br>
>>>>><br>
>>>>> * You use 5 levels of coarsening, so the other
                                                          operators
                                                          should
                                                          represent
                                                          (collectively)<br>
>>>>> 2.1 / 8 + 2.1/8^2 + 2.1/8^3 + 2.1/8^4  ~ 300 MB per
                                                          MPI rank on
                                                          the
                                                          communicator
                                                          with 18432
                                                          ranks.<br>
>>>>> The coarse grid should consume ~ 0.5 MB per MPI
                                                          rank on the
                                                          communicator
                                                          with 18432
                                                          ranks.<br>
>>>>><br>
>>>>> * You use a reduction factor of 64, making the new
                                                          communicator
                                                          with 288 MPI
                                                          ranks.<br>
>>>>> PCTelescope will first gather a temporary matrix
                                                          associated
                                                          with your
                                                          coarse level
                                                          operator
                                                          assuming a
                                                          comm size of
                                                          288 living on
                                                          the comm with
                                                          size 18432.<br>
>>>>> This matrix will require approximately 0.5 * 64 =
                                                          32 MB per core
                                                          on the 288
                                                          ranks.<br>
>>>>> This matrix is then used to form a new MPIAIJ
                                                          matrix on the
                                                          subcomm, thus
                                                          require
                                                          another 32 MB
                                                          per rank.<br>
>>>>> The temporary matrix is now destroyed.<br>
>>>>><br>
>>>>> * Because a DMDA is detected, a permutation matrix
                                                          is assembled.<br>
>>>>> This requires 2 doubles per point in the DMDA.<br>
>>>>> Your coarse DMDA contains 92 x 16 x 48 points.<br>
>>>>> Thus the permutation matrix will require < 1 MB
                                                          per MPI rank
                                                          on the
                                                          sub-comm.<br>
>>>>><br>
>>>>> * Lastly, the matrix is permuted. This uses
                                                          MatPtAP(), but
                                                          the resulting
                                                          operator will
                                                          have the same
                                                          memory
                                                          footprint as
                                                          the unpermuted
                                                          matrix (32
                                                          MB). At any
                                                          stage in
                                                          PCTelescope,
                                                          only 2
                                                          operators of
                                                          size 32 MB are
                                                          held in memory
                                                          when the DMDA
                                                          is provided.<br>
>>>>><br>
>>>>> From my rough estimates, the worst case memory foot
                                                          print for any
                                                          given core,
                                                          given your
                                                          options is
                                                          approximately<br>
>>>>> 2100 MB + 300 MB + 32 MB + 32 MB + 1 MB  = 2465 MB<br>
>>>>> This is way below 8 GB.<br>
>>>>><br>
>>>>> Note this estimate completely ignores:<br>
>>>>> (1) the memory required for the restriction
                                                          operator,<br>
>>>>> (2) the potential growth in the number of non-zeros
                                                          per row due to
                                                          Galerkin
                                                          coarsening (I
                                                          wished
                                                          -ksp_view_pre
                                                          reported the
                                                          output from
                                                          MatView so we
                                                          could see the
                                                          number of
                                                          non-zeros
                                                          required by
                                                          the coarse
                                                          level
                                                          operators)<br>
>>>>> (3) all temporary vectors required by the CG
                                                          solver, and
                                                          those required
                                                          by the
                                                          smoothers.<br>
>>>>> (4) internal memory allocated by MatPtAP<br>
>>>>> (5) memory associated with IS's used within
                                                          PCTelescope<br>
>>>>><br>
>>>>> So either I am completely off in my estimates, or
                                                          you have not
                                                          carefully
                                                          estimated the
                                                          memory usage
                                                          of your
                                                          application
                                                          code.
                                                          Hopefully
                                                          others might
                                                          examine/correct
                                                          my rough
                                                          estimates<br>
>>>>><br>
>>>>> Since I don't have your code I cannot access the
                                                          latter.<br>
>>>>> Since I don't have access to the same machine you
                                                          are running
                                                          on, I think we
                                                          need to take a
                                                          step back.<br>
>>>>><br>
>>>>> [1] What machine are you running on? Send me a URL
                                                          if its
                                                          available<br>
>>>>><br>
>>>>> [2] What discretization are you using? (I am
                                                          guessing a
                                                          scalar 7 point
                                                          FD stencil)<br>
>>>>> If it's a 7 point FD stencil, we should be able to
                                                          examine the
                                                          memory usage
                                                          of your solver
                                                          configuration
                                                          using a
                                                          standard,
                                                          light weight
                                                          existing PETSc
                                                          example, run
                                                          on your
                                                          machine at the
                                                          same scale.<br>
>>>>> This would hopefully enable us to correctly
                                                          evaluate the
                                                          actual memory
                                                          usage required
                                                          by the solver
                                                          configuration
                                                          you are using.<br>
>>>>><br>
>>>>> Thanks,<br>
>>>>>   Dave<br>
>>>>><br>
>>>>><br>
>>>>> Frank<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On 07/08/2016 10:38 PM, Dave May wrote:<br>
>>>>>><br>
>>>>>> On Saturday, 9 July 2016, frank <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
>>>>>> Hi Barry and Dave,<br>
>>>>>><br>
>>>>>> Thank both of you for the advice.<br>
>>>>>><br>
>>>>>> @Barry<br>
>>>>>> I made a mistake in the file names in last
                                                          email. I
                                                          attached the
                                                          correct files
                                                          this time.<br>
>>>>>> For all the three tests, 'Telescope' is used as
                                                          the coarse
                                                          preconditioner.<br>
>>>>>><br>
>>>>>> == Test1:   Grid: 1536*128*384,   Process Mesh:
                                                          48*4*12<br>
>>>>>> Part of the memory usage:  Vector   125       
                                                              124
                                                          3971904     0.<br>
>>>>>>                                             
                                                          Matrix   101
                                                          101     
                                                          9462372     0<br>
>>>>>><br>
>>>>>> == Test2: Grid: 1536*128*384,   Process Mesh:
                                                          96*8*24<br>
>>>>>> Part of the memory usage:  Vector   125       
                                                              124
                                                          681672     0.<br>
>>>>>>                                             
                                                          Matrix   101
                                                          101     
                                                          1462180     0.<br>
>>>>>><br>
>>>>>> In theory, the memory usage in Test1 should be
                                                          8 times of
                                                          Test2. In my
                                                          case, it is
                                                          about 6 times.<br>
>>>>>><br>
>>>>>> == Test3: Grid: 3072*256*768,   Process Mesh:
                                                          96*8*24.
                                                          Sub-domain per
                                                          process:
                                                          32*32*32<br>
>>>>>> Here I get the out of memory error.<br>
>>>>>><br>
>>>>>> I tried to use -mg_coarse jacobi. In this way,
                                                          I don't need
                                                          to set
                                                          -mg_coarse_ksp_type
                                                          and
                                                          -mg_coarse_pc_type
                                                          explicitly,
                                                          right?<br>
>>>>>> The linear solver didn't work in this case.
                                                          Petsc output
                                                          some errors.<br>
>>>>>><br>
>>>>>> @Dave<br>
>>>>>> In test3, I use only one instance of
                                                          'Telescope'.
                                                          On the coarse
                                                          mesh of
                                                          'Telescope', I
                                                          used LU as the
                                                          preconditioner
                                                          instead of
                                                          SVD.<br>
>>>>>> If my set the levels correctly, then on the
                                                          last coarse
                                                          mesh of MG
                                                          where it calls
                                                          'Telescope',
                                                          the sub-domain
                                                          per process is
                                                          2*2*2.<br>
>>>>>> On the last coarse mesh of 'Telescope', there
                                                          is only one
                                                          grid point per
                                                          process.<br>
>>>>>> I still got the OOM error. The detailed petsc
                                                          option file is
                                                          attached.<br>
>>>>>><br>
>>>>>> Do you understand the expected memory usage for
                                                          the particular
                                                          parallel LU
                                                          implementation
                                                          you are using?
                                                          I don't
                                                          (seriously).
                                                          Replace LU
                                                          with bjacobi
                                                          and re-run
                                                          this test. My
                                                          point about
                                                          solver
                                                          debugging is
                                                          still valid.<br>
>>>>>><br>
>>>>>> And please send the result of KSPView so we can
                                                          see what is
                                                          actually used
                                                          in the
                                                          computations<br>
>>>>>><br>
>>>>>> Thanks<br>
>>>>>>   Dave<br>
>>>>>><br>
>>>>>><br>
>>>>>> Thank you so much.<br>
>>>>>><br>
>>>>>> Frank<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On 07/06/2016 02:51 PM, Barry Smith wrote:<br>
>>>>>> On Jul 6, 2016, at 4:19 PM, frank <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
>>>>>><br>
>>>>>> Hi Barry,<br>
>>>>>><br>
>>>>>> Thank you for you advice.<br>
>>>>>> I tried three test. In the 1st test, the grid
                                                          is
                                                          3072*256*768
                                                          and the
                                                          process mesh
                                                          is 96*8*24.<br>
>>>>>> The linear solver is 'cg' the preconditioner is
                                                          'mg' and
                                                          'telescope' is
                                                          used as the
                                                          preconditioner
                                                          at the coarse
                                                          mesh.<br>
>>>>>> The system gives me the "Out of Memory" error
                                                          before the
                                                          linear system
                                                          is completely
                                                          solved.<br>
>>>>>> The info from '-ksp_view_pre' is attached. I
                                                          seems to me
                                                          that the error
                                                          occurs when it
                                                          reaches the
                                                          coarse mesh.<br>
>>>>>><br>
>>>>>> The 2nd test uses a grid of 1536*128*384 and
                                                          process mesh
                                                          is 96*8*24.
                                                          The 3rd       
                                                                       
                                                                       
                                                                   test
                                                          uses the same
                                                          grid but a
                                                          different
                                                          process mesh
                                                          48*4*12.<br>
>>>>>>     Are you sure this is right? The total
                                                          matrix and
                                                          vector memory
                                                          usage goes
                                                          from 2nd test<br>
>>>>>>                Vector   384            383     
                                                          8,193,712   
                                                           0.<br>
>>>>>>                Matrix   103            103   
                                                           11,508,688   
                                                           0.<br>
>>>>>> to 3rd test<br>
>>>>>>               Vector   384            383     
                                                          1,590,520   
                                                           0.<br>
>>>>>>                Matrix   103            103     
                                                          3,508,664   
                                                           0.<br>
>>>>>> that is the memory usage got smaller but if you
                                                          have only
                                                          1/8th the
                                                          processes and
                                                          the same grid
                                                          it should have
                                                          gotten about 8
                                                          times bigger.
                                                          Did you maybe
                                                          cut the grid
                                                          by a factor of
                                                          8 also? If so
                                                          that still
                                                          doesn't
                                                          explain it
                                                          because the
                                                          memory usage
                                                          changed by a
                                                          factor of 5
                                                          something for
                                                          the vectors
                                                          and 3
                                                          something for
                                                          the matrices.<br>
>>>>>><br>
>>>>>><br>
>>>>>> The linear solver and petsc options in 2nd and
                                                          3rd tests are
                                                          the same in
                                                          1st test. The
                                                          linear solver
                                                          works fine in
                                                          both test.<br>
>>>>>> I attached the memory usage of the 2nd and 3rd
                                                          tests. The
                                                          memory info is
                                                          from the
                                                          option
                                                          '-log_summary'.
                                                          I tried to use
                                                          '-momery_info'
                                                          as you
                                                          suggested, but
                                                          in my case
                                                          petsc treated
                                                          it as an
                                                          unused option.
                                                          It output
                                                          nothing about
                                                          the memory. Do
                                                          I need to add
                                                          sth to my code
                                                          so I can use
                                                          '-memory_info'?<br>
>>>>>>     Sorry, my mistake the option is
                                                          -memory_view<br>
>>>>>><br>
>>>>>>    Can you run the one case with -memory_view
                                                          and -mg_coarse
                                                          jacobi
                                                          -ksp_max_it 1
                                                          (just so it
                                                          doesn't
                                                          iterate
                                                          forever) to
                                                          see how much
                                                          memory is used
                                                          without the
                                                          telescope?
                                                          Also run case
                                                          2 the same
                                                          way.<br>
>>>>>><br>
>>>>>>    Barry<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> In both tests the memory usage is not large.<br>
>>>>>><br>
>>>>>> It seems to me that it might be the
                                                          'telescope' 
                                                          preconditioner
                                                          that allocated
                                                          a lot of
                                                          memory and
                                                          caused the
                                                          error in the
                                                          1st test.<br>
>>>>>> Is there is a way to show how much memory it
                                                          allocated?<br>
>>>>>><br>
>>>>>> Frank<br>
>>>>>><br>
>>>>>> On 07/05/2016 03:37 PM, Barry Smith wrote:<br>
>>>>>>    Frank,<br>
>>>>>><br>
>>>>>>      You can run with -ksp_view_pre to have it
                                                          "view" the KSP
                                                          before the
                                                          solve so
                                                          hopefully it
                                                          gets that far.<br>
>>>>>><br>
>>>>>>       Please run the problem that does fit with
                                                          -memory_info
                                                          when the
                                                          problem
                                                          completes it
                                                          will show the
                                                          "high water
                                                          mark" for
                                                          PETSc
                                                          allocated
                                                          memory and
                                                          total memory
                                                          used. We first
                                                          want to look
                                                          at these
                                                          numbers to see
                                                          if it is using
                                                          more memory
                                                          than you
                                                          expect. You
                                                          could also run
                                                          with say half
                                                          the grid
                                                          spacing to see
                                                          how the memory
                                                          usage scaled
                                                          with the
                                                          increase in
                                                          grid points.
                                                          Make the runs
                                                          also with
                                                          -log_view and
                                                          send all the
                                                          output from
                                                          these options.<br>
>>>>>><br>
>>>>>>     Barry<br>
>>>>>><br>
>>>>>> On Jul 5, 2016, at 5:23 PM, frank <<a
                                                          moz-do-not-send="true">hengjiew@uci.edu</a>>
                                                          wrote:<br>
>>>>>><br>
>>>>>> Hi,<br>
>>>>>><br>
>>>>>> I am using the CG ksp solver and Multigrid
                                                          preconditioner 
                                                          to solve a
                                                          linear system
                                                          in parallel.<br>
>>>>>> I chose to use the 'Telescope' as the
                                                          preconditioner
                                                          on the coarse
                                                          mesh for its
                                                          good
                                                          performance.<br>
>>>>>> The petsc options file is attached.<br>
>>>>>><br>
>>>>>> The domain is a 3d box.<br>
>>>>>> It works well when the grid is  1536*128*384
                                                          and the
                                                          process mesh
                                                          is 96*8*24.
                                                          When I double
                                                          the size of
                                                          grid and     
                                                                       
                                                                       
                                                                       
                                                           keep the same
                                                          process mesh
                                                          and petsc
                                                          options, I get
                                                          an "out of
                                                          memory" error
                                                          from the
                                                          super-cluster
                                                          I am using.<br>
>>>>>> Each process has access to at least 8G memory,
                                                          which should
                                                          be more than
                                                          enough for my
                                                          application. I
                                                          am sure that
                                                          all the other
                                                          parts of my
                                                          code( except
                                                          the linear
                                                          solver ) do
                                                          not use much
                                                          memory. So I
                                                          doubt if there
                                                          is something
                                                          wrong with the
                                                          linear solver.<br>
>>>>>> The error occurs before the linear system is
                                                          completely
                                                          solved so I
                                                          don't have the
                                                          info from ksp
                                                          view. I am not
                                                          able to
                                                          re-produce the
                                                          error with a
                                                          smaller
                                                          problem
                                                          either.<br>
>>>>>> In addition,  I tried to use the block jacobi
                                                          as the
                                                          preconditioner
                                                          with the same
                                                          grid and same
                                                          decomposition.
                                                          The linear
                                                          solver runs
                                                          extremely slow
                                                          but there is
                                                          no memory
                                                          error.<br>
>>>>>><br>
>>>>>> How can I diagnose what exactly cause the
                                                          error?<br>
>>>>>> Thank you so much.<br>
>>>>>><br>
>>>>>> Frank<br>
>>>>>> <petsc_options.txt><br>
>>>>>> <ksp_view_pre.txt><memory_test<wbr>2.txt><memory_test3.txt><petsc<wbr>_options.txt><br>
>>>>>><br>
>>>>><br>
>>>><br>
                                                          >>>
                                                          <ksp_view1.txt><ksp_view2.txt><wbr><ksp_view3.txt><memory1.txt><m<wbr>emory2.txt><petsc_options1.txt<wbr>><petsc_options2.txt><petsc_op<wbr>tions3.txt><br>
                                                          ><br>
                                                          <br>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <div> </div>
                                                          </blockquote>
                                                          </div>
                                                        </blockquote>
                                                        <br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </blockquote>
                                              </div>
                                              <br>
                                            </div>
                                          </blockquote>
                                          <br>
                                        </div>
                                      </blockquote>
                                      <div> </div>
                                      <div> </div>
                                      <div> </div>
                                    </div>
                                  </div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                  <br>
                  <br clear="all">
                  <span class="gmail-">
                    <div><br>
                    </div>
                    -- <br>
                    <div
                      class="gmail-m_-6757504983836048137gmail_signature">What
                      most experimenters take for granted before they
                      begin their experiments is infinitely more
                      interesting than any results to which their
                      experiments lead.<br>
                      -- Norbert Wiener</div>
                  </span></div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>