<div dir="ltr">If you clean the RHS of the null space then you can probably (only) use "<i style="color:rgb(0,0,0);white-space:pre-wrap">-mg_coarse_pc_type svd</i>", but you need this or the lu coarse grid solver will have problems.  If your solution starts drifting then you need to set the null space but this is often not needed.<br><div><br></div><div>Also, after you get this working you want to check these PCSetUp times.  This setup will not scale great but this behavior indicates that there is something wrong.  Hypre's default parameters are tune for 2D problems, you have a 3D problem, I assume.  GAMG should be fine.  As a rule of thumb the PCSetup should not be much more than a solve.  An easy 3D Poisson solve might require relatively  more setup and a hard 2D problem might require relatively less.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 3, 2015 at 8:01 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Tue, Nov 3, 2015 at 6:58 AM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <br>
    <div>On 3/11/2015 8:52 PM, Matthew Knepley
      wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">On Tue, Nov 3, 2015 at 6:49 AM, TAY
            wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
              <br>
              I tried and have attached the log.<br>
              <br>
              Ya, my Poisson eqn has Neumann boundary condition. Do I
              need to specify some null space stuff?  Like
              KSPSetNullSpace or MatNullSpaceCreate?</blockquote>
            <div><br>
            </div>
            <div>Yes, you need to attach the constant null space to the
              matrix.</div>
            <div><br>
            </div>
            <div>  Thanks,</div>
            <div><br>
            </div>
            <div>     Matt</div>
          </div>
        </div>
      </div>
    </blockquote>
    Ok so can you point me to a suitable example so that I know which
    one to use specifically?<br></div></blockquote><div><br></div></span><div><a href="https://bitbucket.org/petsc/petsc/src/9ae8fd060698c4d6fc0d13188aca8a1828c138ab/src/snes/examples/tutorials/ex12.c?at=master&fileviewer=file-view-default#ex12.c-761" target="_blank">https://bitbucket.org/petsc/petsc/src/9ae8fd060698c4d6fc0d13188aca8a1828c138ab/src/snes/examples/tutorials/ex12.c?at=master&fileviewer=file-view-default#ex12.c-761</a><br></div><div><br></div><div>  Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
    Thanks.<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><br>
                Thank you<br>
                <br>
                Yours sincerely,<br>
                <br>
                TAY wee-beng<br>
                <br>
              </span>
              <div>
                <div>
                  On 3/11/2015 12:45 PM, Barry Smith wrote:<br>
                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                      On Nov 2, 2015, at 10:37 PM, TAY wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                      wrote:<br>
                      <br>
                      Hi,<br>
                      <br>
                      I tried :<br>
                      <br>
                      1. -poisson_pc_gamg_agg_nsmooths 1
                      -poisson_pc_type gamg<br>
                      <br>
                      2. -poisson_pc_type gamg<br>
                    </blockquote>
                        Run with -poisson_ksp_monitor_true_residual
                    -poisson_ksp_monitor_converged_reason<br>
                    Does your poisson have Neumann boundary conditions?
                    Do you have any zeros on the diagonal for the matrix
                    (you shouldn't).<br>
                    <br>
                       There may be something wrong with your poisson
                    discretization that was also messing up hypre<br>
                    <br>
                    <br>
                    <br>
                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                      Both options give:<br>
                      <br>
                          1      0.00150000      0.00000000     
                      0.00000000 1.00000000             NaN           
                       NaN             NaN<br>
                      M Diverged but why?, time =            2<br>
                      reason =           -9<br>
                      <br>
                      How can I check what's wrong?<br>
                      <br>
                      Thank you<br>
                      <br>
                      Yours sincerely,<br>
                      <br>
                      TAY wee-beng<br>
                      <br>
                      On 3/11/2015 3:18 AM, Barry Smith wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                            hypre is just not scaling well here. I do
                        not know why. Since hypre is a block box for us
                        there is no way to determine why the poor
                        scaling.<br>
                        <br>
                            If you make the same two runs with -pc_type
                        gamg there will be a lot more information in the
                        log summary about in what routines it is scaling
                        well or poorly.<br>
                        <br>
                           Barry<br>
                        <br>
                        <br>
                        <br>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                          On Nov 2, 2015, at 3:17 AM, TAY wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                          wrote:<br>
                          <br>
                          Hi,<br>
                          <br>
                          I have attached the 2 files.<br>
                          <br>
                          Thank you<br>
                          <br>
                          Yours sincerely,<br>
                          <br>
                          TAY wee-beng<br>
                          <br>
                          On 2/11/2015 2:55 PM, Barry Smith wrote:<br>
                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                               Run (158/2)x(266/2)x(150/2) grid on 8
                            processes  and then (158)x(266)x(150) on 64
                            processors  and send the two -log_summary
                            results<br>
                            <br>
                               Barry<br>
                            <br>
                              <br>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                              On Nov 2, 2015, at 12:19 AM, TAY
                              wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                              wrote:<br>
                              <br>
                              Hi,<br>
                              <br>
                              I have attached the new results.<br>
                              <br>
                              Thank you<br>
                              <br>
                              Yours sincerely,<br>
                              <br>
                              TAY wee-beng<br>
                              <br>
                              On 2/11/2015 12:27 PM, Barry Smith wrote:<br>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                   Run without the -momentum_ksp_view
                                -poisson_ksp_view and send the new
                                results<br>
                                <br>
                                <br>
                                   You can see from the log summary that
                                the PCSetUp is taking a much smaller
                                percentage of the time meaning that it
                                is reusing the preconditioner and not
                                rebuilding it each time.<br>
                                <br>
                                Barry<br>
                                <br>
                                   Something makes no sense with the
                                output: it gives<br>
                                <br>
                                KSPSolve             199 1.0 2.3298e+03
                                1.0 5.20e+09 1.8 3.8e+04 9.9e+05 5.0e+02
                                90100 66100 24  90100 66100 24   165<br>
                                <br>
                                90% of the time is in the solve but
                                there is no significant amount of time
                                in other events of the code which is
                                just not possible. I hope it is due to
                                your IO.<br>
                                <br>
                                <br>
                                <br>
                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                  On Nov 1, 2015, at 10:02 PM, TAY
                                  wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                                  wrote:<br>
                                  <br>
                                  Hi,<br>
                                  <br>
                                  I have attached the new run with 100
                                  time steps for 48 and 96 cores.<br>
                                  <br>
                                  Only the Poisson eqn 's RHS changes,
                                  the LHS doesn't. So if I want to reuse
                                  the preconditioner, what must I do? Or
                                  what must I not do?<br>
                                  <br>
                                  Why does the number of processes
                                  increase so much? Is there something
                                  wrong with my coding? Seems to be so
                                  too for my new run.<br>
                                  <br>
                                  Thank you<br>
                                  <br>
                                  Yours sincerely,<br>
                                  <br>
                                  TAY wee-beng<br>
                                  <br>
                                  On 2/11/2015 9:49 AM, Barry Smith
                                  wrote:<br>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                       If you are doing many time steps
                                    with the same linear solver then you
                                    MUST do your weak scaling studies
                                    with MANY time steps since the setup
                                    time of AMG only takes place in the
                                    first stimestep. So run both 48 and
                                    96 processes with the same large
                                    number of time steps.<br>
                                    <br>
                                       Barry<br>
                                    <br>
                                    <br>
                                    <br>
                                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                      On Nov 1, 2015, at 7:35 PM, TAY
                                      wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                                      wrote:<br>
                                      <br>
                                      Hi,<br>
                                      <br>
                                      Sorry I forgot and use the old
                                      a.out. I have attached the new log
                                      for 48cores (log48), together with
                                      the 96cores log (log96).<br>
                                      <br>
                                      Why does the number of processes
                                      increase so much? Is there
                                      something wrong with my coding?<br>
                                      <br>
                                      Only the Poisson eqn 's RHS
                                      changes, the LHS doesn't. So if I
                                      want to reuse the preconditioner,
                                      what must I do? Or what must I not
                                      do?<br>
                                      <br>
                                      Lastly, I only simulated 2 time
                                      steps previously. Now I run for 10
                                      timesteps (log48_10). Is it
                                      building the preconditioner at
                                      every timestep?<br>
                                      <br>
                                      Also, what about momentum eqn? Is
                                      it working well?<br>
                                      <br>
                                      I will try the gamg later too.<br>
                                      <br>
                                      Thank you<br>
                                      <br>
                                      Yours sincerely,<br>
                                      <br>
                                      TAY wee-beng<br>
                                      <br>
                                      On 2/11/2015 12:30 AM, Barry Smith
                                      wrote:<br>
                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                           You used gmres with 48
                                        processes but richardson with
                                        96. You need to be careful and
                                        make sure you don't change the
                                        solvers when you change the
                                        number of processors since you
                                        can get very different
                                        inconsistent results<br>
                                        <br>
                                            Anyways all the time is
                                        being spent in the BoomerAMG
                                        algebraic multigrid setup and it
                                        is is scaling badly. When you
                                        double the problem size and
                                        number of processes it went from
                                        3.2445e+01 to 4.3599e+02
                                        seconds.<br>
                                        <br>
                                        PCSetUp                3 1.0
                                        3.2445e+01 1.0 9.58e+06 2.0
                                        0.0e+00 0.0e+00 4.0e+00 62  8 
                                        0  0  4  62  8  0  0  5    11<br>
                                        <br>
                                        PCSetUp                3 1.0
                                        4.3599e+02 1.0 9.58e+06 2.0
                                        0.0e+00 0.0e+00 4.0e+00 85 18 
                                        0  0  6  85 18  0  0  6     2<br>
                                        <br>
                                           Now is the Poisson problem
                                        changing at each timestep or can
                                        you use the same preconditioner
                                        built with BoomerAMG for all the
                                        time steps? Algebraic multigrid
                                        has a large set up time that you
                                        often doesn't matter if you have
                                        many time steps but if you have
                                        to rebuild it each timestep it
                                        is too large?<br>
                                        <br>
                                           You might also try -pc_type
                                        gamg and see how PETSc's
                                        algebraic multigrid scales for
                                        your problem/machine.<br>
                                        <br>
                                           Barry<br>
                                        <br>
                                        <br>
                                        <br>
                                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                          On Nov 1, 2015, at 7:30 AM,
                                          TAY wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                                          wrote:<br>
                                          <br>
                                          <br>
                                          On 1/11/2015 10:00 AM, Barry
                                          Smith wrote:<br>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                              On Oct 31, 2015, at 8:43
                                              PM, TAY wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                                              wrote:<br>
                                              <br>
                                              <br>
                                              On 1/11/2015 12:47 AM,
                                              Matthew Knepley wrote:<br>
                                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                                On Sat, Oct 31, 2015 at
                                                11:34 AM, TAY
                                                wee-beng<<a href="mailto:zonexo@gmail.com" target="_blank"></a><a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>> 
                                                wrote:<br>
                                                Hi,<br>
                                                <br>
                                                I understand that as
                                                mentioned in the faq,
                                                due to the limitations
                                                in memory, the scaling
                                                is not linear. So, I am
                                                trying to write a
                                                proposal to use a
                                                supercomputer.<br>
                                                Its specs are:<br>
                                                Compute nodes: 82,944
                                                nodes (SPARC64 VIIIfx;
                                                16GB of memory per node)<br>
                                                <br>
                                                8 cores / processor<br>
                                                Interconnect: Tofu
                                                (6-dimensional
                                                mesh/torus) Interconnect<br>
                                                Each cabinet contains 96
                                                computing nodes,<br>
                                                One of the requirement
                                                is to give the
                                                performance of my
                                                current code with my
                                                current set of data, and
                                                there is a formula to
                                                calculate the estimated
                                                parallel efficiency when
                                                using the new large set
                                                of data<br>
                                                There are 2 ways to give
                                                performance:<br>
                                                1. Strong scaling, which
                                                is defined as how the
                                                elapsed time varies with
                                                the number of processors
                                                for a fixed<br>
                                                problem.<br>
                                                2. Weak scaling, which
                                                is defined as how the
                                                elapsed time varies with
                                                the number of processors
                                                for a<br>
                                                fixed problem size per
                                                processor.<br>
                                                I ran my cases with 48
                                                and 96 cores with my
                                                current cluster, giving
                                                140 and 90 mins
                                                respectively. This is
                                                classified as strong
                                                scaling.<br>
                                                Cluster specs:<br>
                                                CPU: AMD 6234 2.4GHz<br>
                                                8 cores / processor
                                                (CPU)<br>
                                                6 CPU / node<br>
                                                So 48 Cores / CPU<br>
                                                Not sure abt the memory
                                                / node<br>
                                                <br>
                                                The parallel efficiency
                                                ‘En’ for a given degree
                                                of parallelism ‘n’
                                                indicates how much the
                                                program is<br>
                                                efficiently accelerated
                                                by parallel processing.
                                                ‘En’ is given by the
                                                following formulae.
                                                Although their<br>
                                                derivation processes are
                                                different depending on
                                                strong and weak scaling,
                                                derived formulae are the<br>
                                                same.<br>
                                                 From the estimated
                                                time, my parallel
                                                efficiency using 
                                                Amdahl's law on the
                                                current old cluster was
                                                52.7%.<br>
                                                So is my results
                                                acceptable?<br>
                                                For the large data set,
                                                if using 2205 nodes
                                                (2205X8cores), my
                                                expected parallel
                                                efficiency is only 0.5%.
                                                The proposal recommends
                                                value of > 50%.<br>
                                                The problem with this
                                                analysis is that the
                                                estimated serial
                                                fraction from Amdahl's
                                                Law  changes as a
                                                function<br>
                                                of problem size, so you
                                                cannot take the strong
                                                scaling from one problem
                                                and apply it to another
                                                without a<br>
                                                model of this
                                                dependence.<br>
                                                <br>
                                                Weak scaling does model
                                                changes with problem
                                                size, so I would measure
                                                weak scaling on your
                                                current<br>
                                                cluster, and extrapolate
                                                to the big machine. I
                                                realize that this does
                                                not make sense for many
                                                scientific<br>
                                                applications, but
                                                neither does requiring a
                                                certain parallel
                                                efficiency.<br>
                                              </blockquote>
                                              Ok I check the results for
                                              my weak scaling it is even
                                              worse for the expected
                                              parallel efficiency. From
                                              the formula used, it's
                                              obvious it's doing some
                                              sort of exponential
                                              extrapolation decrease. So
                                              unless I can achieve a
                                              near > 90% speed up
                                              when I double the cores
                                              and problem size for my
                                              current 48/96 cores
                                              setup,     extrapolating
                                              from about 96 nodes to
                                              10,000 nodes will give a
                                              much lower expected
                                              parallel efficiency for
                                              the new case.<br>
                                              <br>
                                              However, it's mentioned in
                                              the FAQ that due to memory
                                              requirement, it's
                                              impossible to get >90%
                                              speed when I double the
                                              cores and problem size (ie
                                              linear increase in
                                              performance), which means
                                              that I can't get >90%
                                              speed up when I double the
                                              cores and problem size for
                                              my current 48/96 cores
                                              setup. Is that so?<br>
                                            </blockquote>
                                               What is the output of
                                            -ksp_view -log_summary on
                                            the problem and then on the
                                            problem doubled in size and
                                            number of processors?<br>
                                            <br>
                                               Barry<br>
                                          </blockquote>
                                          Hi,<br>
                                          <br>
                                          I have attached the output<br>
                                          <br>
                                          48 cores: log48<br>
                                          96 cores: log96<br>
                                          <br>
                                          There are 2 solvers - The
                                          momentum linear eqn uses bcgs,
                                          while the Poisson eqn uses
                                          hypre BoomerAMG.<br>
                                          <br>
                                          Problem size doubled from
                                          158x266x150 to 158x266x300.<br>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                              So is it fair to say that
                                              the main problem does not
                                              lie in my programming
                                              skills, but rather the way
                                              the linear equations are
                                              solved?<br>
                                              <br>
                                              Thanks.<br>
                                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                                   Thanks,<br>
                                                <br>
                                                      Matt<br>
                                                Is it possible for this
                                                type of scaling in PETSc
                                                (>50%), when using
                                                17640 (2205X8) cores?<br>
                                                Btw, I do not have
                                                access to the system.<br>
                                                <br>
                                                <br>
                                                <br>
                                                Sent using CloudMagic
                                                Email<br>
                                                <br>
                                                <br>
                                                <br>
                                                -- <br>
                                                What most experimenters
                                                take for granted before
                                                they begin their
                                                experiments is
                                                infinitely more
                                                interesting than any
                                                results to which their
                                                experiments lead.<br>
                                                -- Norbert Wiener<br>
                                              </blockquote>
                                            </blockquote>
                                          </blockquote>
<log48.txt><log96.txt><br>
                                        </blockquote>
                                      </blockquote>
<log48_10.txt><log48.txt><log96.txt><br>
                                    </blockquote>
                                  </blockquote>
<log96_100.txt><log48_100.txt><br>
                                </blockquote>
                              </blockquote>
<log96_100_2.txt><log48_100_2.txt><br>
                            </blockquote>
                          </blockquote>
                          <log64_100.txt><log8_100.txt><br>
                        </blockquote>
                      </blockquote>
                    </blockquote>
                  </blockquote>
                  <br>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
          <br clear="all"><span><font color="#888888">
          <div><br>
          </div>
          -- <br>
          <div>What most experimenters take for
            granted before they begin their experiments is infinitely
            more interesting than any results to which their experiments
            lead.<br>
            -- Norbert Wiener</div>
        </font></span></div>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div></div></div>
</blockquote></div><br></div>