[petsc-users] Enquiry regarding log summary results

TAY wee-beng zonexo at gmail.com
Wed Oct 3 15:05:45 CDT 2012


Hi Jed,

I believe they are real cores. Anyway, I have attached the log summary 
for the 12/24/48 cores. I re-run a smaller case because the large 
problem can't run with 12cores.

Yours sincerely,

TAY wee-beng

On 3/10/2012 5:59 PM, Jed Brown wrote:
> There is an inordinate amount of time being spent in VecScatterEnd(). 
> That sometimes indicates a very bad partition. Also, are your "48 
> cores" real physical cores or just "logical cores" (look like cores to 
> the operating system, usually advertised as "threads" by the vendor, 
> nothing like cores in reality)? That can cause a huge load imbalance 
> and very confusing results as over-subscribed threads compete for 
> shared resources. Step it back to 24 threads and 12 threads, send 
> log_summary for each.
>
> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     On 2/10/2012 2:43 PM, Jed Brown wrote:
>>     On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I have combined the momentum linear eqns involving x,y,z into
>>         1 large matrix. The Poisson eqn is solved using HYPRE strcut
>>         format so it's not included. I run the code for 50 timesteps
>>         (hence 50 kspsolve) using 96 procs. The log_summary is given
>>         below. I have some questions:
>>
>>         1. After combining the matrix, I should have only 1 PETSc
>>         matrix. Why does it says there are 4 matrix, 12 vector etc?
>>
>>
>>     They are part of preconditioning. Are you sure you're using Hypre
>>     for this? It looks like you are using bjacobi/ilu.
>>
>>
>>         2. I'm looking at the stages which take the longest time. It
>>         seems that MatAssemblyBegin, VecNorm, VecAssemblyBegin,
>>         VecScatterEnd have very high ratios. The ratios of some
>>         others are also not too good (~ 1.6 - 2). So are these stages
>>         the reason why my code is not scaling well? What can I do to
>>         improve it?
>>
>>
>>     3/4 of the solve time is evenly balanced between MatMult,
>>     MatSolve, MatLUFactorNumeric, and VecNorm+VecDot.
>>
>>     The high VecAssembly time might be due to generating a lot of
>>     entries off-process?
>>
>>     In any case, this looks like an _extremely_ slow network, perhaps
>>     it's misconfigured?
>
>     My cluster is configured with 48 procs per node. I re-run the
>     case, using only 48 procs, thus there's no need to pass over a
>     'slow' interconnect. I'm now also using GAMG and BCGS for the
>     poisson and momentum eqn respectively. I have also separated the
>     x,y,z component of the momentum eqn to 3 separate linear eqns to
>     debug the problem.
>
>     Results show that stage "momentum_z" is taking a lot of time. I
>     wonder if it has to do with the fact that I am partitioning my
>     grids in the z direction. VecScatterEnd, MatMult are taking a lot
>     of time. VecNormalize, VecScatterEnd, VecNorm, VecAssemblyBegin 's
>     ratio are also not good.
>
>     I wonder why a lot of entries are generated off-process.
>
>     I create my RHS vector using:
>
>     /call
>     VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/
>
>     where ijk_xyz_sta and ijk_xyz_end are obtained from
>
>     /call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)/
>
>     I then insert the values into the vector using:
>
>     /call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta ,
>     (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1
>     : ijk_xyz_end) , INSERT_VALUES , ierr)/
>
>     What should I do to correct the problem?
>
>     Thanks
>
>
>>
>>         Btw, I insert matrix using:
>>
>>         /do ijk=ijk_xyz_sta+1,ijk_xyz_end//
>>         //
>>         //    II = ijk - 1//    !Fortran shift to 0-based//
>>         ////
>>         //    call
>>         MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)//
>>         //
>>         //end do/
>>
>>         where ijk_xyz_sta/ijk_xyz_end are the starting/end index
>>
>>         int_semi_xyz(ijk,1:7) stores the 7 column global indices
>>
>>         semi_mat_xyz has the corresponding values.
>>
>>         and I insert vectors using:
>>
>>         call
>>         VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr)
>>
>>         Thanks!
>>
>>         /
>>         /
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>         On 30/9/2012 11:30 PM, Jed Brown wrote:
>>>
>>>         You can measure the time spent in Hypre via PCApply and
>>>         PCSetUp, but you can't get finer grained integrated
>>>         profiling because it was not set up that way.
>>>
>>>         On Sep 30, 2012 3:26 PM, "TAY wee-beng" <zonexo at gmail.com
>>>         <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             On 27/9/2012 1:44 PM, Matthew Knepley wrote:
>>>>             On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I'm doing a log summary for my 3d cfd code. I have
>>>>                 some questions:
>>>>
>>>>                 1. if I'm solving 3 linear equations using ksp, is
>>>>                 the result given in the log summary the total of
>>>>                 the 3 linear eqns' performance? How can I get the
>>>>                 performance for each individual eqn?
>>>>
>>>>
>>>>             Use logging stages:
>>>>             http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html
>>>>
>>>>                 2. If I run my code for 10 time steps, does the log
>>>>                 summary gives the total or avg performance/ratio?
>>>>
>>>>
>>>>             Total.
>>>>
>>>>                 3. Besides PETSc, I'm also using HYPRE's native
>>>>                 geometric MG (Struct) to solve my Cartesian's grid
>>>>                 CFD poisson eqn. Is there any way I can use PETSc's
>>>>                 log summary to get HYPRE's performance? If I use
>>>>                 boomerAMG thru PETSc, can I get its performance?
>>>>
>>>>
>>>>             If you mean flops, only if you count them yourself and
>>>>             tell PETSc using
>>>>             http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html
>>>>
>>>>             This is the disadvantage of using packages that do not
>>>>             properly monitor things :)
>>>>
>>>>                 Matt
>>>             So u mean if I use boomerAMG thru PETSc, there is no
>>>             proper way of evaluating its performance, beside using
>>>             PetscLogFlops?
>>>>
>>>>
>>>>                 -- 
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121003/6de88454/attachment-0001.html>
-------------- next part --------------

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.3-dev_shared_rel named n12-10 with 12 processors, by wtay Wed Oct  3 21:34:22 2012
Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e  HG Date: Sun Sep 30 22:42:36 2012 -0500

                         Max       Max/Min        Avg      Total 
Time (sec):           4.407e+03      1.00084   4.405e+03
Objects:              5.680e+02      1.00000   5.680e+02
Flops:                7.375e+11      1.11505   6.777e+11  8.133e+12
Flops/sec:            1.674e+08      1.11490   1.539e+08  1.846e+09
MPI Messages:         1.460e+05      2.95457   1.080e+05  1.296e+06
MPI Message Lengths:  1.035e+10      2.02602   8.761e+04  1.136e+11
MPI Reductions:       1.571e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.6484e+02   6.0%  0.0000e+00   0.0%  0.000e+00   0.0%  0.000e+00        0.0%  3.600e+01   0.2% 
 1:         poisson: 1.1297e+02   2.6%  1.9050e+11   2.3%  2.900e+04   2.2%  2.173e+03        2.5%  1.215e+03   7.7% 
 2:      momentum_x: 2.8093e+00   0.1%  2.1513e+09   0.0%  8.800e+01   0.0%  1.010e+01        0.0%  2.800e+01   0.2% 
 3:      momentum_y: 2.8534e+00   0.1%  2.1666e+09   0.0%  8.800e+01   0.0%  1.023e+01        0.0%  2.800e+01   0.2% 
 4:      momentum_z: 4.0213e+03  91.3%  7.9380e+12  97.6%  1.267e+06  97.7%  8.541e+04       97.5%  1.440e+04  91.7% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage


--- Event Stage 1: poisson

MatMult             1053 1.0 5.4872e+01 1.3 9.42e+09 1.2 2.0e+04 1.1e+05 0.0e+00  1  1  2  2  0  44 53 69 75  0  1843
MatMultAdd           156 1.0 4.5059e+00 1.3 6.01e+08 1.2 2.7e+03 1.9e+04 0.0e+00  0  0  0  0  0   4  3  9  2  0  1358
MatMultTranspose     156 1.0 5.6541e+00 1.6 6.01e+08 1.2 2.7e+03 1.9e+04 0.0e+00  0  0  0  0  0   4  3  9  2  0  1082
MatSolve              78 0.0 4.3106e-04 0.0 6.75e+04 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   157
MatLUFactorSym         1 1.0 4.1008e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 3.5048e-05 2.9 5.53e+03 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   158
MatConvert             4 1.0 1.9400e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale              12 1.0 1.9127e-01 1.2 5.26e+07 1.2 7.8e+01 1.0e+05 0.0e+00  0  0  0  0  0   0  0  0  0  0  2896
MatAssemblyBegin      69 1.0 3.9459e+00 2.0 0.00e+00 0.0 2.3e+02 1.8e+04 7.4e+01  0  0  0  0  0   2  0  1  0  6     0
MatAssemblyEnd        69 1.0 1.9654e+00 1.1 0.00e+00 0.0 1.0e+03 1.6e+04 2.0e+02  0  0  0  0  1   2  0  4  1 16     0
MatGetRow        3544348 1.0 8.9238e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 0.0 1.6928e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 0.0 7.7009e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.7e-01  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             4 1.0 6.2370e-01 1.1 0.00e+00 0.0 4.3e+02 1.4e+05 9.2e+01  0  0  0  0  1   1  0  1  2  8     0
MatAXPY                4 1.0 6.2184e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             4 1.0 1.7033e+00 1.0 3.72e+07 1.2 4.9e+02 4.9e+04 9.6e+01  0  0  0  0  1   2  0  2  1  8   233
MatMatMultSym          4 1.0 1.2570e+00 1.0 0.00e+00 0.0 4.1e+02 3.9e+04 8.8e+01  0  0  0  0  1   1  0  1  1  7     0
MatMatMultNum          4 1.0 4.5279e-01 1.0 3.72e+07 1.2 7.8e+01 1.0e+05 8.0e+00  0  0  0  0  0   0  0  0  0  1   877
MatPtAP                4 1.0 6.5757e+00 1.0 1.16e+09 1.8 8.4e+02 1.1e+05 1.1e+02  0  0  0  0  1   6  5  3  3  9  1304
MatPtAPSymbolic        4 1.0 3.6004e+00 1.0 0.00e+00 0.0 7.5e+02 9.3e+04 1.0e+02  0  0  0  0  1   3  0  3  2  8     0
MatPtAPNumeric         4 1.0 2.9819e+00 1.0 1.16e+09 1.8 8.9e+01 2.7e+05 8.0e+00  0  0  0  0  0   3  5  0  1  1  2876
MatTrnMatMult          4 1.0 1.6466e+01 1.0 2.04e+09 1.9 4.7e+02 6.6e+05 1.2e+02  0  0  0  0  1  15 12  2 11 10  1385
MatGetLocalMat        20 1.0 6.5459e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01  0  0  0  0  0   1  0  0  0  2     0
MatGetBrAoCol         12 1.0 1.7862e-01 2.1 0.00e+00 0.0 5.5e+02 1.4e+05 1.6e+01  0  0  0  0  0   0  0  2  3  1     0
MatGetSymTrans         8 1.0 8.3050e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        79 1.0 8.9525e-01 1.7 3.90e+08 1.0 0.0e+00 0.0e+00 7.9e+01  0  0  0  0  1   1  2  0  0  7  5139
KSPSetUp              11 1.0 1.8398e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  1     0
KSPSolve               1 1.0 1.1039e+02 1.0 1.67e+10 1.1 2.9e+04 9.7e+04 1.2e+03  3  2  2  2  8  98100100100 99  1726
VecDot                38 1.0 3.4189e+00 2.9 1.22e+08 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  0   2  1  0  0  3   424
VecDotNorm2           19 1.0 3.2140e+00 2.7 2.44e+08 1.0 0.0e+00 0.0e+00 5.7e+01  0  0  0  0  0   2  2  0  0  5   901
VecMDot               79 1.0 5.6992e-01 2.2 1.95e+08 1.0 0.0e+00 0.0e+00 7.9e+01  0  0  0  0  1   0  1  0  0  7  4037
VecNorm              142 1.0 2.7666e+00 5.7 1.03e+08 1.0 0.0e+00 0.0e+00 1.4e+02  0  0  0  0  1   1  1  0  0 12   442
VecScale             746 1.0 1.2098e+00 2.0 2.96e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  2887
VecCopy              201 1.0 4.0863e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               741 1.0 3.8170e-01 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY             1291 1.0 4.7930e+00 2.7 1.11e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  7  0  0  0  2732
VecAYPX             1248 1.0 4.8211e+00 2.0 6.91e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  4  0  0  0  1692
VecAXPBYCZ            38 1.0 9.7988e-01 2.0 2.44e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  2956
VecWAXPY              38 1.0 9.6000e-01 2.5 1.22e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  1  0  0  0  1509
VecMAXPY             122 1.0 3.9971e-01 1.5 2.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  6802
VecAssemblyBegin      49 1.0 1.4084e-01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+02  0  0  0  0  1   0  0  0  0 12     0
VecAssemblyEnd        49 1.0 1.4496e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult     980 1.0 7.1430e+00 3.3 4.34e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  3  0  0  0   717
VecScatterBegin     1420 1.0 2.7552e-01 3.7 0.00e+00 0.0 2.6e+04 9.0e+04 0.0e+00  0  0  2  2  0   0  0 91 84  0     0
VecScatterEnd       1420 1.0 3.7127e+0112.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0  18  0  0  0  0     0
VecSetRandom           4 1.0 4.9694e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize         122 1.0 2.3469e+0021.0 5.85e+07 1.0 0.0e+00 0.0e+00 1.2e+02  0  0  0  0  1   1  0  0  0 10   294
PCSetUp                2 1.0 3.5545e+01 1.0 3.60e+09 1.1 4.4e+03 1.6e+05 9.6e+02  1  1  0  1  6  31 22 15 24 79  1184
PCSetUpOnBlocks       39 1.0 2.9898e-04 1.7 5.53e+03 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0    18
PCApply               39 1.0 6.6028e+01 1.1 1.20e+10 1.1 2.4e+04 8.1e+04 1.3e+02  1  2  2  2  1  56 68 82 68 11  1963
PCGAMGgraph_AGG        4 1.0 4.2616e+00 1.0 3.72e+07 1.2 2.3e+02 5.1e+04 7.6e+01  0  0  0  0  0   4  0  1  0  6    93
PCGAMGcoarse_AGG       4 1.0 1.7933e+01 1.0 2.04e+09 1.9 1.3e+03 3.4e+05 2.9e+02  0  0  0  0  2  16 12  5 16 24  1271
PCGAMGProl_AGG         4 1.0 2.2483e+00 1.0 0.00e+00 0.0 5.5e+02 6.4e+04 1.1e+02  0  0  0  0  1   2  0  2  1  9     0
PCGAMGPOpt_AGG         4 1.0 4.4578e+00 1.0 9.11e+08 1.1 1.3e+03 8.0e+04 2.1e+02  0  0  0  0  1   4  5  4  4 17  2310

--- Event Stage 2: momentum_x

MatMult                2 1.0 9.9061e-02 1.1 4.10e+07 1.0 4.4e+01 2.4e+05 0.0e+00  0  0  0  0  0   3 23 50 80  0  4907
MatSolve               3 1.0 1.4430e-01 1.1 6.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  5026
MatLUFactorNum         1 1.0 2.5046e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   9 19  0  0  0  1622
MatILUFactorSym        1 1.0 1.5358e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   5  0  0  0  4     0
MatAssemblyBegin       1 1.0 2.8467e-013030.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   4  0  0  0  7     0
MatAssemblyEnd         1 1.0 2.2157e-01 1.1 0.00e+00 0.0 4.4e+01 5.9e+04 8.0e+00  0  0  0  0  0   8  0 50 20 29     0
MatGetRowIJ            1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.8516e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  0  0  0  7     0
KSPSetUp               2 1.0 6.5584e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 8.0632e-01 1.0 1.81e+08 1.0 4.4e+01 2.4e+05 1.2e+01  0  0  0  0  0  29100 50 80 43  2668
VecDot                 2 1.0 1.6051e-02 2.0 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  4  0  0  7  4750
VecDotNorm2            1 1.0 1.5273e-02 1.7 1.28e+07 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  7  0  0 11  9983
VecNorm                2 1.0 6.1728e-0212.2 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  4  0  0  7  1235
VecCopy                2 1.0 1.0178e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.7939e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPBYCZ             2 1.0 1.9800e-02 1.4 1.28e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0  7700
VecWAXPY               2 1.0 1.6218e-02 1.4 6.42e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  4701
VecAssemblyBegin       2 1.0 2.2810e-02154.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   0  0  0  0 21     0
VecAssemblyEnd         2 1.0 9.0599e-06 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 7.9203e-04 2.3 0.00e+00 0.0 4.4e+01 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 1.6581e-0214.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCSetUp                2 1.0 4.2014e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0  14 19  0  0 18   967
PCSetUpOnBlocks        1 1.0 4.1998e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0  14 19  0  0 11   968
PCApply                3 1.0 1.4989e-01 1.1 6.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  4838

--- Event Stage 3: momentum_y

MatMult                2 1.0 1.0025e-01 1.1 4.14e+07 1.0 4.4e+01 2.4e+05 0.0e+00  0  0  0  0  0   3 23 50 80  0  4896
MatSolve               3 1.0 1.4338e-01 1.1 6.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  5107
MatLUFactorNum         1 1.0 2.6953e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   9 19  0  0  0  1521
MatILUFactorSym        1 1.0 1.5543e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   5  0  0  0  4     0
MatAssemblyBegin       1 1.0 3.4289e-013174.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   5  0  0  0  7     0
MatAssemblyEnd         1 1.0 2.0907e-01 1.1 0.00e+00 0.0 4.4e+01 6.0e+04 8.0e+00  0  0  0  0  0   7  0 50 20 29     0
MatGetRowIJ            1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.8706e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  0  0  0  7     0
KSPSetUp               2 1.0 6.5596e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 8.3469e-01 1.0 1.83e+08 1.0 4.4e+01 2.4e+05 1.2e+01  0  0  0  0  0  29100 50 80 43  2596
VecDot                 2 1.0 1.5645e-02 1.9 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  4  0  0  7  4873
VecDotNorm2            1 1.0 1.5150e-02 1.7 1.28e+07 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  7  0  0 11 10064
VecNorm                2 1.0 7.3364e-02 6.5 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  4  0  0  7  1039
VecCopy                2 1.0 1.0147e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.7555e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPBYCZ             2 1.0 2.6077e-02 1.9 1.28e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0  5847
VecWAXPY               2 1.0 1.6632e-02 1.5 6.42e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  4584
VecAssemblyBegin       2 1.0 1.9942e-02134.7 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   0  0  0  0 21     0
VecAssemblyEnd         2 1.0 8.1062e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 7.4911e-04 2.2 0.00e+00 0.0 4.4e+01 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 1.8597e-0215.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCSetUp                2 1.0 4.4319e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0  15 19  0  0 18   925
PCSetUpOnBlocks        1 1.0 4.4305e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0  15 19  0  0 11   925
PCApply                3 1.0 1.4899e-01 1.1 6.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  4914

--- Event Stage 4: momentum_z

MatMult            52319 1.0 2.3522e+03 1.1 4.72e+11 1.1 9.9e+05 1.1e+05 0.0e+00 52 62 76 93  0  57 64 78 95  0  2157
MatMultAdd          8012 1.0 1.6520e+02 1.2 3.09e+10 1.2 1.4e+05 1.9e+04 0.0e+00  4  4 11  2  0   4  4 11  2  0  1902
MatMultTranspose    8012 1.0 2.4016e+02 1.4 3.09e+10 1.2 1.4e+05 1.9e+04 0.0e+00  5  4 11  2  0   5  4 11  2  0  1309
MatSolve            444110.2 4.4971e+01 1.7 8.93e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  2354
MatLUFactorNum       145 1.0 4.3540e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  1362
MatILUFactorSym        1 1.0 1.5319e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin     145 1.0 3.5545e+01 8.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02  1  0  0  0  2   1  0  0  0  2     0
MatAssemblyEnd       145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 9.5367e-07 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.8826e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog      2003 1.0 5.7039e-02 1.3 1.74e+05 0.0 0.0e+00 0.0e+00 2.0e+03  0  0  0  0 13   0  0  0  0 14     3
KSPSetUp             290 1.0 6.3499e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve             194 1.0 3.7263e+03 1.0 7.20e+11 1.1 1.3e+06 8.7e+04 1.3e+04 85 98 98 97 82  93100100100 90  2130
VecDot              2244 1.0 1.6008e+02 1.9 7.21e+09 1.0 0.0e+00 0.0e+00 2.2e+03  3  1  0  0 14   3  1  0  0 16   534
VecDotNorm2         1122 1.0 1.5508e+02 1.8 1.44e+10 1.0 0.0e+00 0.0e+00 3.4e+03  3  2  0  0 21   3  2  0  0 23  1103
VecMDot             2003 1.0 4.2973e-02 1.4 8.61e+04 0.0 0.0e+00 0.0e+00 2.0e+03  0  0  0  0 13   0  0  0  0 14     2
VecNorm             5322 1.0 1.1301e+02 3.1 4.23e+09 1.0 0.0e+00 0.0e+00 5.3e+03  1  1  0  0 34   1  1  0  0 37   444
VecScale           36054 1.0 5.6512e+01 1.3 1.42e+10 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  2965
VecCopy            10403 1.0 2.1793e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             35069 1.0 1.9805e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            66099 1.0 1.7941e+02 1.3 5.68e+10 1.0 0.0e+00 0.0e+00 0.0e+00  4  8  0  0  0   4  8  0  0  0  3736
VecAYPX            64096 1.0 1.9224e+02 1.3 3.55e+10 1.0 0.0e+00 0.0e+00 0.0e+00  4  5  0  0  0   4  5  0  0  0  2179
VecAXPBYCZ          2244 1.0 5.0866e+01 1.5 1.44e+10 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  3363
VecWAXPY            2244 1.0 4.8066e+01 1.4 7.21e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  1780
VecMAXPY            4006 1.0 5.5676e-03 1.5 1.76e+05 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    32
VecAssemblyBegin     388 1.0 2.0726e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03  0  0  0  0  7   0  0  0  0  8     0
VecAssemblyEnd       388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult   48072 1.0 2.3956e+02 1.6 2.13e+10 1.0 0.0e+00 0.0e+00 0.0e+00  5  3  0  0  0   5  3  0  0  0  1049
VecScatterBegin    68343 1.0 1.0083e+01 2.4 0.00e+00 0.0 1.3e+06 8.7e+04 0.0e+00  0  0 98 97  0   0  0100100  0     0
VecScatterEnd      68343 1.0 1.1153e+03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 19  0  0  0  0  20  0  0  0  0     0
VecNormalize        4006 1.0 5.8749e+01698.9 2.64e+05 0.0 0.0e+00 0.0e+00 4.0e+03  0  0  0  0 25   0  0  0  0 28     0
PCSetUp              290 1.0 4.3717e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 5.0e+00  1  1  0  0  0   1  1  0  0  0  1357
PCSetUpOnBlocks     2148 1.0 4.3718e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 3.0e+00  1  1  0  0  0   1  1  0  0  0  1357
PCApply             2438 1.0 3.1165e+03 1.0 6.24e+11 1.1 1.2e+06 8.1e+04 6.0e+03 70 83 94 87 38  76 85 96 89 42  2170
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix    12             56   2093884688     0
       Krylov Solver     4             13        31728     0
              Vector     8             85    485916024     0
      Vector Scatter     0             12        12720     0
           Index Set     0             16     19283008     0
      Preconditioner     0             13        12540     0
              Viewer     1              0            0     0

--- Event Stage 1: poisson

              Matrix   117             76   1488820988     0
      Matrix Coarsen     4              4         2480     0
       Krylov Solver    10              4       120512     0
              Vector   225            175    387307616     0
      Vector Scatter    31             22        23320     0
           Index Set    81             74       808508     0
      Preconditioner    11              4         3456     0
         PetscRandom     4              4         2464     0

--- Event Stage 2: momentum_x

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       120488     0
      Preconditioner     2              0            0     0

--- Event Stage 3: momentum_y

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122104     0
      Preconditioner     2              0            0     0

--- Event Stage 4: momentum_z

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122096     0
      Preconditioner     2              0            0     0
========================================================================================================================
Average time to get PetscTime(): 1.90735e-07
Average time for MPI_Barrier(): 1.03951e-05
Average time for zero size MPI_Send(): 1.1007e-05
#PETSc Option Table entries:
-log_summary
-poisson_pc_gamg_agg_nsmooths 1
-poisson_pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Mon Oct  1 11:36:09 2012
Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries
-----------------------------------------
Libraries compiled on Mon Oct  1 11:36:09 2012 on hpc12 
Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final
Using PETSc directory: /home/wtay/Codes/petsc-dev
Using PETSc arch: petsc-3.3-dev_shared_rel
-----------------------------------------

Using C compiler: /opt/openmpi-1.5.3/bin/mpicc  -fPIC -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90  -fPIC -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include
-----------------------------------------

Using C linker: /opt/openmpi-1.5.3/bin/mpicc
Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl 
-----------------------------------------
-------------- next part --------------

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.3-dev_shared_rel named n12-09 with 24 processors, by wtay Wed Oct  3 21:19:36 2012
Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e  HG Date: Sun Sep 30 22:42:36 2012 -0500

                         Max       Max/Min        Avg      Total 
Time (sec):           3.060e+03      1.00172   3.057e+03
Objects:              5.680e+02      1.00000   5.680e+02
Flops:                3.503e+11      1.23913   2.909e+11  6.982e+12
Flops/sec:            1.145e+08      1.23763   9.515e+07  2.284e+09
MPI Messages:         1.542e+05      3.65607   9.905e+04  2.377e+06
MPI Message Lengths:  8.847e+09      2.01744   8.530e+04  2.028e+11
MPI Reductions:       1.398e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.1590e+02   7.1%  0.0000e+00   0.0%  0.000e+00   0.0%  0.000e+00        0.0%  3.600e+01   0.3% 
 1:         poisson: 6.7753e+01   2.2%  1.9064e+11   2.7%  6.527e+04   2.7%  2.477e+03        2.9%  1.307e+03   9.4% 
 2:      momentum_x: 2.0559e+00   0.1%  2.1431e+09   0.0%  1.840e+02   0.0%  1.151e+01        0.0%  2.800e+01   0.2% 
 3:      momentum_y: 1.9997e+00   0.1%  2.1583e+09   0.0%  1.840e+02   0.0%  1.167e+01        0.0%  2.800e+01   0.2% 
 4:      momentum_z: 2.7697e+03  90.6%  6.7873e+12  97.2%  2.312e+06  97.2%  8.280e+04       97.1%  1.258e+04  90.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage


--- Event Stage 1: poisson

MatMult             1053 1.0 3.4221e+01 1.3 5.37e+09 1.3 4.2e+04 1.0e+05 0.0e+00  1  1  2  2  0  44 53 65 75  0  2962
MatMultAdd           156 1.0 1.6699e+00 1.5 3.58e+08 1.5 6.1e+03 1.7e+04 0.0e+00  0  0  0  0  0   2  3  9  2  0  3677
MatMultTranspose     156 1.0 4.1421e+00 3.5 3.58e+08 1.5 6.1e+03 1.7e+04 0.0e+00  0  0  0  0  0   3  3  9  2  0  1482
MatSolve              78 0.0 4.3559e-04 0.0 7.45e+04 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   171
MatLUFactorSym         1 1.0 4.1962e-05 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 3.6955e-05 4.1 6.48e+03 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   175
MatConvert             4 1.0 1.7373e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale              12 1.0 1.8639e-01 1.7 3.05e+07 1.4 1.7e+02 9.8e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  2980
MatAssemblyBegin      69 1.0 4.5507e+00 3.2 0.00e+00 0.0 5.3e+02 1.6e+04 7.4e+01  0  0  0  0  1   4  0  1  0  6     0
MatAssemblyEnd        69 1.0 1.4305e+00 1.2 0.00e+00 0.0 2.6e+03 1.3e+04 2.0e+02  0  0  0  0  1   2  0  4  1 15     0
MatGetRow        1821330 1.1 6.5450e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 0.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 0.0 7.3195e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.3e-02  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             4 1.0 5.3315e-01 1.3 0.00e+00 0.0 3.1e+03 4.7e+04 1.8e+02  0  0  0  0  1   1  0  5  2 14     0
MatAXPY                4 1.0 3.9307e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             4 1.0 1.1955e+00 1.0 2.13e+07 1.3 1.1e+03 4.4e+04 9.6e+01  0  0  0  0  1   2  0  2  1  7   333
MatMatMultSym          4 1.0 9.2042e-01 1.0 0.00e+00 0.0 9.7e+02 3.5e+04 8.8e+01  0  0  0  0  1   1  0  1  1  7     0
MatMatMultNum          4 1.0 2.8186e-01 1.0 2.13e+07 1.3 1.7e+02 9.8e+04 8.0e+00  0  0  0  0  0   0  0  0  0  1  1412
MatPtAP                4 1.0 4.3431e+00 1.0 8.45e+08 2.7 2.2e+03 9.0e+04 1.1e+02  0  0  0  0  1   6  5  3  3  8  1978
MatPtAPSymbolic        4 1.0 2.4485e+00 1.0 0.00e+00 0.0 2.0e+03 7.5e+04 1.0e+02  0  0  0  0  1   4  0  3  2  8     0
MatPtAPNumeric         4 1.0 1.8946e+00 1.0 8.45e+08 2.7 2.4e+02 2.1e+05 8.0e+00  0  0  0  0  0   3  5  0  1  1  4534
MatTrnMatMult          4 1.0 9.8843e+00 1.0 1.03e+09 3.2 1.1e+03 5.9e+05 1.2e+02  0  0  0  0  1  15 12  2 11  9  2291
MatGetLocalMat        20 1.0 5.3209e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01  0  0  0  0  0   1  0  0  0  2     0
MatGetBrAoCol         12 1.0 2.1986e-01 2.3 0.00e+00 0.0 1.2e+03 1.4e+05 1.6e+01  0  0  0  0  0   0  0  2  3  1     0
MatGetSymTrans         8 1.0 5.6299e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        79 1.0 9.2357e-01 2.1 2.00e+08 1.1 0.0e+00 0.0e+00 7.9e+01  0  0  0  0  1   1  2  0  0  6  4984
KSPSetUp              11 1.0 1.5080e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  1     0
KSPSolve               1 1.0 6.5990e+01 1.0 9.22e+09 1.2 6.5e+04 9.0e+04 1.3e+03  2  3  3  3  9  97100100100 99  2889
VecDot                38 1.0 3.3833e+00 9.7 6.22e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  0   2  1  0  0  3   428
VecDotNorm2           19 1.0 3.3915e+00 7.4 1.24e+08 1.0 0.0e+00 0.0e+00 5.7e+01  0  0  0  0  0   2  2  0  0  4   854
VecMDot               79 1.0 7.9980e-01 3.8 1.00e+08 1.1 0.0e+00 0.0e+00 7.9e+01  0  0  0  0  1   1  1  0  0  6  2878
VecNorm              142 1.0 2.0564e+0011.8 5.28e+07 1.0 0.0e+00 0.0e+00 1.4e+02  0  0  0  0  1   1  1  0  0 11   595
VecScale             746 1.0 9.4111e-01 3.6 1.52e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  3713
VecCopy              201 1.0 2.6442e-01 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               741 1.0 3.9763e-01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY             1291 1.0 3.2086e+00 4.0 5.70e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  7  0  0  0  4082
VecAYPX             1248 1.0 3.0970e+00 3.1 3.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  4  0  0  0  2635
VecAXPBYCZ            38 1.0 1.1318e+00 3.5 1.24e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  2560
VecWAXPY              38 1.0 1.0047e+00 3.5 6.22e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  1  0  0  0  1442
VecMAXPY             122 1.0 3.1358e-01 2.1 1.18e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  8674
VecAssemblyBegin      72 1.0 2.3761e-0110.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02  0  0  0  0  2   0  0  0  0 16     0
VecAssemblyEnd        72 1.0 1.7858e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult     980 1.0 3.3524e+00 3.6 2.23e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  3  0  0  0  1529
VecScatterBegin     1443 1.0 3.2857e-01 5.7 0.00e+00 0.0 5.9e+04 8.4e+04 0.0e+00  0  0  2  2  0   0  0 91 84  0     0
VecScatterEnd       1443 1.0 2.1979e+0125.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0  18  0  0  0  0     0
VecSetRandom           4 1.0 4.0149e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize         122 1.0 1.3411e+0013.5 3.01e+07 1.1 0.0e+00 0.0e+00 1.2e+02  0  0  0  0  1   0  0  0  0  9   515
PCSetUp                2 1.0 2.2811e+01 1.0 1.83e+09 1.2 1.2e+04 1.2e+05 1.1e+03  1  1  1  1  8  34 22 19 24 80  1839
PCSetUpOnBlocks       39 1.0 2.9802e-04 1.9 6.48e+03 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0    22
PCApply               39 1.0 3.5861e+01 1.1 6.76e+09 1.3 5.1e+04 7.9e+04 1.3e+02  1  2  2  2  1  51 68 78 68 10  3622
PCGAMGgraph_AGG        4 1.0 2.9758e+00 1.0 2.13e+07 1.3 4.9e+02 4.9e+04 7.6e+01  0  0  0  0  1   4  0  1  0  6   134
PCGAMGcoarse_AGG       4 1.0 1.0956e+01 1.0 1.03e+09 3.2 5.2e+03 1.8e+05 3.8e+02  0  0  0  0  3  16 12  8 16 29  2067
PCGAMGProl_AGG         4 1.0 1.1267e+00 1.0 0.00e+00 0.0 1.3e+03 5.5e+04 1.1e+02  0  0  0  0  1   2  0  2  1  9     0
PCGAMGPOpt_AGG         4 1.0 3.4058e+00 1.0 4.93e+08 1.2 2.8e+03 7.6e+04 2.1e+02  0  0  0  0  2   5  5  4  4 16  3027

--- Event Stage 2: momentum_x

MatMult                2 1.0 9.7961e-02 1.8 2.09e+07 1.1 9.2e+01 2.4e+05 0.0e+00  0  0  0  0  0   4 23 50 80  0  4962
MatSolve               3 1.0 1.1522e-01 1.7 3.10e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   4 34  0  0  0  6257
MatLUFactorNum         1 1.0 1.6440e-01 1.3 1.73e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   7 19  0  0  0  2448
MatILUFactorSym        1 1.0 1.3943e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   5  0  0  0  4     0
MatAssemblyBegin       1 1.0 4.3930e-014067.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0  11  0  0  0  7     0
MatAssemblyEnd         1 1.0 2.0642e-01 1.2 0.00e+00 0.0 9.2e+01 5.9e+04 8.0e+00  0  0  0  0  0  10  0 50 20 29     0
MatGetRowIJ            1 1.0 5.0068e-06 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.8502e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  0  0  0  7     0
KSPSetUp               2 1.0 5.3131e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 6.5504e-01 1.0 9.21e+07 1.1 9.2e+01 2.4e+05 1.2e+01  0  0  0  0  0  32100 50 80 43  3272
VecDot                 2 1.0 5.4927e-0210.2 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  4  0  0  7  1388
VecDotNorm2            1 1.0 5.2273e-02 6.6 6.54e+06 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   1  7  0  0 11  2917
VecNorm                2 1.0 1.5215e-0157.3 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   4  4  0  0  7   501
VecCopy                2 1.0 8.7559e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.7282e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPBYCZ             2 1.0 2.3268e-02 3.2 6.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0  6553
VecWAXPY               2 1.0 1.9122e-02 2.8 3.27e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  3987
VecAssemblyBegin       2 1.0 4.0814e-02200.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   1  0  0  0 21     0
VecAssemblyEnd         2 1.0 1.4067e-05 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 1.3988e-03 3.6 0.00e+00 0.0 9.2e+01 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 4.9311e-0226.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
PCSetUp                2 1.0 3.1949e-01 1.6 1.73e+07 1.1 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0  13 19  0  0 18  1260
PCSetUpOnBlocks        1 1.0 3.1934e-01 1.6 1.73e+07 1.1 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0  13 19  0  0 11  1260
PCApply                3 1.0 1.2085e-01 1.7 3.10e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  5966

--- Event Stage 3: momentum_y

MatMult                2 1.0 9.5242e-02 1.7 2.11e+07 1.1 9.2e+01 2.4e+05 0.0e+00  0  0  0  0  0   4 23 50 80  0  5153
MatSolve               3 1.0 1.1227e-01 1.7 3.13e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  6483
MatLUFactorNum         1 1.0 1.7762e-01 1.4 1.75e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   8 19  0  0  0  2286
MatILUFactorSym        1 1.0 1.3570e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   5  0  0  0  4     0
MatAssemblyBegin       1 1.0 3.2052e-013021.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   8  0  0  0  7     0
MatAssemblyEnd         1 1.0 1.9790e-01 1.2 0.00e+00 0.0 9.2e+01 6.0e+04 8.0e+00  0  0  0  0  0   9  0 50 20 29     0
MatGetRowIJ            1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.7504e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  0  0  0  7     0
KSPSetUp               2 1.0 5.5541e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 6.6664e-01 1.0 9.28e+07 1.1 9.2e+01 2.4e+05 1.2e+01  0  0  0  0  0  33100 50 80 43  3238
VecDot                 2 1.0 5.4068e-02 9.8 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  4  0  0  7  1410
VecDotNorm2            1 1.0 5.0182e-02 6.4 6.54e+06 1.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   1  7  0  0 11  3038
VecNorm                2 1.0 1.5866e-0121.8 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   4  4  0  0  7   480
VecCopy                2 1.0 9.0349e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.6930e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPBYCZ             2 1.0 2.2837e-02 3.0 6.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0  6676
VecWAXPY               2 1.0 1.9680e-02 2.7 3.27e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  3874
VecAssemblyBegin       2 1.0 4.1746e-02243.9 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   1  0  0  0 21     0
VecAssemblyEnd         2 1.0 1.3113e-05 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 1.3828e-03 3.6 0.00e+00 0.0 9.2e+01 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 5.0416e-0228.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
PCSetUp                2 1.0 3.2991e-01 1.6 1.75e+07 1.1 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0  14 19  0  0 18  1231
PCSetUpOnBlocks        1 1.0 3.2973e-01 1.6 1.75e+07 1.1 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0  14 19  0  0 11  1231
PCApply                3 1.0 1.1918e-01 1.7 3.13e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 34  0  0  0  6107

--- Event Stage 4: momentum_z

MatMult            44415 1.0 1.9922e+03 1.5 2.28e+11 1.3 1.8e+06 1.1e+05 0.0e+00 52 62 75 93  0  58 64 77 95  0  2169
MatMultAdd          6796 1.0 1.4178e+02 2.6 1.56e+10 1.5 2.7e+05 1.7e+04 0.0e+00  3  4 11  2  0   3  4 11  2  0  1887
MatMultTranspose    6796 1.0 3.1148e+02 5.0 1.56e+10 1.5 2.7e+05 1.7e+04 0.0e+00  6  4 11  2  0   6  4 11  2  0   859
MatSolve            3833 8.8 2.8625e+01 2.7 4.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  3676
MatLUFactorNum       145 1.0 3.1267e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  1878
MatILUFactorSym        1 1.0 1.3837e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin     145 1.0 2.9783e+0114.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02  1  0  0  0  2   1  0  0  0  2     0
MatAssemblyEnd       145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.9623e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog      1699 1.0 6.9188e-02 1.3 1.55e+05 0.0 0.0e+00 0.0e+00 1.7e+03  0  0  0  0 12   0  0  0  0 14     2
KSPSetUp             290 1.0 6.0805e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve             194 1.0 2.6100e+03 1.0 3.41e+11 1.2 2.3e+06 8.5e+04 1.1e+04 85 97 97 97 80  94100100100 88  2600
VecDot              1940 1.0 2.6695e+0212.3 3.17e+09 1.0 0.0e+00 0.0e+00 1.9e+03  5  1  0  0 14   6  1  0  0 15   277
VecDotNorm2          970 1.0 2.6302e+0210.2 6.35e+09 1.0 0.0e+00 0.0e+00 2.9e+03  5  2  0  0 21   6  2  0  0 23   562
VecMDot             1699 1.0 5.7786e-02 1.7 7.65e+04 0.0 0.0e+00 0.0e+00 1.7e+03  0  0  0  0 12   0  0  0  0 14     1
VecNorm             4562 1.0 1.2351e+0213.8 1.90e+09 1.0 0.0e+00 0.0e+00 4.6e+03  2  1  0  0 33   2  1  0  0 36   359
VecScale           30582 1.0 3.7316e+01 2.8 6.19e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  3810
VecCopy             8883 1.0 1.9754e+01 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             29901 1.0 2.2141e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            56067 1.0 1.5246e+02 3.7 2.48e+10 1.1 0.0e+00 0.0e+00 0.0e+00  2  8  0  0  0   3  8  0  0  0  3731
VecAYPX            54368 1.0 1.8245e+02 3.3 1.55e+10 1.1 0.0e+00 0.0e+00 0.0e+00  3  5  0  0  0   3  5  0  0  0  1948
VecAXPBYCZ          1940 1.0 4.4522e+01 2.5 6.35e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  3322
VecWAXPY            1940 1.0 4.2226e+01 2.2 3.17e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  1751
VecMAXPY            3398 1.0 7.1802e-03 2.8 1.56e+05 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    22
VecAssemblyBegin     388 1.0 1.0391e+01 7.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03  0  0  0  0  8   0  0  0  0  9     0
VecAssemblyEnd       388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult   40776 1.0 1.8958e+02 3.9 9.28e+09 1.1 0.0e+00 0.0e+00 0.0e+00  3  3  0  0  0   3  3  0  0  0  1125
VecScatterBegin    58007 1.0 1.7609e+01 5.9 0.00e+00 0.0 2.3e+06 8.5e+04 0.0e+00  0  0 97 97  0   0  0100100  0     0
VecScatterEnd      58007 1.0 1.3547e+0315.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 29  0  0  0  0  32  0  0  0  0     0
VecNormalize        3398 1.0 6.5100e+01772.4 2.34e+05 0.0 0.0e+00 0.0e+00 3.4e+03  0  0  0  0 24   0  0  0  0 27     0
PCSetUp              290 1.0 3.1397e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 5.0e+00  1  1  0  0  0   1  1  0  0  0  1871
PCSetUpOnBlocks     1844 1.0 3.1398e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 3.0e+00  1  1  0  0  0   1  1  0  0  0  1871
PCApply             2134 1.0 2.1924e+03 1.2 2.99e+11 1.3 2.2e+06 7.9e+04 5.1e+03 65 83 93 86 36  72 85 96 89 41  2629
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix    12             56   1076276336     0
       Krylov Solver     4             13        31728     0
              Vector     8             85    248594232     0
      Vector Scatter     0             12        12720     0
           Index Set     0             16      9829416     0
      Preconditioner     0             13        12540     0
              Viewer     1              0            0     0

--- Event Stage 1: poisson

              Matrix   117             76    734590640     0
      Matrix Coarsen     4              4         2480     0
       Krylov Solver    10              4       120512     0
              Vector   225            175    200940576     0
      Vector Scatter    31             22        23320     0
           Index Set    81             74       765564     0
      Preconditioner    11              4         3456     0
         PetscRandom     4              4         2464     0

--- Event Stage 2: momentum_x

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       120488     0
      Preconditioner     2              0            0     0

--- Event Stage 3: momentum_y

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122104     0
      Preconditioner     2              0            0     0

--- Event Stage 4: momentum_z

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122096     0
      Preconditioner     2              0            0     0
========================================================================================================================
Average time to get PetscTime(): 2.14577e-07
Average time for MPI_Barrier(): 1.45912e-05
Average time for zero size MPI_Send(): 1.23382e-05
#PETSc Option Table entries:
-log_summary
-poisson_pc_gamg_agg_nsmooths 1
-poisson_pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Mon Oct  1 11:36:09 2012
Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries
-----------------------------------------
Libraries compiled on Mon Oct  1 11:36:09 2012 on hpc12 
Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final
Using PETSc directory: /home/wtay/Codes/petsc-dev
Using PETSc arch: petsc-3.3-dev_shared_rel
-----------------------------------------

Using C compiler: /opt/openmpi-1.5.3/bin/mpicc  -fPIC -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90  -fPIC -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include
-----------------------------------------

Using C linker: /opt/openmpi-1.5.3/bin/mpicc
Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl 
-----------------------------------------
-------------- next part --------------

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.3-dev_shared_rel named n12-02 with 48 processors, by wtay Wed Oct  3 21:04:02 2012
Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e  HG Date: Sun Sep 30 22:42:36 2012 -0500

                         Max       Max/Min        Avg      Total 
Time (sec):           1.721e+03      1.00353   1.718e+03
Objects:              5.680e+02      1.00000   5.680e+02
Flops:                1.739e+11      1.26420   1.417e+11  6.803e+12
Flops/sec:            1.010e+08      1.26365   8.249e+07  3.960e+09
MPI Messages:         1.686e+05      3.29646   1.101e+05  5.285e+06
MPI Message Lengths:  8.665e+09      2.00778   7.622e+04  4.029e+11
MPI Reductions:       1.377e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.1566e+02  12.6%  0.0000e+00   0.0%  0.000e+00   0.0%  0.000e+00        0.0%  3.600e+01   0.3% 
 1:         poisson: 5.6307e+01   3.3%  2.1388e+11   3.1%  1.812e+05   3.4%  2.545e+03        3.3%  1.443e+03  10.5% 
 2:      momentum_x: 1.8503e+00   0.1%  2.1267e+09   0.0%  3.760e+02   0.0%  1.058e+01        0.0%  2.800e+01   0.2% 
 3:      momentum_y: 1.5023e+00   0.1%  2.1417e+09   0.0%  3.760e+02   0.0%  1.073e+01        0.0%  2.800e+01   0.2% 
 4:      momentum_z: 1.4426e+03  84.0%  6.5844e+12  96.8%  5.103e+06  96.6%  7.366e+04       96.6%  1.223e+04  88.8% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage


--- Event Stage 1: poisson

MatMult             1209 1.0 2.7780e+01 1.3 3.16e+09 1.3 1.2e+05 8.9e+04 0.0e+00  1  2  2  3  0  43 55 64 77  0  4208
MatMultAdd           180 1.0 1.5880e+00 1.8 2.18e+08 1.5 1.4e+04 1.8e+04 0.0e+00  0  0  0  0  0   2  3  7  2  0  4488
MatMultTranspose     180 1.0 3.2210e+00 3.9 2.18e+08 1.5 1.4e+04 1.8e+04 0.0e+00  0  0  0  0  0   3  3  7  2  0  2212
MatSolve              90 0.0 6.0534e-04 0.0 1.18e+05 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   196
MatLUFactorSym         1 1.0 6.1035e-05 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 5.4121e-05 6.1 1.04e+04 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   191
MatConvert             4 1.0 1.3334e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale              12 1.0 1.3946e-01 1.7 1.58e+07 1.4 4.0e+02 8.3e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  4001
MatAssemblyBegin      69 1.0 4.9522e+00 2.5 0.00e+00 0.0 1.2e+03 1.4e+04 7.4e+01  0  0  0  0  1   6  0  1  0  5     0
MatAssemblyEnd        69 1.0 1.4642e+00 1.1 0.00e+00 0.0 6.7e+03 1.0e+04 2.0e+02  0  0  0  0  1   2  0  4  1 14     0
MatGetRow         961340 1.1 4.5354e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 0.0 1.4782e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 0.0 7.7963e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e-02  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             4 1.0 5.2552e-01 1.2 0.00e+00 0.0 1.9e+04 1.8e+04 2.8e+02  0  0  0  0  2   1  0 10  2 20     0
MatAXPY                4 1.0 4.2417e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             4 1.0 1.0365e+00 1.0 1.10e+07 1.4 2.8e+03 3.7e+04 9.6e+01  0  0  0  0  1   2  0  2  1  7   386
MatMatMultSym          4 1.0 7.9752e-01 1.1 0.00e+00 0.0 2.4e+03 2.9e+04 8.8e+01  0  0  0  0  1   1  0  1  1  6     0
MatMatMultNum          4 1.0 2.5070e-01 1.1 1.10e+07 1.4 4.0e+02 8.3e+04 8.0e+00  0  0  0  0  0   0  0  0  0  1  1594
MatPtAP                4 1.0 3.2046e+00 1.0 4.77e+08 3.1 5.9e+03 6.9e+04 1.1e+02  0  0  0  0  1   6  4  3  3  7  2670
MatPtAPSymbolic        4 1.0 1.9027e+00 1.1 0.00e+00 0.0 5.3e+03 5.7e+04 1.0e+02  0  0  0  0  1   3  0  3  2  7     0
MatPtAPNumeric         4 1.0 1.3268e+00 1.0 4.77e+08 3.1 6.0e+02 1.7e+05 8.0e+00  0  0  0  0  0   2  4  0  1  1  6450
MatTrnMatMult          4 1.0 8.2181e+00 1.0 5.30e+08 4.6 2.7e+03 4.7e+05 1.2e+02  0  0  0  0  1  15 10  1 10  8  2722
MatGetLocalMat        20 1.0 3.5593e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01  0  0  0  0  0   1  0  0  0  2     0
MatGetBrAoCol         12 1.0 2.3268e-01 2.4 0.00e+00 0.0 2.8e+03 1.2e+05 1.6e+01  0  0  0  0  0   0  0  2  3  1     0
MatGetSymTrans         8 1.0 4.9805e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        85 1.0 6.8444e-01 1.7 1.06e+08 1.1 0.0e+00 0.0e+00 8.5e+01  0  0  0  0  1   1  2  0  0  6  6731
KSPSetUp              11 1.0 1.4524e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  1     0
KSPSolve               1 1.0 5.5045e+01 1.0 5.31e+09 1.3 1.8e+05 7.4e+04 1.4e+03  3  3  3  3 10  98100100100 99  3886
VecDot                44 1.0 2.4764e+00 3.6 3.73e+07 1.1 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  0   3  1  0  0  3   677
VecDotNorm2           22 1.0 2.5759e+00 4.2 7.47e+07 1.1 0.0e+00 0.0e+00 6.6e+01  0  0  0  0  0   3  2  0  0  5  1302
VecMDot               85 1.0 5.4579e-01 2.1 5.29e+07 1.1 0.0e+00 0.0e+00 8.5e+01  0  0  0  0  1   1  1  0  0  6  4221
VecNorm              157 1.0 2.4986e+0011.5 3.01e+07 1.1 0.0e+00 0.0e+00 1.6e+02  0  0  0  0  1   2  1  0  0 11   535
VecScale             854 1.0 5.2014e-01 2.8 9.18e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  7690
VecCopy              231 1.0 2.2006e-01 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               843 1.0 2.7842e-01 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY             1489 1.0 1.6388e+00 2.8 3.47e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  7  0  0  0  9226
VecAYPX             1440 1.0 2.2286e+00 2.6 2.16e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  4  0  0  0  4229
VecAXPBYCZ            44 1.0 6.8125e-01 2.2 7.47e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  2  0  0  0  4924
VecWAXPY              44 1.0 7.3601e-01 2.3 3.73e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  1  0  0  0  2279
VecMAXPY             134 1.0 2.0552e-01 1.9 6.25e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0 13246
VecAssemblyBegin      97 1.0 3.6385e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+02  0  0  0  0  2   1  0  0  0 20     0
VecAssemblyEnd        97 1.0 3.5381e-04 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult    1124 1.0 2.1477e+00 2.4 1.35e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  3  0  0  0  2740
VecScatterBegin     1672 1.0 4.9030e-01 3.5 0.00e+00 0.0 1.7e+05 7.0e+04 0.0e+00  0  0  3  3  0   1  0 92 86  0     0
VecScatterEnd       1672 1.0 1.9463e+0113.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0  16  0  0  0  0     0
VecSetRandom           4 1.0 4.0139e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize         134 1.0 2.1754e+0016.8 1.59e+07 1.1 0.0e+00 0.0e+00 1.3e+02  0  0  0  0  1   2  0  0  0  9   318
PCSetUp                2 1.0 1.8591e+01 1.0 9.41e+08 1.3 4.2e+04 7.0e+04 1.2e+03  1  1  1  1  8  33 19 23 22 80  2241
PCSetUpOnBlocks       45 1.0 6.5732e-04 3.5 1.04e+04 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0    16
PCApply               45 1.0 2.9939e+01 1.1 4.01e+09 1.3 1.4e+05 7.0e+04 1.5e+02  2  2  3  2  1  51 70 75 71 10  5025
PCGAMGgraph_AGG        4 1.0 2.3984e+00 1.0 1.10e+07 1.4 1.2e+03 4.2e+04 7.6e+01  0  0  0  0  1   4  0  1  0  5   167
PCGAMGcoarse_AGG       4 1.0 9.2185e+00 1.0 5.30e+08 4.6 2.4e+04 7.9e+04 4.8e+02  1  0  0  0  3  16 10 13 14 33  2427
PCGAMGProl_AGG         4 1.0 9.8128e-01 1.0 0.00e+00 0.0 3.2e+03 4.6e+04 1.1e+02  0  0  0  0  1   2  0  2  1  8     0
PCGAMGPOpt_AGG         4 1.0 2.8071e+00 1.0 2.53e+08 1.2 6.8e+03 6.4e+04 2.1e+02  0  0  0  0  2   5  5  4  3 15  3682

--- Event Stage 2: momentum_x

MatMult                2 1.0 1.0542e-01 2.0 1.08e+07 1.2 1.9e+02 2.4e+05 0.0e+00  0  0  0  0  0   3 23 50 80  0  4611
MatSolve               3 1.0 8.1705e-02 2.0 1.59e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3 33  0  0  0  8719
MatLUFactorNum         1 1.0 1.4021e-01 2.1 8.82e+06 1.2 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   5 19  0  0  0  2814
MatILUFactorSym        1 1.0 1.3817e-01 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   4  0  0  0  4     0
MatAssemblyBegin       1 1.0 7.5712e-014946.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0  27  0  0  0  7     0
MatAssemblyEnd         1 1.0 1.8089e-01 1.2 0.00e+00 0.0 1.9e+02 5.9e+04 8.0e+00  0  0  0  0  0   9  0 50 20 29     0
MatGetRowIJ            1 1.0 1.0014e-0510.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.1192e-02 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  7     0
KSPSetUp               2 1.0 6.2056e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 6.0202e-01 1.0 4.74e+07 1.1 1.9e+02 2.4e+05 1.2e+01  0  0  0  0  0  32100 50 80 43  3533
VecDot                 2 1.0 4.6937e-02 3.8 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   2  4  0  0  7  1624
VecDotNorm2            1 1.0 4.7157e-02 6.1 3.39e+06 1.1 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   2  7  0  0 11  3233
VecNorm                2 1.0 2.2409e-0134.8 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   9  4  0  0  7   340
VecCopy                2 1.0 9.5789e-03 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.3558e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPBYCZ             2 1.0 2.1303e-02 3.6 3.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0  7157
VecWAXPY               2 1.0 1.6192e-02 2.7 1.70e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  4708
VecAssemblyBegin       2 1.0 5.5579e-02156.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   2  0  0  0 21     0
VecAssemblyEnd         2 1.0 1.5020e-05 7.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 1.5790e-03 2.4 0.00e+00 0.0 1.9e+02 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 5.5989e-0228.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
PCSetUp                2 1.0 3.0002e-01 2.4 8.82e+06 1.2 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   9 19  0  0 18  1315
PCSetUpOnBlocks        1 1.0 2.9975e-01 2.4 8.82e+06 1.2 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   9 19  0  0 11  1316
PCApply                3 1.0 8.5135e-02 1.9 1.59e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3 33  0  0  0  8367

--- Event Stage 3: momentum_y

MatMult                2 1.0 1.0478e-01 2.0 1.09e+07 1.2 1.9e+02 2.4e+05 0.0e+00  0  0  0  0  0   4 23 50 80  0  4684
MatSolve               3 1.0 7.1359e-02 1.6 1.61e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   4 34  0  0  0 10078
MatLUFactorNum         1 1.0 1.3938e-01 1.9 8.90e+06 1.2 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   6 19  0  0  0  2856
MatILUFactorSym        1 1.0 1.1233e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   5  0  0  0  4     0
MatAssemblyBegin       1 1.0 3.2594e-0124.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0  13  0  0  0  7     0
MatAssemblyEnd         1 1.0 1.8197e-01 1.1 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00  0  0  0  0  0  11  0 50 20 29     0
MatGetRowIJ            1 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.0951e-02 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   1  0  0  0  7     0
KSPSetUp               2 1.0 6.4847e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
KSPSolve               1 1.0 5.6900e-01 1.0 4.78e+07 1.1 1.9e+02 2.4e+05 1.2e+01  0  0  0  0  0  37100 50 80 43  3764
VecDot                 2 1.0 4.5465e-02 3.5 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   2  4  0  0  7  1677
VecDotNorm2            1 1.0 5.0043e-02 3.8 3.39e+06 1.1 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   3  7  0  0 11  3047
VecNorm                2 1.0 1.7585e-0133.5 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   8  4  0  0  7   434
VecCopy                2 1.0 6.0210e-03 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 7 1.0 1.7671e-02 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPBYCZ             2 1.0 1.4564e-02 2.4 3.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  7  0  0  0 10469
VecWAXPY               2 1.0 1.6496e-02 2.7 1.70e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  4  0  0  0  4621
VecAssemblyBegin       2 1.0 4.9739e-02159.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   2  0  0  0 21     0
VecAssemblyEnd         2 1.0 1.7881e-05 6.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin        2 1.0 1.1569e-0219.1 0.00e+00 0.0 1.9e+02 2.4e+05 0.0e+00  0  0  0  0  0   0  0 50 80  0     0
VecScatterEnd          2 1.0 5.5051e-0223.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
PCSetUp                2 1.0 2.7300e-01 2.1 8.90e+06 1.2 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0  12 19  0  0 18  1458
PCSetUpOnBlocks        1 1.0 2.7282e-01 2.1 8.90e+06 1.2 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0  12 19  0  0 11  1459
PCApply                3 1.0 7.5869e-02 1.6 1.61e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   4 34  0  0  0  9479

--- Event Stage 4: momentum_z

MatMult            42907 1.0 9.8638e+02 1.3 1.13e+11 1.3 4.1e+06 9.0e+04 0.0e+00 51 62 78 92  0  61 64 81 95  0  4250
MatMultAdd          6564 1.0 7.0460e+01 2.2 7.96e+09 1.5 4.9e+05 1.8e+04 0.0e+00  2  4  9  2  0   3  4 10  2  0  3688
MatMultTranspose    6564 1.0 1.2860e+02 4.9 7.96e+09 1.5 4.9e+05 1.8e+04 0.0e+00  3  4  9  2  0   4  4 10  2  0  2021
MatSolve            3717 8.5 1.1711e+01 1.4 2.33e+09 1.2 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  8879
MatLUFactorNum       145 1.0 1.4473e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  3979
MatILUFactorSym        1 1.0 1.2902e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin     145 1.0 2.5208e+01 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02  1  0  0  0  2   1  0  0  0  2     0
MatAssemblyEnd       145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.9673e-02 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog      1641 1.0 1.7774e-01 1.4 1.76e+05 0.0 0.0e+00 0.0e+00 1.6e+03  0  0  0  0 12   0  0  0  0 13     1
KSPSetUp             290 1.0 7.1497e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve             194 1.0 1.3382e+03 1.0 1.68e+11 1.3 5.1e+06 7.6e+04 1.1e+04 78 97 97 97 78  93100100100 88  4920
VecDot              1882 1.0 8.1310e+01 5.0 1.60e+09 1.1 0.0e+00 0.0e+00 1.9e+03  3  1  0  0 14   3  1  0  0 15   882
VecDotNorm2          941 1.0 8.2061e+01 4.1 3.19e+09 1.1 0.0e+00 0.0e+00 2.8e+03  3  2  0  0 21   3  2  0  0 23  1748
VecMDot             1641 1.0 1.6827e-01 1.5 8.70e+04 0.0 0.0e+00 0.0e+00 1.6e+03  0  0  0  0 12   0  0  0  0 13     1
VecNorm             4417 1.0 6.4342e+01 9.1 9.63e+08 1.1 0.0e+00 0.0e+00 4.4e+03  1  1  0  0 32   1  1  0  0 36   672
VecScale           29538 1.0 1.9634e+01 3.2 3.16e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  7001
VecCopy             8593 1.0 1.0218e+01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             28915 1.0 1.3080e+01 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            54153 1.0 6.1113e+01 3.0 1.26e+10 1.1 0.0e+00 0.0e+00 0.0e+00  2  8  0  0  0   3  8  0  0  0  8997
VecAYPX            52512 1.0 8.6173e+01 2.7 7.89e+09 1.1 0.0e+00 0.0e+00 0.0e+00  4  5  0  0  0   4  5  0  0  0  3988
VecAXPBYCZ          1882 1.0 2.8966e+01 2.1 3.19e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  4953
VecWAXPY            1882 1.0 2.8427e+01 2.1 1.60e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  2524
VecMAXPY            3282 1.0 6.8157e-03 3.1 1.77e+05 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    26
VecAssemblyBegin     388 1.0 8.7212e+0011.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03  0  0  0  0  8   0  0  0  0 10     0
VecAssemblyEnd       388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult   39384 1.0 8.7410e+01 2.7 4.73e+09 1.1 0.0e+00 0.0e+00 0.0e+00  3  3  0  0  0   4  3  0  0  0  2359
VecScatterBegin    56035 1.0 2.0136e+01 4.1 0.00e+00 0.0 5.1e+06 7.6e+04 0.0e+00  1  0 97 97  0   1  0100100  0     0
VecScatterEnd      56035 1.0 6.7774e+0213.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18  0  0  0  0  22  0  0  0  0     0
VecNormalize        3282 1.0 4.7531e+01301.1 2.66e+05 0.0 0.0e+00 0.0e+00 3.3e+03  0  0  0  0 24   0  0  0  0 27     0
PCSetUp              290 1.0 1.4621e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 5.0e+00  1  1  0  0  0   1  1  0  0  0  3939
PCSetUpOnBlocks     1786 1.0 1.4621e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 3.0e+00  1  1  0  0  0   1  1  0  0  0  3939
PCApply             2076 1.0 1.1022e+03 1.1 1.48e+11 1.3 4.9e+06 7.0e+04 4.9e+03 61 82 93 86 36  73 85 97 89 40  5071
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix    12             56    568064932     0
       Krylov Solver     4             13        31728     0
              Vector     8             85    129979032     0
      Vector Scatter     0             12        12720     0
           Index Set     0             16      5102648     0
      Preconditioner     0             13        12540     0
              Viewer     1              0            0     0

--- Event Stage 1: poisson

              Matrix   117             76    368468980     0
      Matrix Coarsen     4              4         2480     0
       Krylov Solver    10              4       120512     0
              Vector   225            175    107847696     0
      Vector Scatter    31             22        23320     0
           Index Set    81             74       737472     0
      Preconditioner    11              4         3456     0
         PetscRandom     4              4         2464     0

--- Event Stage 2: momentum_x

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       120488     0
      Preconditioner     2              0            0     0

--- Event Stage 3: momentum_y

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122104     0
      Preconditioner     2              0            0     0

--- Event Stage 4: momentum_z

              Matrix     1              0            0     0
       Krylov Solver     1              0            0     0
              Vector    10              1         1504     0
      Vector Scatter     1              0            0     0
           Index Set     5              2       122096     0
      Preconditioner     2              0            0     0
========================================================================================================================
Average time to get PetscTime(): 1.19209e-07
Average time for MPI_Barrier(): 4.45843e-05
Average time for zero size MPI_Send(): 1.93516e-05
#PETSc Option Table entries:
-log_summary
-poisson_pc_gamg_agg_nsmooths 1
-poisson_pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Mon Oct  1 11:36:09 2012
Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries
-----------------------------------------
Libraries compiled on Mon Oct  1 11:36:09 2012 on hpc12 
Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final
Using PETSc directory: /home/wtay/Codes/petsc-dev
Using PETSc arch: petsc-3.3-dev_shared_rel
-----------------------------------------

Using C compiler: /opt/openmpi-1.5.3/bin/mpicc  -fPIC -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90  -fPIC -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include
-----------------------------------------

Using C linker: /opt/openmpi-1.5.3/bin/mpicc
Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl 
-----------------------------------------


More information about the petsc-users mailing list