[petsc-users] Investigate parallel code to improve parallelism

TAY Wee Beng zonexo at gmail.com
Sun Feb 28 19:08:05 CST 2016


Hi,

I've attached the files for x cells running y procs. hypre is called 
natively I'm not sure if PETSc catches it.

Thanks

On 29/2/2016 1:11 AM, Barry Smith wrote:
>    As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved.
>
>    Barry
>
>> On Feb 28, 2016, at 10:29 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>
>> On 27/2/2016 12:53 AM, Barry Smith wrote:
>>>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>
>>>> On 26/2/2016 11:32 PM, Barry Smith wrote:
>>>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min.
>>>>>     This is actually pretty good!
>>>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs.
>> Hi,
>>
>> I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate:
>>
>> 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf.
>>
>> 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf.
>>
>> So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs.
>>
>> So now my qn is how accurate is this type of calculation, especially wrt to PETSc?
>>
>> Similarly, for weak scaling, is it accurate?
>>
>> Can I argue that this estimation is not suitable for PETSc or hypre?
>>
>> Thanks
>>
>>
>>>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it?
>>>>>    Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well.
>>>>>
>>>>>     Barry
>>>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts?
>>>     You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful.
>>>
>>>    Barry
>>>
>>>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well.
>>>>>>
>>>>>> -- 
>>>>>> Thank you.
>>>>>>
>>>>>> Yours sincerely,
>>>>>>
>>>>>> TAY wee-beng
>>>>>>
>> <temp.pdf>

-- 
Thank you

Yours sincerely,

TAY wee-beng

-------------- next part --------------
  0.000000000000000E+000  0.600000000000000        17.5000000000000     
   120.000000000000       0.000000000000000E+000  0.250000000000000     
   1.00000000000000       0.400000000000000                0     -400000
 AB,AA,BB   -2.78150003711926        2.76500003633555     
   2.78150003711926        2.70650003355695     
 size_x,size_y,size_z          100         172         171
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           0           1          22
           1          29           1       63800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           1          23          44
           1          29       63801      127600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           2          45          66
           1          29      127601      191400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           3          67          88
           1          29      191401      255200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           4          89         109
           1          29      255201      316100
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           5         110         130
           1          29      316101      377000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           6         131         151
           1          29      377001      437900
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           7         152         172
           1          29      437901      498800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           8           1          22
          30          58      498801      562600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           9          23          44
          30          58      562601      626400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          10          45          66
          30          58      626401      690200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          11          67          88
          30          58      690201      754000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          12          89         109
          30          58      754001      814900
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          13         110         130
          30          58      814901      875800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          14         131         151
          30          58      875801      936700
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          15         152         172
          30          58      936701      997600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          16           1          22
          59          87      997601     1061400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          17          23          44
          59          87     1061401     1125200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          18          45          66
          59          87     1125201     1189000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          19          67          88
          59          87     1189001     1252800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          20          89         109
          59          87     1252801     1313700
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          21         110         130
          59          87     1313701     1374600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          22         131         151
          59          87     1374601     1435500
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          23         152         172
          59          87     1435501     1496400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          24           1          22
          88         115     1496401     1558000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          25          23          44
          88         115     1558001     1619600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          26          45          66
          88         115     1619601     1681200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          27          67          88
          88         115     1681201     1742800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          28          89         109
          88         115     1742801     1801600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          29         110         130
          88         115     1801601     1860400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          30         131         151
          88         115     1860401     1919200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          31         152         172
          88         115     1919201     1978000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          32           1          22
         116         143     1978001     2039600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          33          23          44
         116         143     2039601     2101200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          34          45          66
         116         143     2101201     2162800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          35          67          88
         116         143     2162801     2224400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          36          89         109
         116         143     2224401     2283200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          37         110         130
         116         143     2283201     2342000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          38         131         151
         116         143     2342001     2400800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          39         152         172
         116         143     2400801     2459600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          40           1          22
         144         171     2459601     2521200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          41          23          44
         144         171     2521201     2582800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          42          45          66
         144         171     2582801     2644400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          43          67          88
         144         171     2644401     2706000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          44          89         109
         144         171     2706001     2764800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          45         110         130
         144         171     2764801     2823600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          46         131         151
         144         171     2823601     2882400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          47         152         172
         144         171     2882401     2941200
 body_cg_ini  0.850000999999998       9.999999998273846E-007
   6.95771875020604     
        3104  surfaces with wrong vertex ordering
 Warning - length difference between element and cell
 max_element_length,min_element_length,min_delta
  7.847540176996057E-002  3.349995610000001E-002  3.500000000000000E-002
 maximum ngh_surfaces and ngh_vertics are           28          10
 minimum ngh_surfaces and ngh_vertics are           14           5
 min IIB_cell_no           0
 max IIB_cell_no         160
 final initial IIB_cell_no        8000
 min I_cell_no           0
 max I_cell_no         152
 final initial I_cell_no        7600
 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
        8000        7600        8000        7600
 IIB_I_cell_no_uvw_total1        2230        2227        2166        1930
        1926        1847
    1      0.01411765      0.30104754      0.32529731      1.15440698 -0.30539502E+03 -0.29715696E+02  0.29394159E+07
    2      0.00973086      0.41244275      0.45087918      1.22116705 -0.34883625E+03 -0.24811937E+02  0.29392112E+07
    3      0.00918174      0.45383297      0.51181352      1.27753799 -0.35798168E+03 -0.24574571E+02  0.29391681E+07
    4      0.00885756      0.47398509      0.55172076      1.31014211 -0.36230580E+03 -0.25263557E+02  0.29391476E+07
    5      0.00872228      0.48832349      0.57971242      1.32671388 -0.36519380E+03 -0.26103818E+02  0.29391332E+07
  300      0.00157913      4.12249858      6.52091323      3.82250060 -0.19591967E+04  0.43743742E+05  0.29433413E+07
  600      0.00164417      3.72487671      5.92911552      5.37207663 -0.52948695E+03  0.20658440E+05  0.29461349E+07
  900      0.00135887      4.43677663      5.89462632      4.98710090 -0.86683472E+04 -0.13481580E+05  0.29508693E+07
 1200      0.00164451      4.13807017      6.05251254      4.63339167 -0.98993801E+04  0.17681878E+05  0.29569013E+07
 1500      0.00150966      3.40576464      5.20747744      4.68945079 -0.16263794E+05  0.26012616E+05  0.29604137E+07
 1800      0.00132562      3.67104160     10.44718173      4.20460908 -0.18839985E+05 -0.18296757E+05  0.29677661E+07
 escape_time reached, so abort
 cd_cl_cs_mom_implicit1
 -0.892081024888283       -1.32621439664773      -5.584506755655451E-003
  0.350447346081129      -9.754897341839677E-002  -2.09284777911273     
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.6.3_static_rel named n12-04 with 48 processors, by wtay Sat Feb 27 16:15:12 2016
Using Petsc Release Version 3.6.3, Dec, 03, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           3.252e+03      1.00001   3.252e+03
Objects:              2.008e+04      1.00000   2.008e+04
Flops:                1.590e+11      1.09643   1.523e+11  7.310e+12
Flops/sec:            4.891e+07      1.09643   4.683e+07  2.248e+09
MPI Messages:         8.531e+04      1.93829   7.180e+04  3.446e+06
MPI Message Lengths:  4.573e+09      2.08162   5.352e+04  1.844e+11
MPI Reductions:       4.427e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 3.2521e+03 100.0%  7.3101e+12 100.0%  3.446e+06 100.0%  5.352e+04      100.0%  4.427e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecDot              3998 1.0 2.8182e+01 3.2 1.53e+09 1.1 0.0e+00 0.0e+00 4.0e+03  0  1  0  0  9   0  1  0  0  9  2503
VecDotNorm2         1999 1.0 2.3131e+01 5.3 1.53e+09 1.1 0.0e+00 0.0e+00 2.0e+03  0  1  0  0  5   0  1  0  0  5  3050
VecNorm             3998 1.0 1.6799e+01 3.4 1.53e+09 1.1 0.0e+00 0.0e+00 4.0e+03  0  1  0  0  9   0  1  0  0  9  4200
VecCopy             3998 1.0 4.9803e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             12002 1.0 1.1035e+01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPBYCZ          3998 1.0 9.5659e+00 1.1 3.06e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0 14751
VecWAXPY            3998 1.0 1.0318e+01 1.1 1.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  6838
VecAssemblyBegin    3998 1.0 1.1847e+01 8.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04  0  0  0  0 27   0  0  0  0 27     0
VecAssemblyEnd      3998 1.0 1.0632e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin    16002 1.0 1.3053e+01 1.4 0.00e+00 0.0 2.6e+06 7.0e+04 0.0e+00  0  0 76100  0   0  0 76100  0     0
VecScatterEnd      16002 1.0 2.4842e+01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatMult             3998 1.0 1.5690e+02 1.2 5.86e+10 1.1 6.6e+05 1.2e+05 0.0e+00  4 37 19 43  0   4 37 19 43  0 17137
MatSolve            5997 1.0 2.3313e+02 1.1 8.47e+10 1.1 0.0e+00 0.0e+00 0.0e+00  7 53  0  0  0   7 53  0  0  0 16729
MatLUFactorNum       104 1.0 2.2413e+01 1.2 6.54e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0 13280
MatILUFactorSym        1 1.0 1.9445e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               1 1.0 5.4134e-02 1.5 7.43e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6293
MatAssemblyBegin     105 1.0 2.0050e+0010.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd       105 1.0 2.4443e+00 1.1 0.00e+00 0.0 3.3e+02 3.0e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 8.1062e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 8.0609e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp             105 1.0 1.2339e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            1999 1.0 4.6014e+02 1.0 1.59e+11 1.1 6.6e+05 1.2e+05 1.0e+04 14100 19 43 23  14100 19 43 23 15886
PCSetUp              208 1.0 2.2611e+01 1.2 6.54e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0 13163
PCSetUpOnBlocks     1999 1.0 4.2470e-01 1.2 6.29e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6739
PCApply             5997 1.0 2.5994e+02 1.1 9.11e+10 1.1 0.0e+00 0.0e+00 0.0e+00  8 57  0  0  0   8 57  0  0  0 16138
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector  4032           4032     30946112     0
      Vector Scatter  2010             15      3596160     0
              Matrix     4              4    182686524     0
    Distributed Mesh  2003              8        39680     0
Star Forest Bipartite Graph  4006             16        13696     0
     Discrete System  2003              8         6784     0
           Index Set  4013           4013     15434720     0
   IS L to G Mapping  2003              8      2092944     0
       Krylov Solver     2              2         2296     0
      Preconditioner     2              2         1896     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 1.01566e-05
Average time for zero size MPI_Send(): 7.45555e-06
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
-----------------------------------------
Libraries compiled on Thu Jan  7 04:05:35 2016 on hpc12 
Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core
Using PETSc directory: /home/wtay/Codes/petsc-3.6.3
Using PETSc arch: petsc-3.6.3_static_rel
-----------------------------------------

Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc  -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90  -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include
-----------------------------------------

Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc
Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl 
-----------------------------------------

-------------- next part --------------
  0.000000000000000E+000  0.600000000000000        17.5000000000000     
   120.000000000000       0.000000000000000E+000  0.250000000000000     
   1.00000000000000       0.400000000000000                0     -400000
 AB,AA,BB   -2.78150003711926        2.76500003633555     
   2.78150003711926        2.70650003355695     
 size_x,size_y,size_z          100         172         171
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           0           1          29
           1          43           1      124700
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           1          30          58
           1          43      124701      249400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           2          59          87
           1          43      249401      374100
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           3          88         116
           1          43      374101      498800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           4         117         144
           1          43      498801      619200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           5         145         172
           1          43      619201      739600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           6           1          29
          44          86      739601      864300
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           7          30          58
          44          86      864301      989000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           8          59          87
          44          86      989001     1113700
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           9          88         116
          44          86     1113701     1238400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          10         117         144
          44          86     1238401     1358800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          11         145         172
          44          86     1358801     1479200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          12           1          29
          87         129     1479201     1603900
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          13          30          58
          87         129     1603901     1728600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          14          59          87
          87         129     1728601     1853300
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          15          88         116
          87         129     1853301     1978000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          16         117         144
          87         129     1978001     2098400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          17         145         172
          87         129     2098401     2218800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          18           1          29
         130         171     2218801     2340600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          19          30          58
         130         171     2340601     2462400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          20          59          87
         130         171     2462401     2584200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          21          88         116
         130         171     2584201     2706000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          22         117         144
         130         171     2706001     2823600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          23         145         172
         130         171     2823601     2941200
 body_cg_ini  0.850000999999998       9.999999998273846E-007
   6.95771875020604     
        3104  surfaces with wrong vertex ordering
 Warning - length difference between element and cell
 max_element_length,min_element_length,min_delta
  7.847540176996057E-002  3.349995610000001E-002  3.500000000000000E-002
 maximum ngh_surfaces and ngh_vertics are           28          12
 minimum ngh_surfaces and ngh_vertics are           14           5
 min IIB_cell_no           0
 max IIB_cell_no         229
 final initial IIB_cell_no       11450
 min I_cell_no           0
 max I_cell_no         200
 final initial I_cell_no       10000
 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
       11450       10000       11450       10000
 IIB_I_cell_no_uvw_total1        2230        2227        2166        1930
        1926        1847
    1      0.01411765      0.30104754      0.32529731      1.15440698 -0.30539502E+03 -0.29715696E+02  0.29394159E+07
    2      0.00973086      0.41244573      0.45086899      1.22116550 -0.34890134E+03 -0.25062690E+02  0.29392110E+07
    3      0.00918177      0.45383616      0.51179402      1.27757073 -0.35811483E+03 -0.25027396E+02  0.29391677E+07
    4      0.00885764      0.47398774      0.55169119      1.31019526 -0.36250500E+03 -0.25910050E+02  0.29391470E+07
    5      0.00872241      0.48832538      0.57967282      1.32679047 -0.36545763E+03 -0.26947216E+02  0.29391325E+07
  300      0.00163886      4.27898628      6.83028522      3.60837060 -0.19609891E+04  0.43984454E+05  0.29435194E+07
  600      0.00160193      3.91014241      4.97460210      5.10461274 -0.61092521E+03  0.18910563E+05  0.29467790E+07
  900      0.00150521      3.27352854      5.85427996      4.49166453 -0.89281765E+04 -0.12171584E+05  0.29507471E+07
 1200      0.00165280      3.05922213      7.37243530      5.16434634 -0.10954640E+05  0.22049957E+05  0.29575213E+07
 1500      0.00153718      3.54908044      5.42918256      4.84940953 -0.16430153E+05  0.24407130E+05  0.29608940E+07
 1800      0.00155455      3.30956962      8.35799538      4.50638757 -0.20003619E+05 -0.20349497E+05  0.29676102E+07
 escape_time reached, so abort
 cd_cl_cs_mom_implicit1
  -1.29348921431473       -2.44525665200003      -0.238725356553914     
  0.644444280391413      -3.056662699041206E-002  -2.91791118488116     
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.6.3_static_rel named n12-09 with 24 processors, by wtay Sat Feb 27 16:58:01 2016
Using Petsc Release Version 3.6.3, Dec, 03, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           5.791e+03      1.00001   5.791e+03
Objects:              2.008e+04      1.00000   2.008e+04
Flops:                3.129e+11      1.06806   3.066e+11  7.360e+12
Flops/sec:            5.402e+07      1.06807   5.295e+07  1.271e+09
MPI Messages:         8.298e+04      1.89703   6.585e+04  1.580e+06
MPI Message Lengths:  6.456e+09      2.05684   7.780e+04  1.229e+11
MPI Reductions:       4.427e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 5.7911e+03 100.0%  7.3595e+12 100.0%  1.580e+06 100.0%  7.780e+04      100.0%  4.427e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecDot              3998 1.0 1.1437e+02 2.3 2.99e+09 1.1 0.0e+00 0.0e+00 4.0e+03  1  1  0  0  9   1  1  0  0  9   617
VecDotNorm2         1999 1.0 1.0442e+02 2.6 2.99e+09 1.1 0.0e+00 0.0e+00 2.0e+03  1  1  0  0  5   1  1  0  0  5   676
VecNorm             3998 1.0 8.5426e+01 2.2 2.99e+09 1.1 0.0e+00 0.0e+00 4.0e+03  1  1  0  0  9   1  1  0  0  9   826
VecCopy             3998 1.0 7.3321e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             12002 1.0 1.2399e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPBYCZ          3998 1.0 1.8118e+01 1.4 5.98e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  7788
VecWAXPY            3998 1.0 1.6979e+01 1.3 2.99e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4155
VecAssemblyBegin    3998 1.0 4.1001e+01 5.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04  0  0  0  0 27   0  0  0  0 27     0
VecAssemblyEnd      3998 1.0 1.4657e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin    16002 1.0 1.9519e+01 1.5 0.00e+00 0.0 1.2e+06 1.0e+05 0.0e+00  0  0 77100  0   0  0 77100  0     0
VecScatterEnd      16002 1.0 1.3223e+02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
MatMult             3998 1.0 3.0904e+02 1.3 1.15e+11 1.1 3.0e+05 1.7e+05 0.0e+00  5 37 19 43  0   5 37 19 43  0  8700
MatSolve            5997 1.0 3.9285e+02 1.4 1.67e+11 1.1 0.0e+00 0.0e+00 0.0e+00  6 54  0  0  0   6 54  0  0  0 10040
MatLUFactorNum       104 1.0 4.2097e+01 1.2 1.30e+10 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  7190
MatILUFactorSym        1 1.0 2.9875e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               1 1.0 1.3492e-01 3.3 1.45e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2525
MatAssemblyBegin     105 1.0 5.9000e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd       105 1.0 4.7665e+00 1.1 0.00e+00 0.0 1.5e+02 4.3e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 3.6001e-0518.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.6249e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp             105 1.0 2.7945e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            1999 1.0 9.1973e+02 1.0 3.13e+11 1.1 3.0e+05 1.7e+05 1.0e+04 16100 19 43 23  16100 19 43 23  8001
PCSetUp              208 1.0 4.2401e+01 1.2 1.30e+10 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  7138
PCSetUpOnBlocks     1999 1.0 7.2389e-01 1.2 1.25e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4020
PCApply             5997 1.0 4.4054e+02 1.3 1.80e+11 1.1 0.0e+00 0.0e+00 0.0e+00  6 58  0  0  0   6 58  0  0  0  9634
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector  4032           4032     53827712     0
      Vector Scatter  2010             15      7012720     0
              Matrix     4              4    359683260     0
    Distributed Mesh  2003              8        39680     0
Star Forest Bipartite Graph  4006             16        13696     0
     Discrete System  2003              8         6784     0
           Index Set  4013           4013     25819112     0
   IS L to G Mapping  2003              8      3919440     0
       Krylov Solver     2              2         2296     0
      Preconditioner     2              2         1896     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 1.90735e-07
Average time for MPI_Barrier(): 7.20024e-06
Average time for zero size MPI_Send(): 1.83781e-06
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
-----------------------------------------
Libraries compiled on Thu Jan  7 04:05:35 2016 on hpc12 
Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core
Using PETSc directory: /home/wtay/Codes/petsc-3.6.3
Using PETSc arch: petsc-3.6.3_static_rel
-----------------------------------------

Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc  -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90  -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include
-----------------------------------------

Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc
Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl 
-----------------------------------------

-------------- next part --------------
  0.000000000000000E+000  0.600000000000000        17.5000000000000     
   120.000000000000       0.000000000000000E+000  0.250000000000000     
   1.00000000000000       0.400000000000000                0     -400000
 AB,AA,BB   -3.06400005053729        3.04150004946860     
   3.06400005053729        2.98650004685624     
 size_x,size_y,size_z          128         217         210
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           0           1          28
           1          35           1      125440
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           1          29          55
           1          35      125441      246400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           2          56          82
           1          35      246401      367360
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           3          83         109
           1          35      367361      488320
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           4         110         136
           1          35      488321      609280
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           5         137         163
           1          35      609281      730240
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           6         164         190
           1          35      730241      851200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           7         191         217
           1          35      851201      972160
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           8           1          28
          36          70      972161     1097600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           9          29          55
          36          70     1097601     1218560
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          10          56          82
          36          70     1218561     1339520
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          11          83         109
          36          70     1339521     1460480
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          12         110         136
          36          70     1460481     1581440
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          13         137         163
          36          70     1581441     1702400
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          14         164         190
          36          70     1702401     1823360
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          15         191         217
          36          70     1823361     1944320
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          16           1          28
          71         105     1944321     2069760
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          17          29          55
          71         105     2069761     2190720
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          18          56          82
          71         105     2190721     2311680
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          19          83         109
          71         105     2311681     2432640
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          20         110         136
          71         105     2432641     2553600
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          21         137         163
          71         105     2553601     2674560
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          22         164         190
          71         105     2674561     2795520
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          23         191         217
          71         105     2795521     2916480
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          24           1          28
         106         140     2916481     3041920
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          25          29          55
         106         140     3041921     3162880
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          26          56          82
         106         140     3162881     3283840
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          27          83         109
         106         140     3283841     3404800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          28         110         136
         106         140     3404801     3525760
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          29         137         163
         106         140     3525761     3646720
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          30         164         190
         106         140     3646721     3767680
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          31         191         217
         106         140     3767681     3888640
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          32           1          28
         141         175     3888641     4014080
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          33          29          55
         141         175     4014081     4135040
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          34          56          82
         141         175     4135041     4256000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          35          83         109
         141         175     4256001     4376960
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          36         110         136
         141         175     4376961     4497920
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          37         137         163
         141         175     4497921     4618880
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          38         164         190
         141         175     4618881     4739840
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          39         191         217
         141         175     4739841     4860800
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          40           1          28
         176         210     4860801     4986240
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          41          29          55
         176         210     4986241     5107200
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          42          56          82
         176         210     5107201     5228160
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          43          83         109
         176         210     5228161     5349120
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          44         110         136
         176         210     5349121     5470080
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          45         137         163
         176         210     5470081     5591040
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          46         164         190
         176         210     5591041     5712000
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          47         191         217
         176         210     5712001     5832960
 body_cg_ini  0.850000999999998       9.999999998273846E-007
   6.95771875020604     
        3104  surfaces with wrong vertex ordering
 Warning - length difference between element and cell
 max_element_length,min_element_length,min_delta
  7.847540176996057E-002  3.349995610000001E-002  2.600000000000000E-002
 maximum ngh_surfaces and ngh_vertics are           15           5
 minimum ngh_surfaces and ngh_vertics are            9           3
 min IIB_cell_no           0
 max IIB_cell_no         293
 final initial IIB_cell_no       14650
 min I_cell_no           0
 max I_cell_no         330
 final initial I_cell_no       16500
 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
       14650       16500       14650       16500
 IIB_I_cell_no_uvw_total1        4065        4064        3990        4692
        4628        4632
    1      0.01043478      0.34272324      0.35569220      1.16440147 -0.68429429E+03 -0.71309583E+02  0.58290411E+07
    2      0.00697380      0.42763102      0.50182401      1.25425283 -0.77750642E+03 -0.58636989E+02  0.58286176E+07
    3      0.00648381      0.45520992      0.57545722      1.32431608 -0.79650320E+03 -0.55533364E+02  0.58285305E+07
    4      0.00611631      0.46806365      0.62117944      1.37204042 -0.80501258E+03 -0.54624483E+02  0.58284901E+07
    5      0.00592372      0.47659245      0.65166569      1.40276768 -0.81047260E+03 -0.54497615E+02  0.58284626E+07
  300      0.00115066      3.67033285      6.63969480      3.71347160  0.43279356E+04  0.80255072E+05  0.58398256E+07
  600      0.00122543      3.75818573      5.23852131      5.46557154 -0.66201226E+04  0.11004125E+06  0.58427867E+07
  900      0.00103133      3.87890472      6.22189985      4.38435056 -0.71743226E+03  0.42678349E+05  0.58534173E+07
 1200      0.00112141      3.50676323      7.96027994      5.21426706 -0.16564613E+05 -0.28161927E+05  0.58651131E+07
 1500      0.00116845      5.55355360      4.80276829      4.70049706 -0.22633631E+05 -0.12833717E+05  0.58692098E+07
 1800      0.00106663      3.60921979      9.26223511      4.67749400 -0.30654568E+05  0.62643544E+05  0.58867013E+07
 escape_time reached, so abort
 cd_cl_cs_mom_implicit1
  0.515674932183890       -1.12997638014196       0.113834045912081     
  7.478720062203850E-002 -1.299985230479712E-002 -0.226394142710332     
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.6.3_static_rel named n12-04 with 48 processors, by wtay Sat Feb 27 18:01:18 2016
Using Petsc Release Version 3.6.3, Dec, 03, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           6.333e+03      1.00000   6.333e+03
Objects:              2.008e+04      1.00000   2.008e+04
Flops:                3.132e+11      1.04090   3.039e+11  1.459e+13
Flops/sec:            4.945e+07      1.04090   4.799e+07  2.303e+09
MPI Messages:         8.482e+04      1.92729   7.176e+04  3.444e+06
MPI Message Lengths:  7.117e+09      1.99989   8.514e+04  2.933e+11
MPI Reductions:       4.427e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.3329e+03 100.0%  1.4588e+13 100.0%  3.444e+06 100.0%  8.514e+04      100.0%  4.427e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecDot              3998 1.0 4.4692e+01 2.5 3.01e+09 1.0 0.0e+00 0.0e+00 4.0e+03  0  1  0  0  9   0  1  0  0  9  3131
VecDotNorm2         1999 1.0 3.7771e+01 3.2 3.01e+09 1.0 0.0e+00 0.0e+00 2.0e+03  0  1  0  0  5   0  1  0  0  5  3704
VecNorm             3998 1.0 6.2085e+01 5.2 3.01e+09 1.0 0.0e+00 0.0e+00 4.0e+03  1  1  0  0  9   1  1  0  0  9  2254
VecCopy             3998 1.0 8.7522e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             12002 1.0 1.5554e+01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPBYCZ          3998 1.0 1.9816e+01 1.1 6.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0 14122
VecWAXPY            3998 1.0 2.0635e+01 1.1 3.01e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  6781
VecAssemblyBegin    3998 1.0 1.4403e+01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04  0  0  0  0 27   0  0  0  0 27     0
VecAssemblyEnd      3998 1.0 1.6043e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin    16002 1.0 2.8572e+01 1.3 0.00e+00 0.0 2.6e+06 1.1e+05 0.0e+00  0  0 76100  0   0  0 76100  0     0
VecScatterEnd      16002 1.0 5.2833e+01 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatMult             3998 1.0 3.1279e+02 1.1 1.14e+11 1.0 6.6e+05 1.9e+05 0.0e+00  5 37 19 43  0   5 37 19 43  0 17084
MatSolve            5997 1.0 4.8329e+02 1.1 1.68e+11 1.0 0.0e+00 0.0e+00 0.0e+00  7 53  0  0  0   7 53  0  0  0 16147
MatLUFactorNum       104 1.0 9.8655e+01 1.6 1.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  6084
MatILUFactorSym        1 1.0 3.9025e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               1 1.0 1.0553e-01 1.4 1.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6415
MatAssemblyBegin     105 1.0 2.1250e+00 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd       105 1.0 4.9930e+00 1.1 0.00e+00 0.0 3.3e+02 4.8e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 5.0068e-06 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.0131e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp             105 1.0 4.2514e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            1999 1.0 9.8702e+02 1.0 3.13e+11 1.0 6.6e+05 1.9e+05 1.0e+04 16100 19 43 23  16100 19 43 23 14779
PCSetUp              208 1.0 9.9050e+01 1.6 1.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  6060
PCSetUpOnBlocks     1999 1.0 1.3700e+00 1.4 1.22e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4213
PCApply             5997 1.0 5.8749e+02 1.2 1.81e+11 1.0 0.0e+00 0.0e+00 0.0e+00  8 58  0  0  0   8 58  0  0  0 14295
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector  4032           4032     54303424     0
      Vector Scatter  2010             15      7054512     0
              Matrix     4              4    361394700     0
    Distributed Mesh  2003              8        39680     0
Star Forest Bipartite Graph  4006             16        13696     0
     Discrete System  2003              8         6784     0
           Index Set  4013           4013     27475284     0
   IS L to G Mapping  2003              8      3991872     0
       Krylov Solver     2              2         2296     0
      Preconditioner     2              2         1896     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 6.2561e-05
Average time for zero size MPI_Send(): 1.70867e-06
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
-----------------------------------------
Libraries compiled on Thu Jan  7 04:05:35 2016 on hpc12 
Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core
Using PETSc directory: /home/wtay/Codes/petsc-3.6.3
Using PETSc arch: petsc-3.6.3_static_rel
-----------------------------------------

Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc  -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90  -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include
-----------------------------------------

Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc
Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl 
-----------------------------------------

-------------- next part --------------
  0.000000000000000E+000  0.600000000000000        17.5000000000000     
   120.000000000000       0.000000000000000E+000  0.250000000000000     
   1.00000000000000       0.400000000000000                0     -400000
 AB,AA,BB   -2.51050002424745        2.47300002246629     
   2.51050002424745        2.43950002087513     
 size_x,size_y,size_z           79         137         141
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           0           1          35
           1          24           1       66360
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           1          36          69
           1          24       66361      130824
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           2          70         103
           1          24      130825      195288
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           3         104         137
           1          24      195289      259752
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           4           1          35
          25          48      259753      326112
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           5          36          69
          25          48      326113      390576
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           6          70         103
          25          48      390577      455040
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           7         104         137
          25          48      455041      519504
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           8           1          35
          49          72      519505      585864
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end           9          36          69
          49          72      585865      650328
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          10          70         103
          49          72      650329      714792
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          11         104         137
          49          72      714793      779256
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          12           1          35
          73          95      779257      842851
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          13          36          69
          73          95      842852      904629
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          14          70         103
          73          95      904630      966407
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          15         104         137
          73          95      966408     1028185
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          16           1          35
          96         118     1028186     1091780
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          17          36          69
          96         118     1091781     1153558
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          18          70         103
          96         118     1153559     1215336
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          19         104         137
          96         118     1215337     1277114
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          20           1          35
         119         141     1277115     1340709
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          21          36          69
         119         141     1340710     1402487
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          22          70         103
         119         141     1402488     1464265
 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end          23         104         137
         119         141     1464266     1526043
 body_cg_ini  0.850000999999998       9.999999998273846E-007
   6.95771875020604     
        3104  surfaces with wrong vertex ordering
 Warning - length difference between element and cell
 max_element_length,min_element_length,min_delta
  7.847540176996057E-002  3.349995610000001E-002  4.700000000000000E-002
 maximum ngh_surfaces and ngh_vertics are           47          22
 minimum ngh_surfaces and ngh_vertics are           22           9
 min IIB_cell_no           0
 max IIB_cell_no         112
 final initial IIB_cell_no        5600
 min I_cell_no           0
 max I_cell_no          96
 final initial I_cell_no        4800
 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
        5600        4800        5600        4800
 IIB_I_cell_no_uvw_total1        1221        1206        1212         775
         761         751
    1      0.01904762      0.28410536      0.31610359      1.14440147 -0.14430869E+03 -0.13111542E+02  0.15251948E+07
    2      0.01348578      0.34638018      0.42392119      1.23447223 -0.16528393E+03 -0.10238827E+02  0.15250907E+07
    3      0.01252674      0.38305826      0.49569053      1.27891383 -0.16912542E+03 -0.95950253E+01  0.15250695E+07
    4      0.01199639      0.41337279      0.54168038      1.29584768 -0.17048065E+03 -0.94814301E+01  0.15250602E+07
    5      0.01165251      0.43544137      0.57347276      1.30255981 -0.17129184E+03 -0.95170304E+01  0.15250538E+07
  300      0.00236362      3.56353622      5.06727508      4.03923148 -0.78697893E+03  0.15046453E+05  0.15263125E+07
  600      0.00253142      2.94537779      5.74258126      4.71794271 -0.38271069E+04 -0.49150195E+04  0.15289768E+07
  900      0.00220341      3.10439489      6.70144317      4.01105348 -0.71943943E+04  0.13728311E+05  0.15320532E+07
 1200      0.00245748      3.53496741      7.33163591      4.01935315 -0.85017750E+04 -0.77550358E+04  0.15350351E+07
 1500      0.00244299      3.71751725      5.93463559      4.12005108 -0.95364451E+04  0.81223334E+04  0.15373061E+07
 1800      0.00237474      3.49908653      5.20866314      4.69712853 -0.10382365E+05 -0.18966840E+04  0.15385160E+07
 escape_time reached, so abort
 cd_cl_cs_mom_implicit1
  -1.03894256791350       -1.53179673343374       6.737940408853320E-002
  0.357464909626058      -0.103698436387821       -2.42688484514611     
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./a.out on a petsc-3.6.3_static_rel named n12-09 with 24 processors, by wtay Sat Feb 27 16:09:41 2016
Using Petsc Release Version 3.6.3, Dec, 03, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.922e+03      1.00001   2.922e+03
Objects:              2.008e+04      1.00000   2.008e+04
Flops:                1.651e+11      1.08049   1.582e+11  3.797e+12
Flops/sec:            5.652e+07      1.08049   5.414e+07  1.299e+09
MPI Messages:         8.293e+04      1.89333   6.588e+04  1.581e+06
MPI Message Lengths:  4.109e+09      2.03497   4.964e+04  7.849e+10
MPI Reductions:       4.427e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.9219e+03 100.0%  3.7965e+12 100.0%  1.581e+06 100.0%  4.964e+04      100.0%  4.427e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecDot              3998 1.0 4.4655e+01 5.1 1.59e+09 1.1 0.0e+00 0.0e+00 4.0e+03  1  1  0  0  9   1  1  0  0  9   820
VecDotNorm2         1999 1.0 4.0603e+01 7.6 1.59e+09 1.1 0.0e+00 0.0e+00 2.0e+03  1  1  0  0  5   1  1  0  0  5   902
VecNorm             3998 1.0 3.0557e+01 6.2 1.59e+09 1.1 0.0e+00 0.0e+00 4.0e+03  1  1  0  0  9   1  1  0  0  9  1198
VecCopy             3998 1.0 4.4206e+00 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             12002 1.0 9.3725e+00 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPBYCZ          3998 1.0 9.1178e+00 1.5 3.18e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  8030
VecWAXPY            3998 1.0 9.3186e+00 1.5 1.59e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  3928
VecAssemblyBegin    3998 1.0 1.5680e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04  0  0  0  0 27   0  0  0  0 27     0
VecAssemblyEnd      3998 1.0 1.1443e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin    16002 1.0 9.0984e+00 1.4 0.00e+00 0.0 1.2e+06 6.4e+04 0.0e+00  0  0 77100  0   0  0 77100  0     0
VecScatterEnd      16002 1.0 4.4821e+01 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatMult             3998 1.0 1.4268e+02 1.3 6.05e+10 1.1 3.0e+05 1.1e+05 0.0e+00  4 37 19 43  0   4 37 19 43  0  9753
MatSolve            5997 1.0 2.0469e+02 1.4 8.84e+10 1.1 0.0e+00 0.0e+00 0.0e+00  6 53  0  0  0   6 53  0  0  0  9921
MatLUFactorNum       104 1.0 2.2332e+01 1.1 6.70e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  6922
MatILUFactorSym        1 1.0 1.0867e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               1 1.0 3.8305e-02 1.9 7.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4603
MatAssemblyBegin     105 1.0 2.0776e+00 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd       105 1.0 2.4702e+00 1.1 0.00e+00 0.0 1.5e+02 2.8e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 4.0531e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 7.1249e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp             105 1.0 9.8758e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            1999 1.0 4.1857e+02 1.0 1.65e+11 1.1 3.0e+05 1.1e+05 1.0e+04 14100 19 43 23  14100 19 43 23  9070
PCSetUp              208 1.0 2.2440e+01 1.1 6.70e+09 1.1 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  6888
PCSetUpOnBlocks     1999 1.0 2.7087e-01 1.1 6.44e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5487
PCApply             5997 1.0 2.3123e+02 1.3 9.50e+10 1.1 0.0e+00 0.0e+00 0.0e+00  6 58  0  0  0   6 58  0  0  0  9444
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector  4032           4032     31782464     0
      Vector Scatter  2010             15      3738624     0
              Matrix     4              4    190398024     0
    Distributed Mesh  2003              8        39680     0
Star Forest Bipartite Graph  4006             16        13696     0
     Discrete System  2003              8         6784     0
           Index Set  4013           4013     14715400     0
   IS L to G Mapping  2003              8      2137148     0
       Krylov Solver     2              2         2296     0
      Preconditioner     2              2         1896     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 8.15392e-06
Average time for zero size MPI_Send(): 1.12454e-05
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
-----------------------------------------
Libraries compiled on Thu Jan  7 04:05:35 2016 on hpc12 
Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core
Using PETSc directory: /home/wtay/Codes/petsc-3.6.3
Using PETSc arch: petsc-3.6.3_static_rel
-----------------------------------------

Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc  -wd1572 -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90  -O3   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include
-----------------------------------------

Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc
Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90
Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl 
-----------------------------------------



More information about the petsc-users mailing list