[petsc-users] Scaling problem when cores > 600
TAY wee-beng
zonexo at gmail.com
Sun Mar 4 21:23:34 CST 2018
On 1/3/2018 12:14 PM, Smith, Barry F. wrote:
>
>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>
>> On 1/3/2018 12:10 AM, Matthew Knepley wrote:
>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>> Hi,
>>>
>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min.
>>>
>>> So how can I find the bottleneck? Any recommended steps?
>>>
>>> For any performance question, we need to see the output of -log_view for all test cases.
>> Hi,
>>
>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code.
>>
>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE?
> Yes, just send the logs.
>
Hi,
I have attached the logs, with the number indicating the no. of cores
used. Some of the new results are different from the previous runs,
although I'm using the same cluster.
Thanks for the help.
>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES.
>>
>> If I use boomeramg, can log_view be used in this case?
>>
>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge.
>>
>> Thanks.
>>>
>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case.
>>>
>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>> --
>>> Thank you very much
>>>
>>> Yours sincerely,
>>>
>>> ================================================
>>> TAY Wee-Beng 郑伟明 (Zheng Weiming)
>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>> linkedin: www.linkedin.com/in/tay-weebeng
>>> ================================================
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
-------------- next part --------------
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 1 1
min IIB_cell_no 0
max IIB_cell_no 403
IIB_cell_no_sum 88423
min equal_size 1320
max equal_size 1728
min I_cell_no 0
max I_cell_no 1872
I_cell_no_sum 361178
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
403 1872 403 1872 88423 361178
IIB_equal_cell_no_u1_max 403
I_equal_cell_no_u1_max 1872
IIB_I_cell_no_uvw_total1 88423 0 0 361178
0 0
size(IIB_cell_u),IIB_cell_no_max_cur 403 14024
local IIB_cells size exceed, to increase size
size(I_cell_u),I_cell_no_max_cur 1872 87639
local I_cells size exceed, to increase size
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
size(IIB_cell_u),IIB_cell_no_max_cur 403 711
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 1066
size(I_cell_u),I_cell_no_max_cur 1872 4320
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 6480
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 2 88332 0 0
361141 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 6480
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 3 87862 0 0
361081 0 0
IIB_equal_cell_no_u1_max 1066
size(I_cell_u),I_cell_no_max_cur 6480 7854
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 4 87184 0 0
361110 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 5 87458 0 0
361001 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 6 88029 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 7 88405 0 0
361129 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 8 88387 0 0
361154 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 9 88029 0 0
361130 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
size(I_global_cell_u1),I_global_cell_no_u1_max_cur 361178 361221
I global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 10 87355 0 0
361221 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 11 87375 0 0
361183 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 12 87922 0 0
361095 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 13 88331 0 0
361102 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
size(IIB_global_cell_u1),IIB_global_cell_no_u1_max_cur 88423 88440
IIB global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 14 88440 0 0
361060 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 15 88169 0 0
361207 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 16 87545 0 0
361053 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 17 87244 0 0
360987 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 18 87775 0 0
361160 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 19 88292 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 20 88499 0 0
361268 0 0
IIB_equal_cell_no_u1_max 1066
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 21 88244 0 0
361100 0 0
size(IIB_cell_u),IIB_cell_no_max_cur 1066 1204
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 1806
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 22 87725 0 0
361209 0 0
IIB_equal_cell_no_u1_max 1806
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 23 87181 0 0
361181 0 0
IIB_equal_cell_no_u1_max 1806
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 24 87649 0 0
361235 0 0
IIB_equal_cell_no_u1_max 1806
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 25 88185 0 0
361107 0 0
IIB_equal_cell_no_u1_max 1806
I_equal_cell_no_u1_max 11781
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 21036 131458
time,IIB_I_cell_no_uvw_total1 26 88423 0 0
361178 0 0
Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final
_I_equal_no1,final_I_no1,final_I_global_no1
2257 26295 165825 14726 164322 677288
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 1 1
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
26295 164322 2257 14726 165825 677288
IIB_I_cell_no_uvw_total1 88423 88466 88458 361178
361141 363281
1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00
escape_time reached, so abort
cd_cl_cs_mom_implicit1
-4.818844959784658E-002 1.155032241933243E-002 -3.33542637601672
-2.585963889758650E-002 -0.144586029122393 5.34245487853127
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
./a.out on a petsc-3.7.6_intel_rel named std0808 with 960 processors, by tsltaywb Thu Mar 1 15:50:31 2018
Using Petsc Release Version 3.7.6, Apr, 24, 2017
Max Max/Min Avg Total
Time (sec): 5.253e+02 1.00004 5.253e+02
Objects: 1.391e+03 1.00000 1.391e+03
Flops: 2.555e+10 1.15005 2.374e+10 2.279e+13
Flops/sec: 4.864e+07 1.15005 4.519e+07 4.339e+10
MPI Messages: 4.768e+03 1.98998 4.445e+03 4.268e+06
MPI Message Lengths: 1.174e+09 2.12895 2.458e+05 1.049e+12
MPI Reductions: 2.610e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 5.2529e+02 100.0% 2.2791e+13 100.0% 4.268e+06 100.0% 2.458e+05 100.0% 2.609e+03 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecDot 198 1.0 4.2022e+00 3.6 2.56e+08 1.1 0.0e+00 0.0e+00 2.0e+02 0 1 0 0 8 0 1 0 0 8 54441
VecDotNorm2 99 1.0 3.6140e+00 5.1 2.56e+08 1.1 0.0e+00 0.0e+00 9.9e+01 0 1 0 0 4 0 1 0 0 4 63301
VecNorm 198 1.0 2.4231e+00 2.3 2.56e+08 1.1 0.0e+00 0.0e+00 2.0e+02 0 1 0 0 8 0 1 0 0 8 94414
VecCopy 198 1.0 5.2438e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 603 1.0 8.9777e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPBYCZ 198 1.0 8.3300e-01 1.2 5.12e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 549272
VecWAXPY 198 1.0 8.3712e-01 1.2 2.56e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 273287
VecAssemblyBegin 198 1.0 4.8147e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 5.9e+02 1 0 0 0 23 1 0 0 0 23 0
VecAssemblyEnd 198 1.0 2.5415e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 802 1.0 1.7145e+00 2.1 0.00e+00 0.0 3.0e+06 3.5e+05 0.0e+00 0 0 70100 0 0 0 70100 0 0
VecScatterEnd 802 1.0 5.3004e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
MatMult 198 1.0 1.5229e+01 1.3 9.85e+09 1.2 7.3e+05 6.1e+05 0.0e+00 3 39 17 42 0 3 39 17 42 0 576810
MatSolve 297 1.0 2.0612e+01 1.2 1.39e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 602899
MatLUFactorNum 1 1.0 6.0222e-01 1.7 2.08e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 306761
MatILUFactorSym 1 1.0 2.3738e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 1 1.0 8.8569e-01 5.9 2.52e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 25371
MatAssemblyBegin 2 1.0 2.6881e-0111.6 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 2 1.0 4.0890e-01 1.0 0.00e+00 0.0 7.4e+03 1.5e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 1 1.0 3.6955e-0538.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 1 1.0 1.7057e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 2 1.0 7.1274e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 99 1.0 4.2205e+01 1.0 2.55e+10 1.2 7.3e+05 6.1e+05 5.1e+02 8100 17 42 19 8100 17 42 19 539477
PCSetUp 2 1.0 8.0980e-01 1.5 2.08e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 228126
PCSetUpOnBlocks 99 1.0 8.0983e-01 1.5 2.08e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 228119
PCApply 297 1.0 2.1040e+01 1.2 1.39e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 590644
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 296 296 89769992 0.
Vector Scatter 141 141 17112320 0.
Matrix 4 4 610814476 0.
Distributed Mesh 134 134 679648 0.
Star Forest Bipartite Graph 268 268 227264 0.
Discrete System 134 134 115776 0.
Index Set 275 275 19934028 0.
IS L to G Mapping 134 134 10683968 0.
Krylov Solver 2 2 2328 0.
Preconditioner 2 2 1896 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 0.
Average time for MPI_Barrier(): 2.19822e-05
Average time for zero size MPI_Send(): 1.4166e-06
#PETSc Option Table entries:
-log_view
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/source/hypre-2.11.2.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3 -openmp" --CXXFLAGS="-xHost -g -O3 -openmp" --FFLAGS="-xHost -g -O3 -openmp"
-----------------------------------------
Libraries compiled on Sat Jun 17 11:56:07 2017 on nus01
Machine characteristics: Linux-2.6.32-642.11.1.el6.x86_64-x86_64-with-redhat-6.8-Santiago
Using PETSc directory: /home/users/nus/tsltaywb/source/petsc-3.7.6
Using PETSc arch: petsc-3.7.6_intel_rel
-----------------------------------------
Using C compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc -xHost -g -O3 -openmp ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90 -xHost -g -O3 -openmp ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/include -I/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include
-----------------------------------------
Using C linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc
Using Fortran linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90
Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64_lin/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib -lHYPRE -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcoremt -lmpicxx -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -limf -lsvml -lirng -lm -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl
-----------------------------------------
======================================================================================
Resource Usage on 2018-03-01 15:50:44.967725:
JobId: 6584649.wlm01
Project: 11000550
Exit Status: 0
NCPUs Requested: 960 NCPUs Used: 960
CPU Time Used: 03:28:26
Memory Requested: 3840gb Memory Used: 46104992kb
Vmem Used: 76076532kb
Walltime requested: 00:29:00 Walltime Used: 00:09:14
Execution Nodes Used: (std0808:ncpus=24:mem=100663296kb)+(std0809:ncpus=24:mem=100663296kb)+(std0812:ncpus=24:mem=100663296kb)+(std0813:ncpus=24:mem=100663296kb)+(std0814:ncpus=24:mem=100663296kb)+(std0815:ncpus=24:mem=100663296kb)+(std0816:ncpus=24:mem=100663296kb)+(std0823:ncpus=24:mem=100663296kb)+(std0828:ncpus=24:mem=100663296kb)+(std0830:ncpus=24:mem=100663296kb)+(std0832:ncpus=24:mem=100663296kb)+(std0833:ncpus=24:mem=100663296kb)+(std0834:ncpus=24:mem=100663296kb)+(std0835:ncpus=24:mem=100663296kb)+(std0836:ncpus=24:mem=100663296kb)+(std0846:ncpus=24:mem=100663296kb)+(std0848:ncpus=24:mem=100663296kb)+(std0849:ncpus=24:mem=100663296kb)+(std0850:ncpus=24:mem=100663296kb)+(std0866:ncpus=24:mem=100663296kb)+(std0867:ncpus=24:mem=100663296kb)+(std0871:ncpus=24:mem=100663296kb)+(std1247:ncpus=24:mem=100663296kb)+(std1249:ncpus=24:mem=100663296kb)+(std1250:ncpus=24:mem=100663296kb)+(std1251:ncpus=24:mem=100663296kb)+(std1252:ncpus=24:mem=100663296kb)+(std1261:ncpus=24:mem=100663296kb)+(std1263:ncpus=24:mem=100663296kb)+(std1264:ncpus=24:mem=100663296kb)+(std1268:ncpus=24:mem=100663296kb)+(std1269:ncpus=24:mem=100663296kb)+(std1270:ncpus=24:mem=100663296kb)+(std1463:ncpus=24:mem=100663296kb)+(std1464:ncpus=24:mem=100663296kb)+(std1465:ncpus=24:mem=100663296kb)+(std1466:ncpus=24:mem=100663296kb)+(std1467:ncpus=24:mem=100663296kb)+(std1468:ncpus=24:mem=100663296kb)+(std1469:ncpus=24:mem=100663296kb)
======================================================================================
-------------- next part --------------
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 1 1
min IIB_cell_no 0
max IIB_cell_no 569
IIB_cell_no_sum 88423
min equal_size 2310
max equal_size 2880
min I_cell_no 0
max I_cell_no 2880
I_cell_no_sum 361178
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
569 2880 569 2880 88423 361178
IIB_equal_cell_no_u1_max 569
I_equal_cell_no_u1_max 2880
IIB_I_cell_no_uvw_total1 88423 0 0 361178
0 0
size(IIB_cell_u),IIB_cell_no_max_cur 569 21090
local IIB_cells size exceed, to increase size
size(I_cell_u),I_cell_no_max_cur 2880 124835
local I_cells size exceed, to increase size
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
size(IIB_cell_u),IIB_cell_no_max_cur 569 1056
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 1584
size(I_cell_u),I_cell_no_max_cur 2880 6600
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 9900
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 2 88332 0 0
361141 0 0
IIB_equal_cell_no_u1_max 1584
size(I_cell_u),I_cell_no_max_cur 9900 10163
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 3 87862 0 0
361081 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 4 87184 0 0
361110 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 5 87458 0 0
361001 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 6 88029 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 7 88405 0 0
361129 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 8 88387 0 0
361154 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 9 88029 0 0
361130 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
size(I_global_cell_u1),I_global_cell_no_u1_max_cur 361178 361221
I global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 10 87355 0 0
361221 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 11 87375 0 0
361183 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 12 87922 0 0
361095 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 13 88331 0 0
361102 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
size(IIB_global_cell_u1),IIB_global_cell_no_u1_max_cur 88423 88440
IIB global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 14 88440 0 0
361060 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 15 88169 0 0
361207 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 16 87545 0 0
361053 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 17 87244 0 0
360987 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 18 87775 0 0
361160 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 19 88292 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 20 88499 0 0
361268 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 21 88244 0 0
361100 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 22 87725 0 0
361209 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 23 87181 0 0
361181 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 24 87649 0 0
361235 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 25 88185 0 0
361107 0 0
IIB_equal_cell_no_u1_max 1584
I_equal_cell_no_u1_max 15244
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 31635 187252
time,IIB_I_cell_no_uvw_total1 26 88423 0 0
361178 0 0
Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final
_I_equal_no1,final_I_no1,final_I_global_no1
1980 39543 165825 19055 234065 677288
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 1 1
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
39543 234065 1980 19055 165825 677288
IIB_I_cell_no_uvw_total1 88423 88466 88458 361178
361141 363281
1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00
escape_time reached, so abort
cd_cl_cs_mom_implicit1
-4.799898951184345E-002 1.513905249972471E-002 -3.33855059591624
-2.804064947912442E-002 -0.140110944096023 5.34189337593670
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
./a.out on a petsc-3.7.6_intel_rel named std0808 with 600 processors, by tsltaywb Thu Mar 1 15:07:54 2018
Using Petsc Release Version 3.7.6, Apr, 24, 2017
Max Max/Min Avg Total
Time (sec): 7.076e+02 1.00000 7.076e+02
Objects: 1.391e+03 1.00000 1.391e+03
Flops: 4.082e+10 1.12066 3.822e+10 2.293e+13
Flops/sec: 5.769e+07 1.12066 5.402e+07 3.241e+10
MPI Messages: 4.756e+03 1.98497 4.387e+03 2.632e+06
MPI Message Lengths: 1.530e+09 2.09737 3.238e+05 8.523e+11
MPI Reductions: 2.610e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 7.0762e+02 100.0% 2.2934e+13 100.0% 2.632e+06 100.0% 3.238e+05 100.0% 2.609e+03 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecDot 198 1.0 1.6962e+0130.8 4.06e+08 1.1 0.0e+00 0.0e+00 2.0e+02 2 1 0 0 8 2 1 0 0 8 13487
VecDotNorm2 99 1.0 1.5047e+01105.4 4.06e+08 1.1 0.0e+00 0.0e+00 9.9e+01 2 1 0 0 4 2 1 0 0 4 15204
VecNorm 198 1.0 1.5310e+0111.1 4.06e+08 1.1 0.0e+00 0.0e+00 2.0e+02 2 1 0 0 8 2 1 0 0 8 14942
VecCopy 198 1.0 1.4051e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 603 1.0 2.6225e+00 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPBYCZ 198 1.0 2.2252e+00 2.0 8.13e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 205617
VecWAXPY 198 1.0 1.9064e+00 1.9 4.06e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 120005
VecAssemblyBegin 198 1.0 6.2482e+00 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 5.9e+02 1 0 0 0 23 1 0 0 0 23 0
VecAssemblyEnd 198 1.0 2.8229e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 802 1.0 2.6802e+00 2.0 0.00e+00 0.0 1.8e+06 4.6e+05 0.0e+00 0 0 70100 0 0 0 70100 0 0
VecScatterEnd 802 1.0 1.8956e+01 9.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
MatMult 198 1.0 3.1972e+01 1.7 1.56e+10 1.1 4.6e+05 7.9e+05 0.0e+00 3 38 17 42 0 3 38 17 42 0 274736
MatSolve 297 1.0 6.6952e+01 2.4 2.24e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 187707
MatLUFactorNum 1 1.0 1.0193e+00 1.8 3.36e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 184262
MatILUFactorSym 1 1.0 1.8219e+00 6.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 1 1.0 1.5685e+00 6.4 4.00e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14326
MatAssemblyBegin 2 1.0 5.5912e-0134.9 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 2 1.0 3.3589e-01 1.1 0.00e+00 0.0 4.6e+03 2.0e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 1 1.0 2.2888e-0524.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 1 1.0 1.6273e-0112.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 2 1.0 3.6676e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 99 1.0 9.9955e+01 1.0 4.08e+10 1.1 4.6e+05 7.9e+05 5.1e+02 14100 17 42 19 14100 17 42 19 229221
PCSetUp 2 1.0 2.7545e+00 3.2 3.36e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 68187
PCSetUpOnBlocks 99 1.0 2.7544e+00 3.2 3.36e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 68187
PCApply 297 1.0 6.7990e+01 2.4 2.24e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 184844
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 296 296 140331272 0.
Vector Scatter 141 141 26977688 0.
Matrix 4 4 977176396 0.
Distributed Mesh 134 134 679648 0.
Star Forest Bipartite Graph 268 268 227264 0.
Discrete System 134 134 115776 0.
Index Set 275 275 30744768 0.
IS L to G Mapping 134 134 16239536 0.
Krylov Solver 2 2 2328 0.
Preconditioner 2 2 1896 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 1.40667e-06
Average time for MPI_Barrier(): 1.82152e-05
Average time for zero size MPI_Send(): 1.22865e-06
#PETSc Option Table entries:
-log_view
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/source/hypre-2.11.2.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3 -openmp" --CXXFLAGS="-xHost -g -O3 -openmp" --FFLAGS="-xHost -g -O3 -openmp"
-----------------------------------------
Libraries compiled on Sat Jun 17 11:56:07 2017 on nus01
Machine characteristics: Linux-2.6.32-642.11.1.el6.x86_64-x86_64-with-redhat-6.8-Santiago
Using PETSc directory: /home/users/nus/tsltaywb/source/petsc-3.7.6
Using PETSc arch: petsc-3.7.6_intel_rel
-----------------------------------------
Using C compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc -xHost -g -O3 -openmp ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90 -xHost -g -O3 -openmp ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/include -I/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include
-----------------------------------------
Using C linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc
Using Fortran linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90
Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64_lin/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib -lHYPRE -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcoremt -lmpicxx -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -limf -lsvml -lirng -lm -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl
-----------------------------------------
======================================================================================
Resource Usage on 2018-03-01 15:08:03.135255:
JobId: 6584647.wlm01
Project: 11000324
Exit Status: 0
NCPUs Requested: 600 NCPUs Used: 600
CPU Time Used: 04:41:11
Memory Requested: 2400gb Memory Used: 59321596kb
Vmem Used: 88504076kb
Walltime requested: 00:29:00 Walltime Used: 00:12:10
Execution Nodes Used: (std0808:ncpus=24:mem=100663296kb)+(std0812:ncpus=24:mem=100663296kb)+(std0813:ncpus=24:mem=100663296kb)+(std0814:ncpus=24:mem=100663296kb)+(std0815:ncpus=24:mem=100663296kb)+(std0816:ncpus=24:mem=100663296kb)+(std0823:ncpus=24:mem=100663296kb)+(std0828:ncpus=24:mem=100663296kb)+(std0830:ncpus=24:mem=100663296kb)+(std0832:ncpus=24:mem=100663296kb)+(std0833:ncpus=24:mem=100663296kb)+(std0834:ncpus=24:mem=100663296kb)+(std0835:ncpus=24:mem=100663296kb)+(std0836:ncpus=24:mem=100663296kb)+(std0846:ncpus=24:mem=100663296kb)+(std0848:ncpus=24:mem=100663296kb)+(std0849:ncpus=24:mem=100663296kb)+(std0850:ncpus=24:mem=100663296kb)+(std0866:ncpus=24:mem=100663296kb)+(std0867:ncpus=24:mem=100663296kb)+(std0871:ncpus=24:mem=100663296kb)+(std1247:ncpus=24:mem=100663296kb)+(std1249:ncpus=24:mem=100663296kb)+(std1250:ncpus=24:mem=100663296kb)+(std1251:ncpus=24:mem=100663296kb)
======================================================================================
-------------- next part --------------
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 3
minimum ngh_surfaces and ngh_vertics are 1 1
min IIB_cell_no 0
max IIB_cell_no 919
IIB_cell_no_sum 88423
min equal_size 3960
max equal_size 4536
min I_cell_no 0
max I_cell_no 4788
I_cell_no_sum 361178
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
919 4788 919 4788 88423 361178
IIB_equal_cell_no_u1_max 919
I_equal_cell_no_u1_max 4788
IIB_I_cell_no_uvw_total1 88423 0 0 361178
0 0
size(IIB_cell_u),IIB_cell_no_max_cur 919 18928
local IIB_cells size exceed, to increase size
size(I_cell_u),I_cell_no_max_cur 4788 103062
local I_cells size exceed, to increase size
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
size(IIB_cell_u),IIB_cell_no_max_cur 919 1644
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 2466
size(I_cell_u),I_cell_no_max_cur 4788 11018
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 16527
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 2 88332 0 0
361141 0 0
IIB_equal_cell_no_u1_max 2466
size(I_cell_u),I_cell_no_max_cur 16527 17078
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 3 87862 0 0
361081 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 4 87184 0 0
361110 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 5 87458 0 0
361001 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 6 88029 0 0
361094 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 7 88405 0 0
361129 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 8 88387 0 0
361154 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 9 88029 0 0
361130 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
size(I_global_cell_u1),I_global_cell_no_u1_max_cur 361178 361221
I global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 10 87355 0 0
361221 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 11 87375 0 0
361183 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 12 87922 0 0
361095 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 13 88331 0 0
361102 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
size(IIB_global_cell_u1),IIB_global_cell_no_u1_max_cur 88423 88440
IIB global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 14 88440 0 0
361060 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 15 88169 0 0
361207 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 16 87545 0 0
361053 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 17 87244 0 0
360987 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 18 87775 0 0
361160 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 19 88292 0 0
361094 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 20 88499 0 0
361268 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 21 88244 0 0
361100 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 22 87725 0 0
361209 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 23 87181 0 0
361181 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 24 87649 0 0
361235 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 25 88185 0 0
361107 0 0
IIB_equal_cell_no_u1_max 2466
I_equal_cell_no_u1_max 25617
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 28392 154593
time,IIB_I_cell_no_uvw_total1 26 88423 0 0
361178 0 0
Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final
_I_equal_no1,final_I_no1,final_I_global_no1
3082 35490 165825 32021 193241 677288
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 3
minimum ngh_surfaces and ngh_vertics are 1 1
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
35490 193241 3082 32021 165825 677288
IIB_I_cell_no_uvw_total1 88423 88466 88458 361178
361141 363281
1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00
escape_time reached, so abort
cd_cl_cs_mom_implicit1
-4.813795806287376E-002 1.332038908467352E-002 -3.33133748261859
-2.665790156027220E-002 -0.155302336435887 5.32867788906244
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
./a.out on a petsc-3.7.6_intel_rel named std0416 with 360 processors, by tsltaywb Mon Mar 5 03:11:13 2018
Using Petsc Release Version 3.7.6, Apr, 24, 2017
Max Max/Min Avg Total
Time (sec): 1.712e+03 1.00000 1.712e+03
Objects: 1.391e+03 1.00000 1.391e+03
Flops: 6.677e+10 1.08897 6.416e+10 2.310e+13
Flops/sec: 3.901e+07 1.08898 3.749e+07 1.350e+10
MPI Messages: 4.748e+03 1.98164 4.301e+03 1.548e+06
MPI Message Lengths: 1.886e+09 2.07836 4.049e+05 6.269e+11
MPI Reductions: 2.610e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 1.7116e+03 100.0% 2.3099e+13 100.0% 1.548e+06 100.0% 4.049e+05 100.0% 2.609e+03 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecDot 198 1.0 3.3442e+01 7.6 6.61e+08 1.1 0.0e+00 0.0e+00 2.0e+02 1 1 0 0 8 1 1 0 0 8 6841
VecDotNorm2 99 1.0 2.9553e+01139.9 6.61e+08 1.1 0.0e+00 0.0e+00 9.9e+01 1 1 0 0 4 1 1 0 0 4 7741
VecNorm 198 1.0 7.5366e+01 4.3 6.61e+08 1.1 0.0e+00 0.0e+00 2.0e+02 4 1 0 0 8 4 1 0 0 8 3035
VecCopy 198 1.0 4.3418e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 603 1.0 8.7590e+00 4.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPBYCZ 198 1.0 9.1023e+00 4.9 1.32e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 50267
VecWAXPY 198 1.0 5.7899e+00 3.0 6.61e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 39512
VecAssemblyBegin 198 1.0 1.4758e+0119.7 0.00e+00 0.0 0.0e+00 0.0e+00 5.9e+02 1 0 0 0 23 1 0 0 0 23 0
VecAssemblyEnd 198 1.0 3.0422e-04 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 802 1.0 7.1661e+00 3.6 0.00e+00 0.0 1.1e+06 5.7e+05 0.0e+00 0 0 70100 0 0 0 70100 0 0
VecScatterEnd 802 1.0 6.8386e+0131.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
MatMult 198 1.0 7.7412e+01 2.5 2.54e+10 1.1 2.7e+05 9.9e+05 0.0e+00 2 38 17 42 0 2 38 17 42 0 113470
MatSolve 297 1.0 1.0990e+02 3.3 3.68e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 115822
MatLUFactorNum 1 1.0 9.4123e+0010.1 5.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 20335
MatILUFactorSym 1 1.0 3.2815e+0169.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 1 1.0 1.7278e+00 3.6 6.50e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 13005
MatAssemblyBegin 2 1.0 1.1061e+0049.9 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 2 1.0 5.7800e-01 1.1 0.00e+00 0.0 2.7e+03 2.5e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 1 1.0 1.0014e-0510.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 1 1.0 1.0545e+01437.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 2 1.0 5.6577e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 2 0 0 0 0 2 0 0 0 0 0
KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401
PCSetUp 2 1.0 3.3785e+0122.5 5.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 5665
PCSetUpOnBlocks 99 1.0 3.3785e+0122.5 5.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 5665
PCApply 297 1.0 1.1089e+02 3.2 3.68e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 55 0 0 0 4 55 0 0 0 114784
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 296 296 224346632 0.
Vector Scatter 141 141 43730752 0.
Matrix 4 4 1600573996 0.
Distributed Mesh 134 134 679648 0.
Star Forest Bipartite Graph 268 268 227264 0.
Discrete System 134 134 115776 0.
Index Set 275 275 48544868 0.
IS L to G Mapping 134 134 25300768 0.
Krylov Solver 2 2 2328 0.
Preconditioner 2 2 1896 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 0.
Average time for MPI_Barrier(): 1.64032e-05
Average time for zero size MPI_Send(): 1.16958e-06
#PETSc Option Table entries:
-log_view
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/source/hypre-2.11.2.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3 -openmp" --CXXFLAGS="-xHost -g -O3 -openmp" --FFLAGS="-xHost -g -O3 -openmp"
-----------------------------------------
Libraries compiled on Sat Jun 17 11:56:07 2017 on nus01
Machine characteristics: Linux-2.6.32-642.11.1.el6.x86_64-x86_64-with-redhat-6.8-Santiago
Using PETSc directory: /home/users/nus/tsltaywb/source/petsc-3.7.6
Using PETSc arch: petsc-3.7.6_intel_rel
-----------------------------------------
Using C compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc -xHost -g -O3 -openmp ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90 -xHost -g -O3 -openmp ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/include -I/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include
-----------------------------------------
Using C linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc
Using Fortran linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90
Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64_lin/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib -lHYPRE -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcoremt -lmpicxx -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -limf -lsvml -lirng -lm -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl
-----------------------------------------
======================================================================================
Resource Usage on 2018-03-05 03:11:18.338045:
JobId: 6584624.wlm01
Project: 11000324
Exit Status: 0
NCPUs Requested: 360 NCPUs Used: 360
CPU Time Used: 11:21:48
Memory Requested: 1440gb Memory Used: 80705036kb
Vmem Used: 109196412kb
Walltime requested: 00:29:00 Walltime Used: 00:28:52
Execution Nodes Used: (std0416:ncpus=24:mem=100663296kb)+(std0449:ncpus=24:mem=100663296kb)+(std0452:ncpus=24:mem=100663296kb)+(std0471:ncpus=24:mem=100663296kb)+(std0504:ncpus=24:mem=100663296kb)+(std0524:ncpus=24:mem=100663296kb)+(std0127:ncpus=24:mem=100663296kb)+(std0151:ncpus=24:mem=100663296kb)+(std0161:ncpus=24:mem=100663296kb)+(std0163:ncpus=24:mem=100663296kb)+(std0205:ncpus=24:mem=100663296kb)+(std0213:ncpus=24:mem=100663296kb)+(std0231:ncpus=24:mem=100663296kb)+(std0236:ncpus=24:mem=100663296kb)+(std0240:ncpus=24:mem=100663296kb)
======================================================================================
-------------- next part --------------
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 4 1
min IIB_cell_no 0
max IIB_cell_no 210
IIB_cell_no_sum 88423
min equal_size 576
max equal_size 810
min I_cell_no 0
max I_cell_no 990
I_cell_no_sum 361178
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
210 990 210 990 88423 361178
IIB_equal_cell_no_u1_max 210
I_equal_cell_no_u1_max 990
IIB_I_cell_no_uvw_total1 88423 0 0 361178
0 0
size(IIB_cell_u),IIB_cell_no_max_cur 210 7184
local IIB_cells size exceed, to increase size
size(I_cell_u),I_cell_no_max_cur 990 43594
local I_cells size exceed, to increase size
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
size(IIB_cell_u),IIB_cell_no_max_cur 210 407
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 610
size(I_cell_u),I_cell_no_max_cur 990 2160
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 3240
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 2 88332 0 0
361141 0 0
IIB_equal_cell_no_u1_max 610
size(I_cell_u),I_cell_no_max_cur 3240 3456
I_cells size exceed, to increase size
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 3 87862 0 0
361081 0 0
size(IIB_cell_u),IIB_cell_no_max_cur 610 764
IIB_cells size exceed, to increase size
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 4 87184 0 0
361110 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 5 87458 0 0
361001 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 6 88029 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 7 88405 0 0
361129 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 8 88387 0 0
361154 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 361178
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 9 88029 0 0
361130 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
size(I_global_cell_u1),I_global_cell_no_u1_max_cur 361178 361221
I global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 10 87355 0 0
361221 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 11 87375 0 0
361183 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 12 87922 0 0
361095 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 88423 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 13 88331 0 0
361102 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
size(IIB_global_cell_u1),IIB_global_cell_no_u1_max_cur 88423 88440
IIB global cells size exceed, to increase size
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 14 88440 0 0
361060 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 15 88169 0 0
361207 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 16 87545 0 0
361053 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 17 87244 0 0
360987 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 18 87775 0 0
361160 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 19 88292 0 0
361094 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 20 88499 0 0
361268 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 21 88244 0 0
361100 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 22 87725 0 0
361209 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 23 87181 0 0
361181 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 24 87649 0 0
361235 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 25 88185 0 0
361107 0 0
IIB_equal_cell_no_u1_max 1146
I_equal_cell_no_u1_max 5184
IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 132660 541831
IIB_cell_no_u1_max,I_cell_no_u1_max 10776 65391
time,IIB_I_cell_no_uvw_total1 26 88423 0 0
361178 0 0
Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final
_I_equal_no1,final_I_no1,final_I_global_no1
1432 13470 165825 6480 81738 677288
0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.400000000000000 0 -400000
AB,AA,BB,CC -3.41000006697141 3.73100008221809
3.32950006314786 3.32950006314786
size_x,size_y,size_z 792x792x307
total grid size = 192570048
recommended cores (50k / core) = 3851.40096000000
min_area,max_area,min_grid_area,ratio 9.229464518072358E-005
0.108728370237650 3.240000000000000E-004 335.581389622375
ratio bet max_area,min_grid_area not ideal
max element length should be 2.545584412271571E-002
body_cg_ini -2.750916668409947E-003 2.695116216584915E-004
6.75873529067005
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.558213960965656 4.851715434749100E-003 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 4 1
minimum ngh_surfaces and ngh_vertics are 4 1
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz
e(IIB_global_cell_u),size(I_global_cell_u)
13470 81738 1432 6480 165825 677288
IIB_I_cell_no_uvw_total1 88423 88466 88458 361178
361141 363281
1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00
escape_time reached, so abort
cd_cl_cs_mom_implicit1
-4.791973271274874E-002 1.663600030763223E-002 -3.34136260481927
-2.866268789419770E-002 -0.129330978688499 5.36615362813215
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
./a.out on a petsc-3.7.6_intel_rel named std0808 with 1920 processors, by tsltaywb Mon Mar 5 11:07:39 2018
Using Petsc Release Version 3.7.6, Apr, 24, 2017
Max Max/Min Avg Total
Time (sec): 5.634e+02 1.00002 5.634e+02
Objects: 1.391e+03 1.00000 1.391e+03
Flops: 1.326e+10 1.21860 1.170e+10 2.246e+13
Flops/sec: 2.354e+07 1.21862 2.076e+07 3.986e+10
MPI Messages: 4.792e+03 2.00000 4.515e+03 8.668e+06
MPI Message Lengths: 8.540e+08 2.18170 1.737e+05 1.506e+12
MPI Reductions: 2.610e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 5.6336e+02 100.0% 2.2458e+13 100.0% 8.668e+06 100.0% 1.737e+05 100.0% 2.609e+03 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecDot 198 1.0 2.8298e+00 2.8 1.35e+08 1.2 0.0e+00 0.0e+00 2.0e+02 0 1 0 0 8 0 1 0 0 8 80845
VecDotNorm2 99 1.0 2.3738e+00 4.1 1.35e+08 1.2 0.0e+00 0.0e+00 9.9e+01 0 1 0 0 4 0 1 0 0 4 96373
VecNorm 198 1.0 1.7269e+00 2.2 1.35e+08 1.2 0.0e+00 0.0e+00 2.0e+02 0 1 0 0 8 0 1 0 0 8 132473
VecCopy 198 1.0 2.6873e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 603 1.0 4.6638e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPBYCZ 198 1.0 4.1426e-01 1.3 2.69e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 1104480
VecWAXPY 198 1.0 4.3662e-01 1.3 1.35e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 523960
VecAssemblyBegin 198 1.0 4.0228e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.9e+02 1 0 0 0 23 1 0 0 0 23 0
VecAssemblyEnd 198 1.0 2.5105e-04 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 802 1.0 1.0690e+00 2.3 0.00e+00 0.0 6.0e+06 2.5e+05 0.0e+00 0 0 69100 0 0 0 69100 0 0
VecScatterEnd 802 1.0 3.8140e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMult 198 1.0 8.3073e+00 1.4 5.18e+09 1.2 1.5e+06 4.3e+05 0.0e+00 1 39 17 42 0 1 39 17 42 0 1057375
MatSolve 297 1.0 1.0673e+01 1.3 7.16e+09 1.2 0.0e+00 0.0e+00 0.0e+00 2 54 0 0 0 2 54 0 0 0 1133846
MatLUFactorNum 1 1.0 2.6653e-01 1.6 1.06e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 666364
MatILUFactorSym 1 1.0 1.2934e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 1 1.0 1.0649e+00 6.9 1.32e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 21102
MatAssemblyBegin 2 1.0 1.7105e-0113.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 2 1.0 2.2725e-01 1.0 0.00e+00 0.0 1.5e+04 1.1e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 1 1.0 2.0981e-0522.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 1 1.0 8.8160e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 2 1.0 5.2316e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717
PCSetUp 2 1.0 3.7846e-01 1.5 1.06e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 469294
PCSetUpOnBlocks 99 1.0 3.7846e-01 1.5 1.06e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 469297
PCApply 297 1.0 1.0885e+01 1.3 7.16e+09 1.2 0.0e+00 0.0e+00 0.0e+00 2 54 0 0 0 2 54 0 0 0 1111695
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 296 296 48826760 0.
Vector Scatter 141 141 8996640 0.
Matrix 4 4 316139068 0.
Distributed Mesh 134 134 679648 0.
Star Forest Bipartite Graph 268 268 227264 0.
Discrete System 134 134 115776 0.
Index Set 275 275 11040200 0.
IS L to G Mapping 134 134 6043100 0.
Krylov Solver 2 2 2328 0.
Preconditioner 2 2 1896 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 0.000235176
Average time for zero size MPI_Send(): 4.96457e-06
#PETSc Option Table entries:
-log_view
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/source/hypre-2.11.2.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3 -openmp" --CXXFLAGS="-xHost -g -O3 -openmp" --FFLAGS="-xHost -g -O3 -openmp"
-----------------------------------------
Libraries compiled on Sat Jun 17 11:56:07 2017 on nus01
Machine characteristics: Linux-2.6.32-642.11.1.el6.x86_64-x86_64-with-redhat-6.8-Santiago
Using PETSc directory: /home/users/nus/tsltaywb/source/petsc-3.7.6
Using PETSc arch: petsc-3.7.6_intel_rel
-----------------------------------------
Using C compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc -xHost -g -O3 -openmp ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90 -xHost -g -O3 -openmp ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/include -I/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/include -I/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/include -I/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include
-----------------------------------------
Using C linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpicc
Using Fortran linker: /app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpif90
Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/source/petsc-3.7.6/petsc-3.7.6_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.7.6_intel_rel/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -L/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -L/app/intel/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/app/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64_lin/gcc4.4 -L/app/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -Wl,-rpath,/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/5.0/intel64/lib -lHYPRE -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport -lifcoremt -lmpicxx -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -limf -lsvml -lirng -lm -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl
-----------------------------------------
======================================================================================
Resource Usage on 2018-03-05 11:08:06.400057:
JobId: 6600726.wlm01
Project: 11000550
Exit Status: 0
NCPUs Requested: 1920 NCPUs Used: 1920
CPU Time Used: 03:43:20
Memory Requested: 8400gb Memory Used: 34233432kb
Vmem Used: 66999464kb
Walltime requested: 24:00:00 Walltime Used: 00:10:08
Execution Nodes Used: (std0808:ncpus=24:mem=110100480kb)+(std0809:ncpus=24:mem=110100480kb)+(std0812:ncpus=24:mem=110100480kb)+(std0813:ncpus=24:mem=110100480kb)+(std0814:ncpus=24:mem=110100480kb)+(std0815:ncpus=24:mem=110100480kb)+(std0816:ncpus=24:mem=110100480kb)+(std0823:ncpus=24:mem=110100480kb)+(std0828:ncpus=24:mem=110100480kb)+(std0830:ncpus=24:mem=110100480kb)+(std0832:ncpus=24:mem=110100480kb)+(std0833:ncpus=24:mem=110100480kb)+(std0834:ncpus=24:mem=110100480kb)+(std0835:ncpus=24:mem=110100480kb)+(std0836:ncpus=24:mem=110100480kb)+(std0846:ncpus=24:mem=110100480kb)+(std0848:ncpus=24:mem=110100480kb)+(std0849:ncpus=24:mem=110100480kb)+(std0850:ncpus=24:mem=110100480kb)+(std0861:ncpus=24:mem=110100480kb)+(std0866:ncpus=24:mem=110100480kb)+(std0867:ncpus=24:mem=110100480kb)+(std0871:ncpus=24:mem=110100480kb)+(std0950:ncpus=24:mem=110100480kb)+(std0951:ncpus=24:mem=110100480kb)+(std0952:ncpus=24:mem=110100480kb)+(std0953:ncpus=24:mem=110100480kb)+(std0954:ncpus=24:mem=110100480kb)+(std0955:ncpus=24:mem=110100480kb)+(std0956:ncpus=24:mem=110100480kb)+(std0957:ncpus=24:mem=110100480kb)+(std0958:ncpus=24:mem=110100480kb)+(std0959:ncpus=24:mem=110100480kb)+(std0960:ncpus=24:mem=110100480kb)+(std0961:ncpus=24:mem=110100480kb)+(std0962:ncpus=24:mem=110100480kb)+(std0963:ncpus=24:mem=110100480kb)+(std0964:ncpus=24:mem=110100480kb)+(std0965:ncpus=24:mem=110100480kb)+(std0966:ncpus=24:mem=110100480kb)+(std0967:ncpus=24:mem=110100480kb)+(std0968:ncpus=24:mem=110100480kb)+(std0969:ncpus=24:mem=110100480kb)+(std0970:ncpus=24:mem=110100480kb)+(std0971:ncpus=24:mem=110100480kb)+(std0972:ncpus=24:mem=110100480kb)+(std1001:ncpus=24:mem=110100480kb)+(std1002:ncpus=24:mem=110100480kb)+(std1004:ncpus=24:mem=110100480kb)+(std1005:ncpus=24:mem=110100480kb)+(std1006:ncpus=24:mem=110100480kb)+(std1007:ncpus=24:mem=110100480kb)+(std1008:ncpus=24:mem=110100480kb)+(std1009:ncpus=24:mem=110100480kb)+(std1010:ncpus=24:mem=110100480kb)+(std1011:ncpus=24:mem=110100480kb)+(std1012:ncpus=24:mem=110100480kb)+(std1013:ncpus=24:mem=110100480kb)+(std1014:ncpus=24:mem=110100480kb)+(std1015:ncpus=24:mem=110100480kb)+(std1016:ncpus=24:mem=110100480kb)+(std1017:ncpus=24:mem=110100480kb)+(std1018:ncpus=24:mem=110100480kb)+(std1019:ncpus=24:mem=110100480kb)+(std1021:ncpus=24:mem=110100480kb)+(std1022:ncpus=24:mem=110100480kb)+(std1023:ncpus=24:mem=110100480kb)+(std1024:ncpus=24:mem=110100480kb)+(std1025:ncpus=24:mem=110100480kb)+(std1026:ncpus=24:mem=110100480kb)+(std1027:ncpus=24:mem=110100480kb)+(std1028:ncpus=24:mem=110100480kb)+(std1029:ncpus=24:mem=110100480kb)+(std1030:ncpus=24:mem=110100480kb)+(std1032:ncpus=24:mem=110100480kb)+(std1041:ncpus=24:mem=110100480kb)+(std1042:ncpus=24:mem=110100480kb)+(std1055:ncpus=24:mem=110100480kb)+(std1067:ncpus=24:mem=110100480kb)+(std1101:ncpus=24:mem=110100480kb)
======================================================================================
More information about the petsc-users
mailing list