[Nek5000-users] very low calculation speed
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Thu Sep 21 01:27:29 CDT 2017
This problem compiled with PPLIST="NEKNEK"
/----------------------------------------------------------\\
| _ __ ______ __ __ ______ ____ ____ ____ |
| / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |
| / |/ // __/ / ,< /___ \\ / / / // / / // / / / |
| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |
| /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ |
| |
|----------------------------------------------------------|
| |
| NEK5000: Open Source Spectral Element Solver |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |
| Version: 1.0rc1 / SVN r1115 |
| Web: http://nek5000.mcs.anl.gov |
| |
\\----------------------------------------------------------/
Number of processors: 192
REAL wdsize : 8
INTEGER wdsize : 4
Timer accuracy : 7.87E-07
/----------------------------------------------------------\\
| _ __ ______ __ __ ______ ____ ____ ____ |
| / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |
| / |/ // __/ / ,< /___ \\ / / / // / / // / / / |
| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |
| /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ |
| |
|----------------------------------------------------------|
| |
| NEK5000: Open Source Spectral Element Solver |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |
| Version: 1.0rc1 / SVN r1115 |
| Web: http://nek5000.mcs.anl.gov |
| |
\\----------------------------------------------------------/
Number of processors: 24
REAL wdsize : 8
INTEGER wdsize : 4
Timer accuracy : 5.25E-07
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.rea
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.re2
byte swap: F 6.543210 -2.9312772E+35
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.rea
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.re2
byte swap: F 6.543210 -2.9312772E+35
mapping elements to processors
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.map
mapping elements to processors
Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.map
0 213 213 5120 5120 NELV
3 213 213 5120 5120 NELV
4 213 213 5120 5120 NELV
5 213 213 5120 5120 NELV
1 213 213 5120 5120 NELV
2 213 213 5120 5120 NELV
6 213 213 5120 5120 NELV
7 213 213 5120 5120 NELV
8 213 213 5120 5120 NELV
9 213 213 5120 5120 NELV
10 213 213 5120 5120 NELV
11 213 213 5120 5120 NELV
element load imbalance: 1 418 419
done :: mapping elements to processors
reading mesh
14 213 213 5120 5120 NELV
RANK 0 IEG 3873 3874 3875 3876 3877 3878 3879 3880
3881 3882 3883 3884 3885 3886 3887 3888
3889 3890 3891 3892 3893 3894 3895 3896
3897 3898 3899 3900 3901 3902 3903 3904
4129 4130 4131 4132 4133 4134 4135 4136
4137 4138 4139 4140 4141 4142 4143 4144
4145 4146 4147 4148 4149 4150 4151 4152
16 214 214 5120 5120 NELV
4153 4154 4155 4156 4157 4158 4159 4160
4353 4354 4355 4356 4361 4362 4363 4364
4369 4370 4371 4372 4377 4378 4379 4380
4385 4386 4387 4388 4389 4390 4391 4392
4393 4394 4395 4396 4397 4398 4399 4400
4401 4402 4403 4404 4405 4406 4407 4408
4409 4410 4411 4412 4413 4414 4415 4416
4609 4610 4611 4612 4613 4617 4618 4619
4620 4621 4625 4626 4627 4628 4633 4634
4635 4636 4637 4638 4641 4642 4643 4644
4645 4646 4647 4648 4649 4650 4651 4652
17 214 214 5120 5120 NELV
4653 4654 4655 4656 4657 4658 4659 4660
4661 4662 4663 4664 4665 4666 4667 4668
4669 4670 4671 4672 4865 4866 4867 4868
4869 4873 4874 4875 4876 4877 4881 4882
4883 4884 4889 4890 4891 4897 4898 4899
4900 4901 4902 4903 4904 4905 4906 4907
12 213 213 5120 5120 NELV
4908 4909 4910 4911 4912 4913 4914 4915
4916 4917 4918 4919 4920 4921 4922 4923
4924 4925 4926 4927 4928
15 213 213 5120 5120 NELV
element load imbalance: 1 213 214
done :: mapping elements to processors
18 214 214 5120 5120 NELV
reading mesh
22 214 214 5120 5120 NELV
reading curved sides
13 213 213 5120 5120 NELV
reading bc for ifld 1
20 214 214 5120 5120 NELV
done :: read .re2 file
21 214 214 5120 5120 NELV
0 objects found
23 214 214 5120 5120 NELV
118 Parameters from file:
1 1.00000 p001 DENSITY
2 -8900.00 p002 VISCOS
19 214 214 5120 5120 NELV
7 1.00000 p007 RHOCP
8 -8900.00 p008 CONDUCT
11 100000.0 p011 NSTEPS
12 -3.0E-04 p012 DT
15 1000.0 p015 IOSTEP
18 5.0E-02 p018 GRID < 0 --> # cells on screen
19 -1.00000 p019 INTYPE
20 4.00000 p020 NORDER
21 1.0E-06 p021 DIVERGENCE
22 1.0E-08 p022 HELMHOLTZ
24 1.0E-02 p024 TOLREL
25 1.0E-02 p025 TOLABS
26 0.50000 p026 COURANT/NTAU
27 2.00000 p027 TORDER
28 1.00000 p028 TORDER: mesh velocity (0: p28=p27)
66 0.00000 p066 output : <0=ascii, else binary
67 0.00000 p067 restart: <0=ascii, else binary
93 20.0000 p093 Number of previous pressure solns saved
95 5.00000 p095 start projecting pressure after p95 step
99 3.00000 p099 dealiasing: <0--> off/3--> old/4--> new
101 1.00000 p101 Number of additional modes to filter
102 1.00000 p102 Dump out divergence at each time step
103 9.500000E-01 p103 weight of stabilizing filter (.01)
nelgt/nelgv/lelt: 5120 5120 4000
lx1 /lx2 /lx3 : 12 12 12
done :: read .rea file 0.54454 sec
setup mesh topology
Right-handed check complete for 5120 elements. OK.
setvert3d: 12 1772160 6892160 1772160 1772160
call usrsetvert
done :: usrsetvert
gs_setup: 340773 unique labels shared
pairwise times (avg, min, max): 0.000373997 0.000350785 0.000392795
reading curved sides
crystal router : 0.00137192 0.00134859 0.0013859
used all_to_all method: pairwise
handle bytes (avg, min, max): 2.16376e+06 2106108 2251852
buffer bytes (avg, min, max): 476643 410800 577024
setupds time 3.9776E-01 seconds 0 12 1772160 5120
8 max multiplicity
done :: setup mesh topology
call usrdat
done :: usrdat
generate geometry data
reading bc for ifld 1
done :: read .re2 file
0 objects found
118 Parameters from file:
1 1.00000 p001 DENSITY
2 -8900.00 p002 VISCOS
7 1.00000 p007 RHOCP
8 -8900.00 p008 CONDUCT
11 100000.0 p011 NSTEPS
12 -3.0E-04 p012 DT
15 1000.0 p015 IOSTEP
18 5.0E-02 p018 GRID < 0 --> # cells on screen
19 -1.00000 p019 INTYPE
20 4.00000 p020 NORDER
21 1.0E-06 p021 DIVERGENCE
22 1.0E-08 p022 HELMHOLTZ
24 1.0E-02 p024 TOLREL
25 1.0E-02 p025 TOLABS
26 0.50000 p026 COURANT/NTAU
27 2.00000 p027 TORDER
28 1.00000 p028 TORDER: mesh velocity (0: p28=p27)
66 0.00000 p066 output : <0=ascii, else binary
67 0.00000 p067 restart: <0=ascii, else binary
93 20.0000 p093 Number of previous pressure solns saved
95 5.00000 p095 start projecting pressure after p95 step
99 3.00000 p099 dealiasing: <0--> off/3--> old/4--> new
101 1.00000 p101 Number of additional modes to filter
102 1.00000 p102 Dump out divergence at each time step
103 9.500000E-01 p103 weight of stabilizing filter (.01)
nelgt/nelgv/lelt: 80320 80320 4000
lx1 /lx2 /lx3 : 12 12 12
done :: read .rea file 1.1102 sec
setup mesh topology
Right-handed check complete for 80320 elements. OK.
setvert3d: 12 26906208 107226208 26906208 26906208
call usrsetvert
done :: usrsetvert
gs_setup: 5553221 unique labels shared
pairwise times (avg, min, max): done :: generate geometry data
call usrdat2
0.0016136 0.0012603 0.00186529
crystal router : 0.00763995 0.00749459 0.00784841
used all_to_all method: pairwise
handle bytes (avg, min, max): 4.35947e+06 3848020 4681812
buffer bytes (avg, min, max): 976972 546496 1320848
setupds time 2.1430E+00 seconds 0 12 26906208 80320
8 max multiplicity
done :: setup mesh topology
call usrdat
done :: usrdat
generate geometry data
done :: generate geometry data
call usrdat2
initializing intpts(), tol= 1.000000000000000E-013
initializing intpts(), tol= 1.000000000000000E-013
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 1536 pipe_per
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
done :: usrdat2
done :: usrdat2
found 0 pipe
regenerate geometry data 1
regenerate geometry data 1
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
found 0 pipe
done :: regenerate geometry data 1
verify mesh topology
-1.00000000000000 1.00000000000000 Xrange
-1.00000000000000 1.00000000000000 Yrange
0.000000000000000E+000 4.50000000000000 Zrange
done :: verify mesh topology
IFTRAN = T
IFFLOW = T
IFHEAT = F
IFSPLIT = T
IFLOMACH = F
IFUSERVP = F
IFUSERMV = F
IFSTRS = T
IFCHAR = F
IFCYCLIC = F
IFAXIS = F
IFMVBD = F
IFMELT = F
IFMODEL = F
IFKEPS = F
IFMOAB = F
IFNEKNEK = T
IFSYNC = T
IFVCOR = T
IFINTQ = F
IFCWUZ = F
IFSWALL = F
IFGEOM = F
IFSURT = F
IFWCNO = F
IFCMT = F
IFVISC = F
IFFLTR = F
IFTMSH for field 1 = F
IFADVC for field 1 = T
IFNONL for field 1 = F
Dealiasing enabled, lxd= 18
done :: regenerate geometry data 1
verify mesh topology
-12.0000000000000 12.0000000000000 Xrange
-12.0000000000000 12.0000000000000 Yrange
0.000000000000000E+000 36.0000000000000 Zrange
done :: verify mesh topology
IFTRAN = T
IFFLOW = T
IFHEAT = F
IFSPLIT = T
IFLOMACH = F
IFUSERVP = F
IFUSERMV = F
IFSTRS = T
IFCHAR = F
IFCYCLIC = F
IFAXIS = F
IFMVBD = F
IFMELT = F
IFMODEL = F
IFKEPS = F
IFMOAB = F
IFNEKNEK = T
IFSYNC = T
IFVCOR = F
IFINTQ = F
IFCWUZ = F
IFSWALL = F
IFGEOM = F
IFSURT = F
IFWCNO = F
IFCMT = F
IFVISC = F
IFFLTR = F
IFTMSH for field 1 = F
IFADVC for field 1 = T
IFNONL for field 1 = F
Dealiasing enabled, lxd= 18
Estimated eigenvalues
EIGAA = 1.80739669073447
EIGGA = 13884400.9520853
EIGAE = 0.487387871658733
EIGAS = 2.247191011235953E-002
EIGGE = 13884400.9520853
EIGGS = 2.00000000000000
verify mesh topology
-1.00000000000000 1.00000000000000 Xrange
-1.00000000000000 1.00000000000000 Yrange
0.000000000000000E+000 4.50000000000000 Zrange
done :: verify mesh topology
E-solver strategy: 0 itr
mg_nx: 1 7 11
mg_ny: 1 7 11
mg_nz: 1 7 11
call usrsetvert
done :: usrsetvert
gs_setup: 2723 unique labels shared
pairwise times (avg, min, max): 2.96265e-05 2.85149e-05 3.04937e-05
crystal router : 1.43409e-05 1.38998e-05 1.49012e-05
all reduce : 6.69807e-05 6.60896e-05 6.82116e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 66207.7 54708 77100
buffer bytes (avg, min, max): 9284 7536 10800
setupds time 4.3750E-03 seconds 1 2 5760 5120
setvert3d: 4 103040 144000 103040 103040
call usrsetvert
done :: usrsetvert
gs_setup: 25165 unique labels shared
pairwise times (avg, min, max): 2.84771e-05 2.72989e-05 2.95877e-05
crystal router : 4.75297e-05 4.67062e-05 4.81129e-05
all reduce : 0.00061613 0.000604296 0.000622106
used all_to_all method: pairwise
handle bytes (avg, min, max): 171156 165372 179980
buffer bytes (avg, min, max): 39800 34608 47744
setupds time 2.7411E-02 seconds 2 4 103040 5120
setvert3d: 8 681600 1787520 681600 681600
call usrsetvert
done :: usrsetvert
gs_setup: 137801 unique labels shared
pairwise times (avg, min, max): 8.8045e-05 8.38041e-05 9.10997e-05
crystal router : 0.000316096 0.00030961 0.000320601
used all_to_all method: pairwise
handle bytes (avg, min, max): 886769 861564 925228
buffer bytes (avg, min, max): 197997 170992 239168
setupds time 1.0851E-01 seconds 3 8 681600 5120
setvert3d: 10 1162880 3784320 1162880 1162880
call usrsetvert
done :: usrsetvert
gs_setup: 227995 unique labels shared
pairwise times (avg, min, max): 0.000147719 0.00013988 0.000153089
crystal router : 0.000699271 0.000687408 0.000709796
used all_to_all method: pairwise
handle bytes (avg, min, max): 1.45509e+06 1415292 1515868
buffer bytes (avg, min, max): 322264 277968 389792
setupds time 1.9729E-01 seconds 4 10 1162880 5120
regenerate geometry data 1
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data 1
h1_mg_nx: 1 3 11
h1_mg_ny: 1 3 11
h1_mg_nz: 1 3 11
call usrsetvert
done :: usrsetvert
gs_setup: 2723 unique labels shared
pairwise times (avg, min, max): 2.57045e-05 2.50816e-05 2.63929e-05
crystal router : 1.31597e-05 1.27077e-05 1.35183e-05
all reduce : 6.57449e-05 6.45876e-05 6.69956e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 66207.7 54708 77100
buffer bytes (avg, min, max): 9284 7536 10800
setupds time 3.6449E-03 seconds 5 2 5760 5120
setvert3d: 4 103040 144000 103040 103040
call usrsetvert
done :: usrsetvert
gs_setup: 25165 unique labels shared
pairwise times (avg, min, max): 2.73794e-05 2.62976e-05 2.83003e-05
crystal router : 4.36117e-05 4.29153e-05 4.42028e-05
all reduce : 0.000558805 0.000551605 0.000561714
used all_to_all method: pairwise
handle bytes (avg, min, max): 171156 165372 179980
buffer bytes (avg, min, max): 39800 34608 47744
setupds time 2.3185E-02 seconds 6 4 103040 5120
setvert3d: 4 103040 144000 103040 103040
call usrsetvert
done :: usrsetvert
gs_setup: 25165 unique labels shared
pairwise times (avg, min, max): 3.27577e-05 3.13044e-05 3.38078e-05
crystal router : 4.4166e-05 4.32968e-05 4.48942e-05
all reduce : 0.000555371 0.00054791 0.000558496
used all_to_all method: pairwise
handle bytes (avg, min, max): 171156 165372 179980
buffer bytes (avg, min, max): 39800 34608 47744
setupds time 2.2302E-02 seconds 7 4 103040 5120
setvert3d: 6 328320 656000 328320 328320
call usrsetvert
done :: usrsetvert
gs_setup: 70191 unique labels shared
pairwise times (avg, min, max): 5.42253e-05 5.29051e-05 5.60999e-05
crystal router : 0.000130047 0.000128007 0.000131607
all reduce : 0.00147395 0.0014652 0.00147929
used all_to_all method: pairwise
handle bytes (avg, min, max): 458791 444924 479932
buffer bytes (avg, min, max): 103843 89872 125152
setupds time 6.3579E-02 seconds 8 6 328320 5120
setvert3d: 12 1772160 6892160 1772160 1772160
call usrsetvert
done :: usrsetvert
gs_setup: 340773 unique labels shared
pairwise times (avg, min, max): 0.000211759 0.000206304 0.00021801
crystal router : 0.00135557 0.00133109 0.00137
used all_to_all method: pairwise
handle bytes (avg, min, max): 2.16376e+06 2106108 2251852
buffer bytes (avg, min, max): 476643 410800 577024
setupds time 3.5228E-01 seconds 9 12 1772160 5120
setvert3d: 14 2509440 11356800 2509440 2509440
call usrsetvert
done :: usrsetvert
gs_setup: 476135 unique labels shared
pairwise times (avg, min, max): 0.000281647 0.000266504 0.000291586
crystal router : 0.00225826 0.0022172 0.00228479
used all_to_all method: pairwise
handle bytes (avg, min, max): 3.01277e+06 2934012 3133180
buffer bytes (avg, min, max): 661133 569488 800864
setupds time 6.1860E-01 seconds 10 14 2509440 5120
setup h1 coarse grid, nx_crs= 2
call usrsetvert
done :: usrsetvert
gs_setup: 2723 unique labels shared
pairwise times (avg, min, max): 1.48118e-05 1.41859e-05 1.58072e-05
crystal router : 1.33614e-05 1.21117e-05 1.38044e-05
all reduce : 6.4381e-05 6.33001e-05 6.53982e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 66207.7 54708 77100
buffer bytes (avg, min, max): 9284 7536 10800
Estimated eigenvalues
EIGAA = 1.396163174022414E-002
EIGGA = 26467996.3219686
EIGAE = 7.615435494667707E-003
EIGAS = 4.545454545454543E-002
EIGGE = 26467996.3219686
EIGGS = 2.00000000000000
verify mesh topology
-12.0000000000000 12.0000000000000 Xrange
-12.0000000000000 12.0000000000000 Yrange
0.000000000000000E+000 36.0000000000000 Zrange
done :: setup h1 coarse grid 0.402811050415039 sec
call usrdat3
done :: usrdat3
set initial conditions
Checking restart options: pipe_per0.f00010
nekuic (1) for ifld 1
Reading checkpoint data
0 0 OPEN: pipe_per0.f00010
byte swap: F 6.543210 -2.9312772E+35
done :: verify mesh topology
E-solver strategy: 0 itr
mg_nx: 1 7 11
mg_ny: 1 7 11
mg_nz: 1 7 11
call usrsetvert
done :: usrsetvert
gs_setup: 40651 unique labels shared
pairwise times (avg, min, max): 0.000218946 0.00018239 0.000236607
crystal router : 5.72741e-05 5.60999e-05 5.85079e-05
all reduce : 0.00148923 0.00145738 0.0015028
used all_to_all method: crystal router
handle bytes (avg, min, max): 171689 112124 221476
buffer bytes (avg, min, max): 18153.2 12304 23232
setupds time 7.1854E-02 seconds 1 2 83038 80320
setvert3d: 4 1550072 2192632 1550072 1550072
call usrsetvert
done :: usrsetvert
gs_setup: 400269 unique labels shared
pairwise times (avg, min, max): 0.000248798 0.000221109 0.000264406
crystal router : 0.000225847 0.000221992 0.000230503
used all_to_all method: crystal router
handle bytes (avg, min, max): 1.41297e+06 878540 1853668
buffer bytes (avg, min, max): 151639 98096 195920
setupds time 7.4004E-02 seconds 2 4 1550072 80320
setvert3d: 8 10330540 27679660 10330540 10330540
call usrsetvert
done :: usrsetvert
gs_setup: 2233849 unique labels shared
pairwise times (avg, min, max): 0.000514414 0.000456595 0.000552988
crystal router : 0.00232402 0.00227511 0.00238209
used all_to_all method: pairwise
handle bytes (avg, min, max): 1.77983e+06 1567956 1917012
buffer bytes (avg, min, max): 405180 225984 547728
setupds time 4.5856E-01 seconds 3 8 10330540 80320
setvert3d: 10 17643974 58767814 17643974 17643974
call usrsetvert
done :: usrsetvert
gs_setup: 3707811 unique labels shared
pairwise times (avg, min, max): 0.00074541 0.000669909 0.000782418
0 2.8664E+02 done :: Read checkpoint data
avg data-throughput = 84.5MBps
io-nodes = 1
crystal router : xyz min -1.0000 -1.0000
0.0000
uvwpt min -0.34051 -0.37479 -0.16102E-01 -0.90090E-01 0.0000
PS min 0.0000 0.0000 0.99000E+22
xyz max 1.0000 1.0000 4.5000
uvwpt max 0.32898 0.39774 1.3656 0.78843E-01 0.0000
PS max 0.0000 0.0000 -0.99000E+22
Restart: recompute geom. factors.
regenerate geometry data 1
0.00430806 0.00423291 0.00441771
used all_to_all method: pairwise
handle bytes (avg, min, max): 2.92727e+06 2581844 3147348
buffer bytes (avg, min, max): 660122 368832 892432
setupds time 1.0661E+00 seconds 4 10 17643974 80320
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data 1
regenerate geometry data 1
done :: set initial conditions
call userchk
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data 1
h1_mg_nx: 1 3 11
h1_mg_ny: 1 3 11
h1_mg_nz: 1 3 11
call usrsetvert
done :: usrsetvert
gs_setup: 40651 unique labels shared
pairwise times (avg, min, max): 0.000203656 0.000173688 0.000223613
crystal router : 5.95078e-05 5.80072e-05 6.10113e-05
all reduce : 0.00126102 0.0012404 0.00127389
used all_to_all method: crystal router
handle bytes (avg, min, max): 171689 112124 221476
buffer bytes (avg, min, max): 18153.2 12304 23232
setupds time 2.9112E-02 seconds 5 2 83038 80320
setvert3d: 4 1550072 2192632 1550072 1550072
call usrsetvert
done :: usrsetvert
gs_setup: 400269 unique labels shared
pairwise times (avg, min, max): 0.000249024 0.000214601 0.000264192
crystal router : 0.000207164 0.000203013 0.000212216
used all_to_all method: crystal router
handle bytes (avg, min, max): 1.41297e+06 878540 1853668
buffer bytes (avg, min, max): 151639 98096 195920
setupds time 5.8844E-02 seconds 6 4 1550072 80320
setvert3d: 4 1550072 2192632 1550072 1550072
call usrsetvert
done :: usrsetvert
gs_setup: 400269 unique labels shared
pairwise times (avg, min, max): 0.000250614 0.000216007 0.000266099
crystal router : 0.000206963 0.000201392 0.000213099
used all_to_all method: crystal router
handle bytes (avg, min, max): 1.41297e+06 878540 1853668
buffer bytes (avg, min, max): 151639 98096 195920
setupds time 5.8632E-02 seconds 7 4 1550072 80320
setvert3d: 6 4965906 10106386 4965906 4965906
call usrsetvert
done :: usrsetvert
gs_setup: 1131335 unique labels shared
pairwise times (avg, min, max): 0.000345911 0.000313807 0.000366807
crystal router : 0.000918429 0.000901699 0.000948811
used all_to_all method: pairwise
handle bytes (avg, min, max): 917140 806356 990804
buffer bytes (avg, min, max): 212147 117952 286736
setupds time 1.7421E-01 seconds 8 6 4965906 80320
setvert3d: 12 26906208 107226208 26906208 26906208
call usrsetvert
done :: usrsetvert
gs_setup: 5553221 unique labels shared
pairwise times (avg, min, max): 0.00103432 0.000936699 0.00110071
crystal router : 0.00765613 0.0074913 0.0078582
used all_to_all method: pairwise
handle bytes (avg, min, max): 4.35947e+06 3848020 4681812
buffer bytes (avg, min, max): 976972 546496 1320848
setupds time 1.9455E+00 seconds 9 12 26906208 80320
setvert3d: 14 38117242 176910202 38117242 38117242
call usrsetvert
done :: usrsetvert
gs_setup: 7770079 unique labels shared
pairwise times (avg, min, max): 0.00144147 0.00128691 0.0015306
crystal router : 0.0118797 0.011621 0.0121873
used all_to_all method: pairwise
handle bytes (avg, min, max): 6.07641e+06 5366484 6520404
buffer bytes (avg, min, max): 1.35573e+06 758976 1832976
setupds time 2.9645E+00 seconds 10 14 38117242 80320
setup h1 coarse grid, nx_crs= 2
call usrsetvert
done :: usrsetvert
gs_setup: 40651 unique labels shared
pairwise times (avg, min, max): 0.000223077 0.000198388 0.000240588
crystal router : 5.7518e-05 5.61953e-05 5.86987e-05
all reduce : 0.00124078 0.0012202 0.00125399
used all_to_all method: crystal router
handle bytes (avg, min, max): 171689 112124 221476
buffer bytes (avg, min, max): 18153.2 12304 23232
done :: setup h1 coarse grid 22.2913181781769 sec
call usrdat3
done :: usrdat3
set initial conditions
Checking restart options: pipe0.f00010
nekuic (1) for ifld 1
Reading checkpoint data
0 0 OPEN: pipe0.f00010
byte swap: F 6.543210 -2.9312772E+35
0 2.8664E+02 done :: Read checkpoint data
avg data-throughput = 98.5MBps
io-nodes = 1
xyz min -12.000 -12.000 0.0000
uvwpt min -0.99492 -0.76080 -0.82980 -1.4131 0.0000
PS min 0.0000 0.0000 0.99000E+22
xyz max 12.000 12.000 36.000
uvwpt max 0.81959 0.91894 1.4289 0.99220 0.0000
PS max 0.0000 0.0000 -0.99000E+22
Restart: recompute geom. factors.
regenerate geometry data 1
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data 1
done :: set initial conditions
call userchk
force z: 1.000000000000000E-005
done :: userchk
gridpoints unique/tot: 6892160 8847360
dofs: 6737280 6892160
Initial time: 0.2866404E+03
Initialization successfully completed 71.437 sec
Starting time loop ...
DT/DTCFL/DTFS/DTINIT 0.300E-03 0.000E+00 0.233-309 0.300E-03
force z: 1.000000000000000E-005
dump history points
reading hpts.in
found 80 points
initializing intpts(), tol= 1.000000000000000E-013
done :: dump history points
Building mesh for avg
Avg mesh builded
initializing intpts(), tol= 1.000000000000000E-008
call intpts
call findpts
total number of points = 1920000
failed = 0
done :: intpts
Writting interp data avg
Interp data avg written
call intpts
call findpts
total number of points = 1920000
failed = 0
done :: intpts
Writting interp data avg
Interp data avg written
call intpts
call findpts
total number of points = 1920000
failed = 0
done :: intpts
Writting interp data avg
Interp data avg written
Building the mesh for pod1
Mesh for pod1 builded
call intpts
call findpts
total number of points = 21504000
failed = 0
done :: intpts
Writting pod1 data
pod1 data written
Building the mesh for pod2
Mesh for pod2 builded
call intpts
call findpts
total number of points = 21504000
failed = 0
done :: intpts
Writting pod2 data
pod2 data written
done :: userchk
gridpoints unique/tot: 107226208 138792960
dofs: 106989839 106823343
Initial time: 0.2866404E+03
Initialization successfully completed 4728.4 sec
Starting time loop ...
DT/DTCFL/DTFS/DTINIT 0.300E-03 0.000E+00 0.236-309 0.300E-03
Step 1, t= 2.8664073E+02, DT= 3.0000000E-04, C= 0.149 0.0000E+00
0.0000E+00
Step 1, t= 2.8664073E+02, DT= 3.0000000E-04, C= 0.438 0.0000E+00
0.0000E+00
Solving for fluid T T T
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.2375 0.9500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
1.0000 1.0000 0.7625 0.0500
1.000000000000000E-008 p22 1 1
1 1.00000E-06 3.08669E-03 5.12889E-03 6.01823E-01 1 Divergence
2 1.00000E-06 1.63507E-03 5.12889E-03 3.18795E-01 1 Divergence
3 1.00000E-06 5.70865E-04 5.12889E-03 1.11304E-01 1 Divergence
4 1.00000E-06 3.19087E-04 5.12889E-03 6.22137E-02 1 Divergence
5 1.00000E-06 1.67594E-04 5.12889E-03 3.26764E-02 1 Divergence
6 1.00000E-06 8.78601E-05 5.12889E-03 1.71304E-02 1 Divergence
7 1.00000E-06 4.10072E-05 5.12889E-03 7.99534E-03 1 Divergence
8 1.00000E-06 2.09226E-05 5.12889E-03 4.07936E-03 1 Divergence
9 1.00000E-06 1.17707E-05 5.12889E-03 2.29498E-03 1 Divergence
10 1.00000E-06 6.32563E-06 5.12889E-03 1.23333E-03 1 Divergence
11 1.00000E-06 3.53710E-06 5.12889E-03 6.89642E-04 1 Divergence
12 1.00000E-06 1.96804E-06 5.12889E-03 3.83716E-04 1 Divergence
13 1.00000E-06 1.14795E-06 5.12889E-03 2.23821E-04 1 Divergence
14 1.00000E-06 7.12487E-07 5.12889E-03 1.38916E-04 1 Divergence
1 PRES gmres: 14 7.1249E-07 1.0000E-06 5.1289E-03
1.8504E+00 4.1575E+00 F
0 1.0000E-08 3.4398E-07 1.0000E-08 tol,matmod
1 Helmholtz3/fluid: 7 0.1612E-08 0.1000E-07 0.1453E-01
L1/L2 DIV(V) : -3.3387E-15 4.5580E-04
L1/L2 QTL : 0.0000E+00 0.0000E+00
L1/L2 DIV(V)-QTL: -3.3387E-15 4.5580E-04
1 2.8664E+02 6.3868E+00 Fluid done
Solving for fluid T T T
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.2375 0.9500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
1.0000 1.0000 0.7625 0.0500
> Can you please post your logfile otherwise it’s hard to tell what’s going wrong.
> On 20 Sep 2017, at 12:20, "nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users> <mailto:nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>> " <nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users> <mailto:nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>> > wrote:
> Dear Nek devs
> Usually I start some calculation with 100-150 processors and it runs with acceptable speed. But sometimes the speed of calculation is extremely low, something like 1 step per 10 hours with 50 million mesh points. It seems like all computational nodes fall on one processors, while the rest just stay without any job, despite that all requirements about nodes per processor are satisfied.
> Have you ever seen something like that ? And how it can be fixed ?
> Regards, Vatslav
С уважением, Владимир Рыженьков
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20170921/763dc537/attachment-0001.html>
More information about the Nek5000-users
mailing list