[Nek5000-users] very low calculation speed

nek5000-users at lists.mcs.anl.gov nek5000-users at lists.mcs.anl.gov
Thu Sep 21 01:27:29 CDT 2017


This problem compiled with PPLIST="NEKNEK"

/----------------------------------------------------------\\
|      _   __ ______ __ __  ______  ____    ____    ____   |
|     / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |
|    /  |/ // __/  / ,<   /___ \\  / / / // / / // / / /   |
|   / /|  // /___ / /| | ____/ / / /_/ // /_/ // /_/ /     |
|  /_/ |_//_____//_/ |_|/_____/  \\____/ \\____/ \\____/   |
|                                                          |
|----------------------------------------------------------|
|                                                          |
| NEK5000:  Open Source Spectral Element Solver            |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC            |
| Version:  1.0rc1 / SVN  r1115                            |
| Web:      http://nek5000.mcs.anl.gov                     |
|                                                          |
\\----------------------------------------------------------/


 Number of processors:         192
 REAL    wdsize      :           8
 INTEGER wdsize      :           4
 Timer accuracy      : 7.87E-07
/----------------------------------------------------------\\
|      _   __ ______ __ __  ______  ____    ____    ____   |
|     / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |
|    /  |/ // __/  / ,<   /___ \\  / / / // / / // / / /   |
|   / /|  // /___ / /| | ____/ / / /_/ // /_/ // /_/ /     |
|  /_/ |_//_____//_/ |_|/_____/  \\____/ \\____/ \\____/   |
|                                                          |
|----------------------------------------------------------|
|                                                          |
| NEK5000:  Open Source Spectral Element Solver            |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC            |
| Version:  1.0rc1 / SVN  r1115                            |
| Web:      http://nek5000.mcs.anl.gov                     |
|                                                          |
\\----------------------------------------------------------/


 Number of processors:          24
 REAL    wdsize      :           8
 INTEGER wdsize      :           4
 Timer accuracy      : 5.25E-07
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.rea
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.re2
 byte swap: F   6.543210     -2.9312772E+35
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.rea
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.re2
 byte swap: F   6.543210     -2.9312772E+35

 mapping elements to processors
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.map

 mapping elements to processors
 Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.map
           0         213         213        5120        5120  NELV
           3         213         213        5120        5120  NELV
           4         213         213        5120        5120  NELV
           5         213         213        5120        5120  NELV
           1         213         213        5120        5120  NELV
           2         213         213        5120        5120  NELV
           6         213         213        5120        5120  NELV
           7         213         213        5120        5120  NELV
           8         213         213        5120        5120  NELV
           9         213         213        5120        5120  NELV
          10         213         213        5120        5120  NELV
          11         213         213        5120        5120  NELV
 element load imbalance:            1         418         419
 done :: mapping elements to processors

   reading mesh
          14         213         213        5120        5120  NELV
 RANK     0 IEG    3873    3874    3875    3876    3877    3878    3879    3880
                   3881    3882    3883    3884    3885    3886    3887    3888
                   3889    3890    3891    3892    3893    3894    3895    3896
                   3897    3898    3899    3900    3901    3902    3903    3904
                   4129    4130    4131    4132    4133    4134    4135    4136
                   4137    4138    4139    4140    4141    4142    4143    4144
                   4145    4146    4147    4148    4149    4150    4151    4152
          16         214         214        5120        5120  NELV
                   4153    4154    4155    4156    4157    4158    4159    4160
                   4353    4354    4355    4356    4361    4362    4363    4364
                   4369    4370    4371    4372    4377    4378    4379    4380
                   4385    4386    4387    4388    4389    4390    4391    4392
                   4393    4394    4395    4396    4397    4398    4399    4400
                   4401    4402    4403    4404    4405    4406    4407    4408
                   4409    4410    4411    4412    4413    4414    4415    4416
                   4609    4610    4611    4612    4613    4617    4618    4619
                   4620    4621    4625    4626    4627    4628    4633    4634
                   4635    4636    4637    4638    4641    4642    4643    4644
                   4645    4646    4647    4648    4649    4650    4651    4652
          17         214         214        5120        5120  NELV
                   4653    4654    4655    4656    4657    4658    4659    4660
                   4661    4662    4663    4664    4665    4666    4667    4668
                   4669    4670    4671    4672    4865    4866    4867    4868
                   4869    4873    4874    4875    4876    4877    4881    4882
                   4883    4884    4889    4890    4891    4897    4898    4899
                   4900    4901    4902    4903    4904    4905    4906    4907
          12         213         213        5120        5120  NELV
                   4908    4909    4910    4911    4912    4913    4914    4915
                   4916    4917    4918    4919    4920    4921    4922    4923
                   4924    4925    4926    4927    4928
          15         213         213        5120        5120  NELV
 element load imbalance:            1         213         214
 done :: mapping elements to processors

          18         214         214        5120        5120  NELV
   reading mesh
          22         214         214        5120        5120  NELV
   reading curved sides
          13         213         213        5120        5120  NELV
   reading bc for ifld           1
          20         214         214        5120        5120  NELV
 done :: read .re2 file

          21         214         214        5120        5120  NELV
           0  objects found
          23         214         214        5120        5120  NELV
 118   Parameters from file:
   1      1.00000     p001 DENSITY
   2     -8900.00     p002 VISCOS
          19         214         214        5120        5120  NELV
   7      1.00000     p007 RHOCP
   8     -8900.00     p008 CONDUCT
  11      100000.0        p011 NSTEPS
  12     -3.0E-04     p012 DT
  15      1000.0         p015 IOSTEP
  18      5.0E-02     p018 GRID < 0 --> # cells on screen
  19     -1.00000     p019 INTYPE
  20      4.00000     p020 NORDER
  21      1.0E-06     p021 DIVERGENCE
  22      1.0E-08     p022 HELMHOLTZ
  24      1.0E-02     p024 TOLREL
  25      1.0E-02     p025 TOLABS
  26      0.50000     p026 COURANT/NTAU
  27      2.00000     p027 TORDER
  28      1.00000     p028 TORDER: mesh velocity (0: p28=p27)
  66      0.00000     p066 output : <0=ascii, else binary
  67      0.00000     p067 restart: <0=ascii, else binary
  93      20.0000     p093 Number of previous pressure solns saved
  95      5.00000     p095 start projecting pressure after p95 step
  99      3.00000     p099 dealiasing: <0--> off/3--> old/4--> new
 101      1.00000     p101 Number of additional modes to filter
 102      1.00000     p102 Dump out divergence at each time step
 103     9.500000E-01 p103 weight of stabilizing filter (.01)

 nelgt/nelgv/lelt:        5120        5120        4000
 lx1  /lx2  /lx3 :          12          12          12
 done :: read .rea file   0.54454     sec

 setup mesh topology
   Right-handed check complete for    5120 elements. OK.
   setvert3d:  12     1772160     6892160     1772160     1772160
 call usrsetvert
 done :: usrsetvert

gs_setup: 340773 unique labels shared
   pairwise times (avg, min, max): 0.000373997 0.000350785 0.000392795
   reading curved sides
   crystal router                : 0.00137192 0.00134859 0.0013859
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 2.16376e+06 2106108 2251852
   buffer bytes (avg, min, max): 476643 410800 577024
   setupds time 3.9776E-01 seconds   0 12     1772160        5120
           8  max multiplicity
 done :: setup mesh topology

 call usrdat
 done :: usrdat

 generate geometry data
   reading bc for ifld           1
 done :: read .re2 file

           0  objects found
 118   Parameters from file:
   1      1.00000     p001 DENSITY
   2     -8900.00     p002 VISCOS
   7      1.00000     p007 RHOCP
   8     -8900.00     p008 CONDUCT
  11      100000.0        p011 NSTEPS
  12     -3.0E-04     p012 DT
  15      1000.0         p015 IOSTEP
  18      5.0E-02     p018 GRID < 0 --> # cells on screen
  19     -1.00000     p019 INTYPE
  20      4.00000     p020 NORDER
  21      1.0E-06     p021 DIVERGENCE
  22      1.0E-08     p022 HELMHOLTZ
  24      1.0E-02     p024 TOLREL
  25      1.0E-02     p025 TOLABS
  26      0.50000     p026 COURANT/NTAU
  27      2.00000     p027 TORDER
  28      1.00000     p028 TORDER: mesh velocity (0: p28=p27)
  66      0.00000     p066 output : <0=ascii, else binary
  67      0.00000     p067 restart: <0=ascii, else binary
  93      20.0000     p093 Number of previous pressure solns saved
  95      5.00000     p095 start projecting pressure after p95 step
  99      3.00000     p099 dealiasing: <0--> off/3--> old/4--> new
 101      1.00000     p101 Number of additional modes to filter
 102      1.00000     p102 Dump out divergence at each time step
 103     9.500000E-01 p103 weight of stabilizing filter (.01)

 nelgt/nelgv/lelt:       80320       80320        4000
 lx1  /lx2  /lx3 :          12          12          12
 done :: read .rea file    1.1102     sec

 setup mesh topology
   Right-handed check complete for   80320 elements. OK.
   setvert3d:  12    26906208   107226208    26906208    26906208
 call usrsetvert
 done :: usrsetvert

gs_setup: 5553221 unique labels shared
   pairwise times (avg, min, max):  done :: generate geometry data

 call usrdat2
0.0016136 0.0012603 0.00186529
   crystal router                : 0.00763995 0.00749459 0.00784841
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 4.35947e+06 3848020 4681812
   buffer bytes (avg, min, max): 976972 546496 1320848
   setupds time 2.1430E+00 seconds   0 12    26906208       80320
           8  max multiplicity
 done :: setup mesh topology

 call usrdat
 done :: usrdat

 generate geometry data
 done :: generate geometry data

 call usrdat2
 initializing intpts(), tol=  1.000000000000000E-013
 initializing intpts(), tol=  1.000000000000000E-013
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found        1536 pipe_per
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
 done :: usrdat2

 done :: usrdat2

  found           0 pipe
 regenerate geometry data           1
 regenerate geometry data           1
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
  found           0 pipe
 done :: regenerate geometry data           1

 verify mesh topology
  -1.00000000000000        1.00000000000000       Xrange
  -1.00000000000000        1.00000000000000       Yrange
  0.000000000000000E+000   4.50000000000000       Zrange
 done :: verify mesh topology

 IFTRAN   = T
 IFFLOW   = T
 IFHEAT   = F
 IFSPLIT  = T
 IFLOMACH = F
 IFUSERVP = F
 IFUSERMV = F
 IFSTRS   = T
 IFCHAR   = F
 IFCYCLIC = F
 IFAXIS   = F
 IFMVBD   = F
 IFMELT   = F
 IFMODEL  = F
 IFKEPS   = F
 IFMOAB   = F
 IFNEKNEK = T
 IFSYNC   = T

 IFVCOR   = T
 IFINTQ   = F
 IFCWUZ   = F
 IFSWALL  = F
 IFGEOM   = F
 IFSURT   = F
 IFWCNO   = F
 IFCMT    = F
 IFVISC   = F
 IFFLTR   = F

 IFTMSH for field           1    =  F
 IFADVC for field           1    =  T
 IFNONL for field           1    =  F

 Dealiasing enabled, lxd=          18
 done :: regenerate geometry data           1

 verify mesh topology
  -12.0000000000000        12.0000000000000       Xrange
  -12.0000000000000        12.0000000000000       Yrange
  0.000000000000000E+000   36.0000000000000       Zrange
 done :: verify mesh topology

 IFTRAN   = T
 IFFLOW   = T
 IFHEAT   = F
 IFSPLIT  = T
 IFLOMACH = F
 IFUSERVP = F
 IFUSERMV = F
 IFSTRS   = T
 IFCHAR   = F
 IFCYCLIC = F
 IFAXIS   = F
 IFMVBD   = F
 IFMELT   = F
 IFMODEL  = F
 IFKEPS   = F
 IFMOAB   = F
 IFNEKNEK = T
 IFSYNC   = T

 IFVCOR   = F
 IFINTQ   = F
 IFCWUZ   = F
 IFSWALL  = F
 IFGEOM   = F
 IFSURT   = F
 IFWCNO   = F
 IFCMT    = F
 IFVISC   = F
 IFFLTR   = F

 IFTMSH for field           1    =  F
 IFADVC for field           1    =  T
 IFNONL for field           1    =  F

 Dealiasing enabled, lxd=          18

 Estimated eigenvalues
 EIGAA =    1.80739669073447
 EIGGA =    13884400.9520853
 EIGAE =   0.487387871658733
 EIGAS =   2.247191011235953E-002
 EIGGE =    13884400.9520853
 EIGGS =    2.00000000000000

 verify mesh topology
  -1.00000000000000        1.00000000000000       Xrange
  -1.00000000000000        1.00000000000000       Yrange
  0.000000000000000E+000   4.50000000000000       Zrange
 done :: verify mesh topology

  E-solver strategy:  0 itr
 mg_nx:           1           7          11
 mg_ny:           1           7          11
 mg_nz:           1           7          11
 call usrsetvert
 done :: usrsetvert

gs_setup: 2723 unique labels shared
   pairwise times (avg, min, max): 2.96265e-05 2.85149e-05 3.04937e-05
   crystal router                : 1.43409e-05 1.38998e-05 1.49012e-05
   all reduce                    : 6.69807e-05 6.60896e-05 6.82116e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 66207.7 54708 77100
   buffer bytes (avg, min, max): 9284 7536 10800
   setupds time 4.3750E-03 seconds   1  2        5760        5120
   setvert3d:   4      103040      144000      103040      103040
 call usrsetvert
 done :: usrsetvert

gs_setup: 25165 unique labels shared
   pairwise times (avg, min, max): 2.84771e-05 2.72989e-05 2.95877e-05
   crystal router                : 4.75297e-05 4.67062e-05 4.81129e-05
   all reduce                    : 0.00061613 0.000604296 0.000622106
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 171156 165372 179980
   buffer bytes (avg, min, max): 39800 34608 47744
   setupds time 2.7411E-02 seconds   2  4      103040        5120
   setvert3d:   8      681600     1787520      681600      681600
 call usrsetvert
 done :: usrsetvert

gs_setup: 137801 unique labels shared
   pairwise times (avg, min, max): 8.8045e-05 8.38041e-05 9.10997e-05
   crystal router                : 0.000316096 0.00030961 0.000320601
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 886769 861564 925228
   buffer bytes (avg, min, max): 197997 170992 239168
   setupds time 1.0851E-01 seconds   3  8      681600        5120
   setvert3d:  10     1162880     3784320     1162880     1162880
 call usrsetvert
 done :: usrsetvert

gs_setup: 227995 unique labels shared
   pairwise times (avg, min, max): 0.000147719 0.00013988 0.000153089
   crystal router                : 0.000699271 0.000687408 0.000709796
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 1.45509e+06 1415292 1515868
   buffer bytes (avg, min, max): 322264 277968 389792
   setupds time 1.9729E-01 seconds   4 10     1162880        5120
 regenerate geometry data           1
 NOTE: All elements deformed , param(59) ^=0
 done :: regenerate geometry data           1

 h1_mg_nx:           1           3          11
 h1_mg_ny:           1           3          11
 h1_mg_nz:           1           3          11
 call usrsetvert
 done :: usrsetvert

gs_setup: 2723 unique labels shared
   pairwise times (avg, min, max): 2.57045e-05 2.50816e-05 2.63929e-05
   crystal router                : 1.31597e-05 1.27077e-05 1.35183e-05
   all reduce                    : 6.57449e-05 6.45876e-05 6.69956e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 66207.7 54708 77100
   buffer bytes (avg, min, max): 9284 7536 10800
   setupds time 3.6449E-03 seconds   5  2        5760        5120
   setvert3d:   4      103040      144000      103040      103040
 call usrsetvert
 done :: usrsetvert

gs_setup: 25165 unique labels shared
   pairwise times (avg, min, max): 2.73794e-05 2.62976e-05 2.83003e-05
   crystal router                : 4.36117e-05 4.29153e-05 4.42028e-05
   all reduce                    : 0.000558805 0.000551605 0.000561714
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 171156 165372 179980
   buffer bytes (avg, min, max): 39800 34608 47744
   setupds time 2.3185E-02 seconds   6  4      103040        5120
   setvert3d:   4      103040      144000      103040      103040
 call usrsetvert
 done :: usrsetvert

gs_setup: 25165 unique labels shared
   pairwise times (avg, min, max): 3.27577e-05 3.13044e-05 3.38078e-05
   crystal router                : 4.4166e-05 4.32968e-05 4.48942e-05
   all reduce                    : 0.000555371 0.00054791 0.000558496
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 171156 165372 179980
   buffer bytes (avg, min, max): 39800 34608 47744
   setupds time 2.2302E-02 seconds   7  4      103040        5120
   setvert3d:   6      328320      656000      328320      328320
 call usrsetvert
 done :: usrsetvert

gs_setup: 70191 unique labels shared
   pairwise times (avg, min, max): 5.42253e-05 5.29051e-05 5.60999e-05
   crystal router                : 0.000130047 0.000128007 0.000131607
   all reduce                    : 0.00147395 0.0014652 0.00147929
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 458791 444924 479932
   buffer bytes (avg, min, max): 103843 89872 125152
   setupds time 6.3579E-02 seconds   8  6      328320        5120
   setvert3d:  12     1772160     6892160     1772160     1772160
 call usrsetvert
 done :: usrsetvert

gs_setup: 340773 unique labels shared
   pairwise times (avg, min, max): 0.000211759 0.000206304 0.00021801
   crystal router                : 0.00135557 0.00133109 0.00137
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 2.16376e+06 2106108 2251852
   buffer bytes (avg, min, max): 476643 410800 577024
   setupds time 3.5228E-01 seconds   9 12     1772160        5120
   setvert3d:  14     2509440    11356800     2509440     2509440
 call usrsetvert
 done :: usrsetvert

gs_setup: 476135 unique labels shared
   pairwise times (avg, min, max): 0.000281647 0.000266504 0.000291586
   crystal router                : 0.00225826 0.0022172 0.00228479
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 3.01277e+06 2934012 3133180
   buffer bytes (avg, min, max): 661133 569488 800864
   setupds time 6.1860E-01 seconds  10 14     2509440        5120
 setup h1 coarse grid, nx_crs=           2
 call usrsetvert
 done :: usrsetvert

gs_setup: 2723 unique labels shared
   pairwise times (avg, min, max): 1.48118e-05 1.41859e-05 1.58072e-05
   crystal router                : 1.33614e-05 1.21117e-05 1.38044e-05
   all reduce                    : 6.4381e-05 6.33001e-05 6.53982e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 66207.7 54708 77100
   buffer bytes (avg, min, max): 9284 7536 10800

 Estimated eigenvalues
 EIGAA =   1.396163174022414E-002
 EIGGA =    26467996.3219686
 EIGAE =   7.615435494667707E-003
 EIGAS =   4.545454545454543E-002
 EIGGE =    26467996.3219686
 EIGGS =    2.00000000000000

 verify mesh topology
  -12.0000000000000        12.0000000000000       Xrange
  -12.0000000000000        12.0000000000000       Yrange
  0.000000000000000E+000   36.0000000000000       Zrange
 done :: setup h1 coarse grid   0.402811050415039       sec

 call usrdat3
 done :: usrdat3

 set initial conditions
 Checking restart options: pipe_per0.f00010
 nekuic (1) for ifld            1
 Reading checkpoint data
       0       0 OPEN: pipe_per0.f00010
 byte swap: F   6.543210     -2.9312772E+35
 done :: verify mesh topology

  E-solver strategy:  0 itr
 mg_nx:           1           7          11
 mg_ny:           1           7          11
 mg_nz:           1           7          11
 call usrsetvert
 done :: usrsetvert

gs_setup: 40651 unique labels shared
   pairwise times (avg, min, max): 0.000218946 0.00018239 0.000236607
   crystal router                : 5.72741e-05 5.60999e-05 5.85079e-05
   all reduce                    : 0.00148923 0.00145738 0.0015028
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 171689 112124 221476
   buffer bytes (avg, min, max): 18153.2 12304 23232
   setupds time 7.1854E-02 seconds   1  2       83038       80320
   setvert3d:   4     1550072     2192632     1550072     1550072
 call usrsetvert
 done :: usrsetvert

gs_setup: 400269 unique labels shared
   pairwise times (avg, min, max): 0.000248798 0.000221109 0.000264406
   crystal router                : 0.000225847 0.000221992 0.000230503
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 1.41297e+06 878540 1853668
   buffer bytes (avg, min, max): 151639 98096 195920
   setupds time 7.4004E-02 seconds   2  4     1550072       80320
   setvert3d:   8    10330540    27679660    10330540    10330540
 call usrsetvert
 done :: usrsetvert

gs_setup: 2233849 unique labels shared
   pairwise times (avg, min, max): 0.000514414 0.000456595 0.000552988
   crystal router                : 0.00232402 0.00227511 0.00238209
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 1.77983e+06 1567956 1917012
   buffer bytes (avg, min, max): 405180 225984 547728
   setupds time 4.5856E-01 seconds   3  8    10330540       80320
   setvert3d:  10    17643974    58767814    17643974    17643974
 call usrsetvert
 done :: usrsetvert

gs_setup: 3707811 unique labels shared
   pairwise times (avg, min, max): 0.00074541 0.000669909 0.000782418

        0  2.8664E+02 done :: Read checkpoint data
                              avg data-throughput =    84.5MBps
                              io-nodes =     1

   crystal router                :  xyz min    -1.0000      -1.0000
   0.0000
 uvwpt min -0.34051     -0.37479     -0.16102E-01 -0.90090E-01   0.0000
 PS min      0.0000       0.0000      0.99000E+22
 xyz max     1.0000       1.0000       4.5000
 uvwpt max  0.32898      0.39774       1.3656      0.78843E-01   0.0000
 PS max      0.0000       0.0000     -0.99000E+22
 Restart: recompute geom. factors.
 regenerate geometry data           1
0.00430806 0.00423291 0.00441771
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 2.92727e+06 2581844 3147348
   buffer bytes (avg, min, max): 660122 368832 892432
   setupds time 1.0661E+00 seconds   4 10    17643974       80320
 NOTE: All elements deformed , param(59) ^=0
 done :: regenerate geometry data           1

 regenerate geometry data           1
 done :: set initial conditions

 call userchk
 NOTE: All elements deformed , param(59) ^=0
 done :: regenerate geometry data           1

 h1_mg_nx:           1           3          11
 h1_mg_ny:           1           3          11
 h1_mg_nz:           1           3          11
 call usrsetvert
 done :: usrsetvert

gs_setup: 40651 unique labels shared
   pairwise times (avg, min, max): 0.000203656 0.000173688 0.000223613
   crystal router                : 5.95078e-05 5.80072e-05 6.10113e-05
   all reduce                    : 0.00126102 0.0012404 0.00127389
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 171689 112124 221476
   buffer bytes (avg, min, max): 18153.2 12304 23232
   setupds time 2.9112E-02 seconds   5  2       83038       80320
   setvert3d:   4     1550072     2192632     1550072     1550072
 call usrsetvert
 done :: usrsetvert

gs_setup: 400269 unique labels shared
   pairwise times (avg, min, max): 0.000249024 0.000214601 0.000264192
   crystal router                : 0.000207164 0.000203013 0.000212216
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 1.41297e+06 878540 1853668
   buffer bytes (avg, min, max): 151639 98096 195920
   setupds time 5.8844E-02 seconds   6  4     1550072       80320
   setvert3d:   4     1550072     2192632     1550072     1550072
 call usrsetvert
 done :: usrsetvert

gs_setup: 400269 unique labels shared
   pairwise times (avg, min, max): 0.000250614 0.000216007 0.000266099
   crystal router                : 0.000206963 0.000201392 0.000213099
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 1.41297e+06 878540 1853668
   buffer bytes (avg, min, max): 151639 98096 195920
   setupds time 5.8632E-02 seconds   7  4     1550072       80320
   setvert3d:   6     4965906    10106386     4965906     4965906
 call usrsetvert
 done :: usrsetvert

gs_setup: 1131335 unique labels shared
   pairwise times (avg, min, max): 0.000345911 0.000313807 0.000366807
   crystal router                : 0.000918429 0.000901699 0.000948811
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 917140 806356 990804
   buffer bytes (avg, min, max): 212147 117952 286736
   setupds time 1.7421E-01 seconds   8  6     4965906       80320
   setvert3d:  12    26906208   107226208    26906208    26906208
 call usrsetvert
 done :: usrsetvert

gs_setup: 5553221 unique labels shared
   pairwise times (avg, min, max): 0.00103432 0.000936699 0.00110071
   crystal router                : 0.00765613 0.0074913 0.0078582
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 4.35947e+06 3848020 4681812
   buffer bytes (avg, min, max): 976972 546496 1320848
   setupds time 1.9455E+00 seconds   9 12    26906208       80320
   setvert3d:  14    38117242   176910202    38117242    38117242
 call usrsetvert
 done :: usrsetvert

gs_setup: 7770079 unique labels shared
   pairwise times (avg, min, max): 0.00144147 0.00128691 0.0015306
   crystal router                : 0.0118797 0.011621 0.0121873
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 6.07641e+06 5366484 6520404
   buffer bytes (avg, min, max): 1.35573e+06 758976 1832976
   setupds time 2.9645E+00 seconds  10 14    38117242       80320
 setup h1 coarse grid, nx_crs=           2
 call usrsetvert
 done :: usrsetvert

gs_setup: 40651 unique labels shared
   pairwise times (avg, min, max): 0.000223077 0.000198388 0.000240588
   crystal router                : 5.7518e-05 5.61953e-05 5.86987e-05
   all reduce                    : 0.00124078 0.0012202 0.00125399
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 171689 112124 221476
   buffer bytes (avg, min, max): 18153.2 12304 23232
 done :: setup h1 coarse grid    22.2913181781769       sec

 call usrdat3
 done :: usrdat3

 set initial conditions
 Checking restart options: pipe0.f00010
 nekuic (1) for ifld            1
 Reading checkpoint data
       0       0 OPEN: pipe0.f00010
 byte swap: F   6.543210     -2.9312772E+35

        0  2.8664E+02 done :: Read checkpoint data
                              avg data-throughput =    98.5MBps
                              io-nodes =     1

 xyz min    -12.000      -12.000       0.0000
 uvwpt min -0.99492     -0.76080     -0.82980      -1.4131       0.0000
 PS min      0.0000       0.0000      0.99000E+22
 xyz max     12.000       12.000       36.000
 uvwpt max  0.81959      0.91894       1.4289      0.99220       0.0000
 PS max      0.0000       0.0000     -0.99000E+22
 Restart: recompute geom. factors.
 regenerate geometry data           1
 NOTE: All elements deformed , param(59) ^=0
 done :: regenerate geometry data           1

 done :: set initial conditions

 call userchk
 force z:   1.000000000000000E-005
 done :: userchk

gridpoints unique/tot:       6892160      8847360
  dofs:               6737280               6892160

 Initial time: 0.2866404E+03
 Initialization successfully completed    71.437     sec

Starting time loop ...

     DT/DTCFL/DTFS/DTINIT   0.300E-03   0.000E+00   0.233-309   0.300E-03
 force z:   1.000000000000000E-005
 dump history points
 reading hpts.in
 found           80  points
 initializing intpts(), tol=  1.000000000000000E-013
 done :: dump history points
 Building mesh for avg
 Avg mesh builded
 initializing intpts(), tol=  1.000000000000000E-008
 call intpts
 call findpts
   total number of points =      1920000
   failed =            0
 done :: intpts
 Writting interp data avg
 Interp data avg written
 call intpts
 call findpts
   total number of points =      1920000
   failed =            0
 done :: intpts
 Writting interp data avg
 Interp data avg written
 call intpts
 call findpts
   total number of points =      1920000
   failed =            0
 done :: intpts
 Writting interp data avg
 Interp data avg written
 Building the mesh for pod1
 Mesh for pod1 builded
 call intpts
 call findpts
   total number of points =     21504000
   failed =            0
 done :: intpts
 Writting pod1 data
 pod1 data written
 Building the mesh for pod2
 Mesh for pod2 builded
 call intpts
 call findpts
   total number of points =     21504000
   failed =            0
 done :: intpts
 Writting pod2 data
 pod2 data written
 done :: userchk

gridpoints unique/tot:     107226208    138792960
  dofs:             106989839             106823343

 Initial time: 0.2866404E+03
 Initialization successfully completed    4728.4     sec

Starting time loop ...

     DT/DTCFL/DTFS/DTINIT   0.300E-03   0.000E+00   0.236-309   0.300E-03
Step      1, t= 2.8664073E+02, DT= 3.0000000E-04, C=  0.149 0.0000E+00
0.0000E+00
Step      1, t= 2.8664073E+02, DT= 3.0000000E-04, C=  0.438 0.0000E+00
0.0000E+00
 Solving for fluid T T T
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.2375 0.9500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
1.0000 1.0000 0.7625 0.0500
  1.000000000000000E-008  p22            1           1
    1 1.00000E-06 3.08669E-03 5.12889E-03 6.01823E-01       1 Divergence
    2 1.00000E-06 1.63507E-03 5.12889E-03 3.18795E-01       1 Divergence
    3 1.00000E-06 5.70865E-04 5.12889E-03 1.11304E-01       1 Divergence
    4 1.00000E-06 3.19087E-04 5.12889E-03 6.22137E-02       1 Divergence
    5 1.00000E-06 1.67594E-04 5.12889E-03 3.26764E-02       1 Divergence
    6 1.00000E-06 8.78601E-05 5.12889E-03 1.71304E-02       1 Divergence
    7 1.00000E-06 4.10072E-05 5.12889E-03 7.99534E-03       1 Divergence
    8 1.00000E-06 2.09226E-05 5.12889E-03 4.07936E-03       1 Divergence
    9 1.00000E-06 1.17707E-05 5.12889E-03 2.29498E-03       1 Divergence
   10 1.00000E-06 6.32563E-06 5.12889E-03 1.23333E-03       1 Divergence
   11 1.00000E-06 3.53710E-06 5.12889E-03 6.89642E-04       1 Divergence
   12 1.00000E-06 1.96804E-06 5.12889E-03 3.83716E-04       1 Divergence
   13 1.00000E-06 1.14795E-06 5.12889E-03 2.23821E-04       1 Divergence
   14 1.00000E-06 7.12487E-07 5.12889E-03 1.38916E-04       1 Divergence
        1 PRES gmres:   14  7.1249E-07  1.0000E-06  5.1289E-03
1.8504E+00  4.1575E+00    F
  0  1.0000E-08  3.4398E-07  1.0000E-08 tol,matmod
           1 Helmholtz3/fluid:      7   0.1612E-08   0.1000E-07   0.1453E-01
               L1/L2 DIV(V)    :  -3.3387E-15   4.5580E-04
               L1/L2 QTL       :   0.0000E+00   0.0000E+00
               L1/L2 DIV(V)-QTL:  -3.3387E-15   4.5580E-04
          1   2.8664E+02  6.3868E+00 Fluid done
 Solving for fluid T T T
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.2375 0.9500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
1.0000 1.0000 0.7625 0.0500


> Can you please post your logfile otherwise it’s hard to tell what’s going wrong.
> On 20 Sep 2017, at 12:20, "nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users> <mailto:nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>> " <nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users> <mailto:nek5000-users at lists.mcs.anl.gov <https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>> > wrote:
> Dear Nek devs
> Usually I start some calculation with 100-150 processors and it runs with acceptable speed. But sometimes the speed of calculation is extremely low, something like 1 step per 10 hours with 50 million mesh points. It seems like all computational nodes fall on one processors, while the rest just stay without any job, despite that all requirements about nodes per processor are satisfied.
> Have you ever seen something like that ? And how it can be fixed ?
> Regards, Vatslav




С уважением, Владимир Рыженьков
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20170921/763dc537/attachment-0001.html>


More information about the Nek5000-users mailing list