[Nek5000-users] os7000: Imbalance>20%

nek5000-users at lists.mcs.anl.gov nek5000-users at lists.mcs.anl.gov
Thu Jun 21 20:40:07 CDT 2018


Hello Nek’s,

I recently started NEK with the example of os7000. I tried reproduce the results first with the original settings. However I noticed that in my logfile there was an warning saying imbalance > 20%. I tried to fix it by re-generate the .map file with higher mesh tolerance (0.1) but  I still received the same message. The relevant part of the logfile is attached below. Could anyone help me on this please?

Thanks,

Jim.



Number of processors:           8
REAL    wdsize      :           8
INTEGER wdsize      :           4
Timer accuracy      : 1.09E-07

 Reading /global/home/hpc4076/Nek5000/examples/os7000/u3_t020_n13.rea
 mapping elements to processors
Reading /global/home/hpc4076/Nek5000/examples/os7000/u3_t020_n13.map
           6           2           2          15          15  NELV
           7           2           2          15          15  NELV
           0           1           1          15          15  NELV
           1           2           2          15          15  NELV
           2           2           2          15          15  NELV
           3           2           2          15          15  NELV
           4           2           2          15          15  NELV
           5           2           2          15          15  NELV
RANK     0 IEG      14

 element load imbalance:            1           1           2
WARNING: imbalance >20% !!!
done :: mapping   0.14527E-01 sec


           0  objects found
103   Parameters from file:
   1          1.000000         p01 DENSITY
   2         -7000.000         p02 VISCOS
   8          1.000000         p08 CONDUCT
  10          200.0000         p10 FINTIME
  12        -2.0000000E-02     p12 DT
  14          200.0000         p14 IOTIME
  21         5.0000000E-12     p21 DIVERGENCE
  22         5.0000000E-12     p22 HELMHOLTZ
  24         1.0000000E-10     p24 TOLREL
  25         1.0000000E-10     p25 TOLABS
  26          1.000000         p26 COURANT/NTAU
  27          3.000000         p27 TORDER
  59         0.0000000E+00     p59 !=0 --> use std axhelm for all elem
  66          6.000000         p66 write fmt:ONLY postx uses rea value
  67          6.000000         p67 read fmt: same modes as p66
  76          1.00000          p76 1 = use distributed Eo-1
  80          1.000000E-05     p80 eps for Orr-Som I.C.
  93          80.00000         p93 Numbr of prev pressure solns saved
  95          8.000000         p95 start projecting pr after p95 step
  99         0.0000000E+00     p99    dealiasing:if <0 disable
103         5.0000001E-02     p103   weight of stabilizing filter (.01)

 done :: read .rea file   0.43078E-01 sec

nelgt/nelgv/lelt:          15          15         104
lx1  /lx2  /lx3 :          14          12          14
setup mesh topology
   Right-handed check complete for      15 elements. OK.
   setvert2d:  14         414        2574         414         414
call usrsetvert
done :: usrsetvert

gs_setup: 256 unique labels shared
   pairwise times (avg, min, max): 7.71275e-06 7.50176e-06 7.93398e-06
   crystal router                : 5.75845e-06 5.68549e-06 5.80889e-06
   all reduce                    : 1.63446e-05 1.61897e-05 1.65087e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 9759 7716 11724
   buffer bytes (avg, min, max): 2254 1728 2736
   setupds time 4.8408E-03 seconds   0 14         414          15
           4  max multiplicity
done :: setup mesh topology

 call usrdat
done :: usrdat

generate geometry data
NOTE: All elements deformed , param(59) ^=0
done :: generate geometry data

 call usrdat2
done :: usrdat2

regenerate geometry data           1
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data           1

 verify mesh topology
  0.000000000000000E+000   6.28318530717959       Xrange
  -1.00000000000000        1.00000000000000       Yrange
  0.000000000000000E+000  0.000000000000000E+000  Zrange
done :: verify mesh topology

 IFTRAN   = T
IFFLOW   = T
IFHEAT   = F
IFSPLIT  = F
IFLOMACH = F
IFUSERVP = F
IFUSERMV = F
IFPERT   = F
IFADJ    = F
IFSTRS   = F
IFCHAR   = T
IFCYCLIC = F
IFAXIS   = F
IFMVBD   = F
IFMELT   = F
IFNEKNEK = F
IFSYNC   = F

 IFVCOR   = T
IFINTQ   = F
IFGEOM   = F
IFSURT   = F
IFWCNO   = F

 IFTMSH for field           1    =  F
IFADVC for field           1    =  T
IFNONL for field           1    =  F

 Dealiasing enabled, lxd=          21

 Estimated eigenvalues
EIGAA =    1.35870055013617
 EIGGA =    511517.514390660
 EIGAE =   0.250000000000000
 EIGAS =   2.299991708759382E-002
EIGGE =    511517.514390660
 EIGGS =    2.00000000000000

 verify mesh topology
  0.000000000000000E+000   6.28318530717959       Xrange
  -1.00000000000000        1.00000000000000       Yrange
  0.000000000000000E+000  0.000000000000000E+000  Zrange
done :: verify mesh topology

  E-solver strategy:  1 itr
mg_nx:           1           7          13
mg_ny:           1           7          13
mg_nz:           0           0           0
call usrsetvert
done :: usrsetvert

gs_setup: 16 unique labels shared
   pairwise times (avg, min, max): 1.24085e-05 1.22588e-05 1.25484e-05
   crystal router                : 3.45357e-06 3.35569e-06 3.55481e-06
   all reduce                    : 4.82029e-06 4.70788e-06 5.02933e-06
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 1515 1308 1644
   buffer bytes (avg, min, max): 214 192 240
   setupds time 7.8091E-04 seconds   1  2          18          15
   setvert2d:   4          84         144          84          84
call usrsetvert
done :: usrsetvert

gs_setup: 56 unique labels shared
   pairwise times (avg, min, max): 3.56551e-06 3.41511e-06 3.74098e-06
   crystal router                : 3.82754e-06 3.75211e-06 3.85903e-06
   all reduce                    : 6.63e-06 6.52331e-06 6.71241e-06
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 820 644 980
   buffer bytes (avg, min, max): 328 240 416
   setupds time 6.5815E-04 seconds   2  4          84          15
   setvert2d:   8         216         756         216         216
call usrsetvert
done :: usrsetvert

gs_setup: 136 unique labels shared
   pairwise times (avg, min, max): 4.08253e-06 3.8004e-06 4.26793e-06
   crystal router                : 4.19207e-06 4.09321e-06 4.26732e-06
   all reduce                    : 1.05027e-05 1.04371e-05 1.0623e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 5637 4548 6684
   buffer bytes (avg, min, max): 1234 960 1488
   setupds time 8.5796E-04 seconds   3  8         216          15
   setvert2d:  10         282        1242         282         282
call usrsetvert
done :: usrsetvert

gs_setup: 176 unique labels shared
   pairwise times (avg, min, max): 4.27077e-06 4.10192e-06 4.48413e-06
   crystal router                : 4.99089e-06 4.8595e-06 5.07678e-06
   all reduce                    : 1.03434e-05 1.02228e-05 1.0458e-05
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 1666 1220 1988
   buffer bytes (avg, min, max): 808 624 992
   setupds time 9.1538E-04 seconds   4 10         282          15
setup h1 coarse grid, nx_crs=           2
call usrsetvert
done :: usrsetvert

gs_setup: 16 unique labels shared
   pairwise times (avg, min, max): 5.17112e-06 4.7449e-06 5.40493e-06
   crystal router                : 4.43193e-06 4.32599e-06 4.55589e-06
   all reduce                    : 6.15264e-06 6.08228e-06 6.2291e-06
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 1515 1308 1644
   buffer bytes (avg, min, max): 214 192 240
done :: setup h1 coarse grid   6.504331249743700E-003  sec

 call usrdat3
done :: usrdat3

set initial conditions
nekuic (1) for ifld            1
vmax:  3.128112017413449E-002
call nekuic for vel
 xyz min     0.0000      -1.0000       0.0000
 uvwpt min -0.44449E-15 -0.99263E-05   0.0000       0.0000       0.0000
 xyz max     6.2832       1.0000       0.0000
 uvwpt max  0.99726      0.99769E-05   0.0000       0.0000       0.0000
 done :: set initial conditions

 call userchk
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
chk0:           0           1  0.000000000000000E+000  1.267294399195043E-009
done :: userchk

gridpoints unique/tot:          2574         2940
  dofs:                  2496                  2160

 Initialization successfully completed   0.12906     sec

Starting time loop ...

     DT/DTCFL/DTFS/DTINIT   0.200E-01   0.000E+00   0.294-316   0.200E-01
Step      1, time= 2.0000000E-02, DT= 2.0000000E-02, C=  0.475 0.0000E+00 0.0000E+00
             Solving for fluid
          1  Hmholtz VELX       1   6.7551E-07   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       2   6.7015E-08   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       3   1.0385E-08   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       4   2.9327E-09   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       5   3.8378E-10   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       6   3.9996E-11   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       7   3.2944E-12   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELX       6   3.2944E-12   6.7551E-07   5.0000E-12
          1  Hmholtz VELY       1   4.5665E-06   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       2   3.9358E-08   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       3   2.7778E-09   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       4   4.2156E-10   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       5   7.7009E-11   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       6   5.6563E-12   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       7   4.9541E-13   1.4286E-04   5.0000E-12   5.0000E+01   F
          1  Hmholtz VELY       6   4.9541E-13   4.5665E-06   5.0000E-12
    1 5.00000E-12 1.59732E-07 1.61240E-07 9.90645E-01       1 Divergence
    2 5.00000E-12 1.58558E-07 1.61240E-07 9.83365E-01       1 Divergence
    3 5.00000E-12 1.54730E-07 1.61240E-07 9.59626E-01       1 Divergence
    4 5.00000E-12 1.54653E-07 1.61240E-07 9.59146E-01       1 Divergence
    5 5.00000E-12 1.54636E-07 1.61240E-07 9.59037E-01       1 Divergence
    6 5.00000E-12 1.43472E-07 1.61240E-07 8.89799E-01       1 Divergence
    7 5.00000E-12 1.38062E-07 1.61240E-07 8.56248E-01       1 Divergence
    8 5.00000E-12 1.37377E-07 1.61240E-07 8.52001E-01       1 Divergence
    9 5.00000E-12 1.09765E-07 1.61240E-07 6.80751E-01       1 Divergence
  10 5.00000E-12 8.56204E-08 1.61240E-07 5.31011E-01       1 Divergence
   11 5.00000E-12 6.13826E-08 1.61240E-07 3.80690E-01       1 Divergence
   12 5.00000E-12 3.35784E-08 1.61240E-07 2.08250E-01       1 Divergence
   13 5.00000E-12 2.89932E-08 1.61240E-07 1.79813E-01       1 Divergence
   14 5.00000E-12 1.36096E-08 1.61240E-07 8.44059E-02       1 Divergence
   15 5.00000E-12 6.17297E-09 1.61240E-07 3.82842E-02       1 Divergence
   16 5.00000E-12 3.83161E-09 1.61240E-07 2.37633E-02       1 Divergence
   17 5.00000E-12 2.36087E-09 1.61240E-07 1.46420E-02       1 Divergence
   18 5.00000E-12 1.60883E-09 1.61240E-07 9.97784E-03       1 Divergence
   19 5.00000E-12 8.99316E-10 1.61240E-07 5.57748E-03       1 Divergence
   20 5.00000E-12 5.25463E-10 1.61240E-07 3.25888E-03       1 Divergence
   21 5.00000E-12 3.15168E-10 1.61240E-07 1.95464E-03       1 Divergence
   22 5.00000E-12 2.42188E-10 1.61240E-07 1.50203E-03       1 Divergence
   23 5.00000E-12 1.70044E-10 1.61240E-07 1.05460E-03       1 Divergence
   24 5.00000E-12 1.15207E-10 1.61240E-07 7.14503E-04       1 Divergence
   25 5.00000E-12 3.82692E-11 1.61240E-07 2.37342E-04       1 Divergence
   26 5.00000E-12 1.77564E-11 1.61240E-07 1.10124E-04       1 Divergence
   27 5.00000E-12 9.30569E-12 1.61240E-07 5.77131E-05       1 Divergence
   28 5.00000E-12 5.61156E-12 1.61240E-07 3.48025E-05       1 Divergence
   29 5.00000E-12 3.83841E-12 1.61240E-07 2.38055E-05       1 Divergence
          1  U-PRES gmres      29   3.8384E-12   1.6124E-07   5.0000E-12   9.7656E-03   1.4128E-02
          1  Fluid done  2.0000E-02  2.2119E-02
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9500
2.000000E-02     1  1.71539181E-03  1.71539181E-03  1.00006862E+00  1.00006862E+00  1.92512672E-13  2.80416536E-09 egn
Step      2, time= 4.0000000E-02, DT= 2.0000000E-02, C=  0.475 3.2527E-02 3.2526E-02
             Solving for fluid
          2  Hmholtz VELX       6   1.1943E-12   3.2477E-06   5.0000E-12
          2  Hmholtz VELY       5   9.7511E-13   1.9934E-06   5.0000E-12
          2  U-PRES gmres      16   4.9607E-12   5.1929E-09   5.0000E-12   1.5295E-03   2.8241E-03
          2  Fluid done  4.0000E-02  3.7955E-03
4.000000E-02     2  1.71539180E-03  1.71539181E-03  1.00013724E+00  1.00013724E+00  3.05755421E-13  1.65220610E-09 egn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20180622/c6d0e95c/attachment-0001.html>


More information about the Nek5000-users mailing list