[Nek5000-users] os7000: Imbalance>20%
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Fri Jun 22 01:16:24 CDT 2018
Well if you run a 15 element case on 8 ranks. In this case your 100% imbalanced because one rank will have just one element where all others have two.
On 22 Jun 2018, at 03:41, "nek5000-users at lists.mcs.anl.gov <mailto:nek5000-users at lists.mcs.anl.gov> " <nek5000-users at lists.mcs.anl.gov <mailto:nek5000-users at lists.mcs.anl.gov> > wrote:
Hello Nek’s,
I recently started NEK with the example of os7000. I tried reproduce the results first with the original settings. However I noticed that in my logfile there was an warning saying imbalance > 20%. I tried to fix it by re-generate the .map file with higher mesh tolerance (0.1) but I still received the same message. The relevant part of the logfile is attached below. Could anyone help me on this please?
Thanks,
Jim.
Number of processors: 8
REAL wdsize : 8
INTEGER wdsize : 4
Timer accuracy : 1.09E-07
Reading /global/home/hpc4076/Nek5000/examples/os7000/u3_t020_n13.rea
mapping elements to processors
Reading /global/home/hpc4076/Nek5000/examples/os7000/u3_t020_n13.map
6 2 2 15 15 NELV
7 2 2 15 15 NELV
0 1 1 15 15 NELV
1 2 2 15 15 NELV
2 2 2 15 15 NELV
3 2 2 15 15 NELV
4 2 2 15 15 NELV
5 2 2 15 15 NELV
RANK 0 IEG 14
element load imbalance: 1 1 2
WARNING: imbalance >20% !!!
done :: mapping 0.14527E-01 sec
0 objects found
103 Parameters from file:
1 1.000000 p01 DENSITY
2 -7000.000 p02 VISCOS
8 1.000000 p08 CONDUCT
10 200.0000 p10 FINTIME
12 -2.0000000E-02 p12 DT
14 200.0000 p14 IOTIME
21 5.0000000E-12 p21 DIVERGENCE
22 5.0000000E-12 p22 HELMHOLTZ
24 1.0000000E-10 p24 TOLREL
25 1.0000000E-10 p25 TOLABS
26 1.000000 p26 COURANT/NTAU
27 3.000000 p27 TORDER
59 0.0000000E+00 p59 !=0 --> use std axhelm for all elem
66 6.000000 p66 write fmt:ONLY postx uses rea value
67 6.000000 p67 read fmt: same modes as p66
76 1.00000 p76 1 = use distributed Eo-1
80 1.000000E-05 p80 eps for Orr-Som I.C.
93 80.00000 p93 Numbr of prev pressure solns saved
95 8.000000 p95 start projecting pr after p95 step
99 0.0000000E+00 p99 dealiasing:if <0 disable
103 5.0000001E-02 p103 weight of stabilizing filter (.01)
done :: read .rea file 0.43078E-01 sec
nelgt/nelgv/lelt: 15 15 104
lx1 /lx2 /lx3 : 14 12 14
setup mesh topology
Right-handed check complete for 15 elements. OK.
setvert2d: 14 414 2574 414 414
call usrsetvert
done :: usrsetvert
gs_setup: 256 unique labels shared
pairwise times (avg, min, max): 7.71275e-06 7.50176e-06 7.93398e-06
crystal router : 5.75845e-06 5.68549e-06 5.80889e-06
all reduce : 1.63446e-05 1.61897e-05 1.65087e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 9759 7716 11724
buffer bytes (avg, min, max): 2254 1728 2736
setupds time 4.8408E-03 seconds 0 14 414 15
4 max multiplicity
done :: setup mesh topology
call usrdat
done :: usrdat
generate geometry data
NOTE: All elements deformed , param(59) ^=0
done :: generate geometry data
call usrdat2
done :: usrdat2
regenerate geometry data 1
NOTE: All elements deformed , param(59) ^=0
done :: regenerate geometry data 1
verify mesh topology
0.000000000000000E+000 6.28318530717959 Xrange
-1.00000000000000 1.00000000000000 Yrange
0.000000000000000E+000 0.000000000000000E+000 Zrange
done :: verify mesh topology
IFTRAN = T
IFFLOW = T
IFHEAT = F
IFSPLIT = F
IFLOMACH = F
IFUSERVP = F
IFUSERMV = F
IFPERT = F
IFADJ = F
IFSTRS = F
IFCHAR = T
IFCYCLIC = F
IFAXIS = F
IFMVBD = F
IFMELT = F
IFNEKNEK = F
IFSYNC = F
IFVCOR = T
IFINTQ = F
IFGEOM = F
IFSURT = F
IFWCNO = F
IFTMSH for field 1 = F
IFADVC for field 1 = T
IFNONL for field 1 = F
Dealiasing enabled, lxd= 21
Estimated eigenvalues
EIGAA = 1.35870055013617
EIGGA = 511517.514390660
EIGAE = 0.250000000000000
EIGAS = 2.299991708759382E-002
EIGGE = 511517.514390660
EIGGS = 2.00000000000000
verify mesh topology
0.000000000000000E+000 6.28318530717959 Xrange
-1.00000000000000 1.00000000000000 Yrange
0.000000000000000E+000 0.000000000000000E+000 Zrange
done :: verify mesh topology
E-solver strategy: 1 itr
mg_nx: 1 7 13
mg_ny: 1 7 13
mg_nz: 0 0 0
call usrsetvert
done :: usrsetvert
gs_setup: 16 unique labels shared
pairwise times (avg, min, max): 1.24085e-05 1.22588e-05 1.25484e-05
crystal router : 3.45357e-06 3.35569e-06 3.55481e-06
all reduce : 4.82029e-06 4.70788e-06 5.02933e-06
used all_to_all method: crystal router
handle bytes (avg, min, max): 1515 1308 1644
buffer bytes (avg, min, max): 214 192 240
setupds time 7.8091E-04 seconds 1 2 18 15
setvert2d: 4 84 144 84 84
call usrsetvert
done :: usrsetvert
gs_setup: 56 unique labels shared
pairwise times (avg, min, max): 3.56551e-06 3.41511e-06 3.74098e-06
crystal router : 3.82754e-06 3.75211e-06 3.85903e-06
all reduce : 6.63e-06 6.52331e-06 6.71241e-06
used all_to_all method: pairwise
handle bytes (avg, min, max): 820 644 980
buffer bytes (avg, min, max): 328 240 416
setupds time 6.5815E-04 seconds 2 4 84 15
setvert2d: 8 216 756 216 216
call usrsetvert
done :: usrsetvert
gs_setup: 136 unique labels shared
pairwise times (avg, min, max): 4.08253e-06 3.8004e-06 4.26793e-06
crystal router : 4.19207e-06 4.09321e-06 4.26732e-06
all reduce : 1.05027e-05 1.04371e-05 1.0623e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 5637 4548 6684
buffer bytes (avg, min, max): 1234 960 1488
setupds time 8.5796E-04 seconds 3 8 216 15
setvert2d: 10 282 1242 282 282
call usrsetvert
done :: usrsetvert
gs_setup: 176 unique labels shared
pairwise times (avg, min, max): 4.27077e-06 4.10192e-06 4.48413e-06
crystal router : 4.99089e-06 4.8595e-06 5.07678e-06
all reduce : 1.03434e-05 1.02228e-05 1.0458e-05
used all_to_all method: pairwise
handle bytes (avg, min, max): 1666 1220 1988
buffer bytes (avg, min, max): 808 624 992
setupds time 9.1538E-04 seconds 4 10 282 15
setup h1 coarse grid, nx_crs= 2
call usrsetvert
done :: usrsetvert
gs_setup: 16 unique labels shared
pairwise times (avg, min, max): 5.17112e-06 4.7449e-06 5.40493e-06
crystal router : 4.43193e-06 4.32599e-06 4.55589e-06
all reduce : 6.15264e-06 6.08228e-06 6.2291e-06
used all_to_all method: crystal router
handle bytes (avg, min, max): 1515 1308 1644
buffer bytes (avg, min, max): 214 192 240
done :: setup h1 coarse grid 6.504331249743700E-003 sec
call usrdat3
done :: usrdat3
set initial conditions
nekuic (1) for ifld 1
vmax: 3.128112017413449E-002
call nekuic for vel
xyz min 0.0000 -1.0000 0.0000
uvwpt min -0.44449E-15 -0.99263E-05 0.0000 0.0000 0.0000
xyz max 6.2832 1.0000 0.0000
uvwpt max 0.99726 0.99769E-05 0.0000 0.0000 0.0000
done :: set initial conditions
call userchk
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
chk0: 0 1 0.000000000000000E+000 1.267294399195043E-009
done :: userchk
gridpoints unique/tot: 2574 2940
dofs: 2496 2160
Initialization successfully completed 0.12906 sec
Starting time loop ...
DT/DTCFL/DTFS/DTINIT 0.200E-01 0.000E+00 0.294-316 0.200E-01
Step 1, time= 2.0000000E-02, DT= 2.0000000E-02, C= 0.475 0.0000E+00 0.0000E+00
Solving for fluid
1 Hmholtz VELX 1 6.7551E-07 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 2 6.7015E-08 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 3 1.0385E-08 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 4 2.9327E-09 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 5 3.8378E-10 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 6 3.9996E-11 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 7 3.2944E-12 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELX 6 3.2944E-12 6.7551E-07 5.0000E-12
1 Hmholtz VELY 1 4.5665E-06 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 2 3.9358E-08 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 3 2.7778E-09 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 4 4.2156E-10 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 5 7.7009E-11 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 6 5.6563E-12 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 7 4.9541E-13 1.4286E-04 5.0000E-12 5.0000E+01 F
1 Hmholtz VELY 6 4.9541E-13 4.5665E-06 5.0000E-12
1 5.00000E-12 1.59732E-07 1.61240E-07 9.90645E-01 1 Divergence
2 5.00000E-12 1.58558E-07 1.61240E-07 9.83365E-01 1 Divergence
3 5.00000E-12 1.54730E-07 1.61240E-07 9.59626E-01 1 Divergence
4 5.00000E-12 1.54653E-07 1.61240E-07 9.59146E-01 1 Divergence
5 5.00000E-12 1.54636E-07 1.61240E-07 9.59037E-01 1 Divergence
6 5.00000E-12 1.43472E-07 1.61240E-07 8.89799E-01 1 Divergence
7 5.00000E-12 1.38062E-07 1.61240E-07 8.56248E-01 1 Divergence
8 5.00000E-12 1.37377E-07 1.61240E-07 8.52001E-01 1 Divergence
9 5.00000E-12 1.09765E-07 1.61240E-07 6.80751E-01 1 Divergence
10 5.00000E-12 8.56204E-08 1.61240E-07 5.31011E-01 1 Divergence
11 5.00000E-12 6.13826E-08 1.61240E-07 3.80690E-01 1 Divergence
12 5.00000E-12 3.35784E-08 1.61240E-07 2.08250E-01 1 Divergence
13 5.00000E-12 2.89932E-08 1.61240E-07 1.79813E-01 1 Divergence
14 5.00000E-12 1.36096E-08 1.61240E-07 8.44059E-02 1 Divergence
15 5.00000E-12 6.17297E-09 1.61240E-07 3.82842E-02 1 Divergence
16 5.00000E-12 3.83161E-09 1.61240E-07 2.37633E-02 1 Divergence
17 5.00000E-12 2.36087E-09 1.61240E-07 1.46420E-02 1 Divergence
18 5.00000E-12 1.60883E-09 1.61240E-07 9.97784E-03 1 Divergence
19 5.00000E-12 8.99316E-10 1.61240E-07 5.57748E-03 1 Divergence
20 5.00000E-12 5.25463E-10 1.61240E-07 3.25888E-03 1 Divergence
21 5.00000E-12 3.15168E-10 1.61240E-07 1.95464E-03 1 Divergence
22 5.00000E-12 2.42188E-10 1.61240E-07 1.50203E-03 1 Divergence
23 5.00000E-12 1.70044E-10 1.61240E-07 1.05460E-03 1 Divergence
24 5.00000E-12 1.15207E-10 1.61240E-07 7.14503E-04 1 Divergence
25 5.00000E-12 3.82692E-11 1.61240E-07 2.37342E-04 1 Divergence
26 5.00000E-12 1.77564E-11 1.61240E-07 1.10124E-04 1 Divergence
27 5.00000E-12 9.30569E-12 1.61240E-07 5.77131E-05 1 Divergence
28 5.00000E-12 5.61156E-12 1.61240E-07 3.48025E-05 1 Divergence
29 5.00000E-12 3.83841E-12 1.61240E-07 2.38055E-05 1 Divergence
1 U-PRES gmres 29 3.8384E-12 1.6124E-07 5.0000E-12 9.7656E-03 1.4128E-02
1 Fluid done 2.0000E-02 2.2119E-02
filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0500
filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9500
2.000000E-02 1 1.71539181E-03 1.71539181E-03 1.00006862E+00 1.00006862E+00 1.92512672E-13 2.80416536E-09 egn
Step 2, time= 4.0000000E-02, DT= 2.0000000E-02, C= 0.475 3.2527E-02 3.2526E-02
Solving for fluid
2 Hmholtz VELX 6 1.1943E-12 3.2477E-06 5.0000E-12
2 Hmholtz VELY 5 9.7511E-13 1.9934E-06 5.0000E-12
2 U-PRES gmres 16 4.9607E-12 5.1929E-09 5.0000E-12 1.5295E-03 2.8241E-03
2 Fluid done 4.0000E-02 3.7955E-03
4.000000E-02 2 1.71539180E-03 1.71539181E-03 1.00013724E+00 1.00013724E+00 3.05755421E-13 1.65220610E-09 egn
_______________________________________________
Nek5000-users mailing list
Nek5000-users at lists.mcs.anl.gov <mailto:Nek5000-users at lists.mcs.anl.gov>
https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20180622/8d7c0d6f/attachment-0001.html>
More information about the Nek5000-users
mailing list