[Nek5000-users] run-time hang up in gs_setup
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Tue Jan 10 05:02:49 CST 2012
Dear NEKs;
I am trying to run a simulation of a turbulent flow in a straight pipe
in high Reynolds number (Re_tau = 1000). After generating the grid with
PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
elements. It compiled properly; however, trying to run it, hanged up in
the last stage:
########################################################################
verify mesh topology
-1.000000000000000 1.000000000000000 Xrange
-1.000000000000000 1.000000000000000 Yrange
0.000000000000000 25.00000000000000 Zrange
done :: verify mesh topology
E-solver strategy: 1 itr
mg_nx: 1 3
mg_ny: 1 3
mg_nz: 1 3
call usrsetvert
done :: usrsetvert
gs_setup: 866937 unique labels shared
pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
crystal router : 0.000458177 0.000445795 0.000471807
used all_to_all method: pairwise
setupds time 5.6048E-02 seconds 1 2 4565612 4495920
setvert3d: 4 86046564 122013924 86046564 86046564
call usrsetvert
done :: usrsetvert
gs_setup: 8041169 unique labels shared
pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
crystal router : 0.0040165 0.00392921 0.00411811
used all_to_all method: pairwise
setupds time 1.0465E+00 seconds 2 4 86046564 4495920
setup h1 coarse grid, nx_crs= 2
call usrsetvert
done :: usrsetvert
gs_setup: 866937 unique labels shared
pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
crystal router : 0.000466869 0.00045588 0.000478101
used all_to_all method: pairwise
########################################################################
I was wondering if you could help me with that. I attached the run
logfile and also genmap.out.
Many thanks
Azad
-------------- next part --------------
Input (.rea) file name:
PIPE
Input mesh tolerance (default 0.2):
NOTE: smaller is better, but generous is more forgiving for bad meshes.
0.1
reading .rea file data ...
4495920 3 4495920 F nelt,ndim,nelv,ifre2
start locglob_lexico: 8 4495920 35967360 0.10000000
locglob: 1 1 35967360
locglob: 2 340 35967360
locglob: 3 8490 35967360
locglob: 1 4457250 35967360
locglob: 2 4574325 35967360
locglob: 3 4574325 35967360
locglob: 1 4574325 35967360
locglob: 2 4574325 35967360
locglob: 3 4574325 35967360
done locglob_lexico: 4574325 4574325 35967360 8
start periodic vtx: 4495920 4574325
1000 5 5 3 0.00000000E+00 1000 shift
2000 5 5 3 1.19209290E-07 2000 shift
3000 5 5 3 1.19209290E-07 3000 shift
4000 5 5 3 0.00000000E+00 4000 shift
5000 5 5 3 0.00000000E+00 5000 shift
6000 5 5 3 0.00000000E+00 6000 shift
7000 5 5 3 0.00000000E+00 7000 shift
8000 5 5 3 5.96046448E-08 8000 shift
4487760 6 5 3 1.19209290E-07 9000 shift
4488760 6 5 3 1.19209290E-07 10000 shift
4489760 6 5 3 0.00000000E+00 11000 shift
4490760 6 5 3 0.00000000E+00 12000 shift
4491760 6 5 3 1.49011612E-08 13000 shift
4492760 6 5 3 2.98023224E-08 14000 shift
4493760 6 5 3 0.00000000E+00 15000 shift
4494760 6 5 3 0.00000000E+00 16000 shift
4495760 6 5 3 1.49011612E-08 17000 shift
done periodic vtx
start rec_bisect: 4495920
done: 0.0%
not connected 1 2 1 0
done: 1.0%
done: 2.0%
done: 3.0%
not connected 1 2 2 0
done: 4.0%
not connected 1 2 3 0
done: 5.0%
done: 6.0%
done: 7.0%
not connected 1 2 4 0
not connected 1 2 5 0
done: 8.0%
not connected 1 2 6 0
done: 9.0%
done: 10.0%
done: 11.0%
done: 12.0%
done: 13.0%
done: 14.0%
not connected 1 2 7 0
done: 15.0%
done: 16.0%
done: 17.0%
not connected 1 2 8 0
done: 18.0%
done: 19.0%
not connected 1 2 9 0
done: 20.0%
not connected 1 2 10 0
done: 21.0%
not connected 1 2 11 0
done: 22.0%
not connected 1 2 12 0
not connected 1 2 13 0
done: 23.0%
not connected 1 2 14 0
not connected 1 2 15 0
done: 24.0%
not connected 1 2 16 0
not connected 1 2 17 0
done: 25.0%
not connected 1 2 18 0
not connected 1 2 19 0
not connected 1 2 20 0
not connected 1 2 21 0
done: 26.0%
not connected 1 2 22 0
not connected 1 2 23 0
done: 27.0%
done: 28.0%
not connected 1 2 24 0
done: 29.0%
done: 30.0%
not connected 1 2 25 0
done: 31.0%
done: 32.0%
not connected 1 2 26 0
done: 33.0%
done: 34.0%
not connected 1 2 27 0
done: 35.0%
done: 36.0%
done: 37.0%
not connected 1 2 28 0
done: 38.0%
done: 39.0%
done: 40.0%
done: 41.0%
not connected 1 2 29 0
done: 42.0%
done: 43.0%
not connected 1 2 30 0
done: 44.0%
not connected 1 2 31 0
done: 45.0%
done: 46.0%
not connected 1 2 32 0
not connected 1 2 33 0
done: 47.0%
not connected 1 2 34 0
done: 48.0%
done: 49.0%
not connected 1 2 35 0
not connected 1 2 36 0
done: 50.0%
not connected 1 2 37 0
done: 51.0%
not connected 1 2 38 0
done: 52.0%
not connected 1 2 39 0
done: 53.0%
done: 54.0%
done: 55.0%
done: 56.0%
done: 57.0%
done: 58.0%
not connected 1 2 40 0
done: 59.0%
not connected 1 2 41 0
done: 60.0%
not connected 1 2 42 0
not connected 1 2 43 0
done: 61.0%
not connected 1 2 44 0
done: 62.0%
not connected 1 2 45 0
not connected 1 2 46 0
done: 63.0%
done: 64.0%
done: 65.0%
done: 66.0%
not connected 1 2 47 0
not connected 1 2 48 0
done: 67.0%
done: 68.0%
done: 69.0%
done: 70.0%
not connected 1 2 49 0
done: 71.0%
not connected 1 2 50 0
done: 72.0%
not connected 1 2 51 0
done: 73.0%
done: 74.0%
not connected 1 2 52 0
done: 75.0%
not connected 1 2 53 0
not connected 1 2 54 0
done: 76.0%
done: 77.0%
done: 78.0%
not connected 1 2 55 0
done: 79.0%
not connected 1 2 56 0
not connected 1 2 57 0
done: 80.0%
not connected 1 2 58 0
done: 81.0%
done: 82.0%
not connected 1 2 59 0
done: 83.0%
done: 84.0%
not connected 1 2 60 0
done: 85.0%
not connected 1 2 61 0
not connected 1 2 62 0
done: 86.0%
not connected 1 2 63 0
done: 87.0%
done: 88.0%
done: 89.0%
done: 90.0%
done: 91.0%
done: 92.0%
not connected 1 2 64 0
done: 93.0%
not connected 1 2 65 0
done: 94.0%
done: 95.0%
done: 96.0%
done: 97.0%
done: 98.0%
done: 99.0%
done: 100.0%
done rec_bisect
writing PIPE.map
-------------- next part --------------
PE 0: MPICH/GNI environment settings:
PE 0: MPICH_GNI_RECV_CQ_SIZE = 40960
PE 0: MPICH_GNI_LOCAL_CQ_SIZE = 8192
PE 0: MPICH_GNI_DEBUG_LEVEL = 0
PE 0: MPICH_GNI_MAX_VSHORT_MSG_SIZE = 464
PE 0: MPICH_GNI_MAX_EAGER_MSG_SIZE = 8192
PE 0: MPICH_GNI_NUM_BUFS = 64
PE 0: MPICH_GNI_NUM_MBOXES = -1
PE 0: MPICH_GNI_RDMA_THRESHOLD = 1024
PE 0: MPICH_GNI_RCVCQ_PROCNUM = 1
PE 0: MPICH_GNI_NDREG_ENTRIES(req.) = 151
PE 0: MPICH_GNI_NDREG_MAXSIZE = 524288
PE 0: MPICH_GNI_NDREG_LAZYMEM = LAZY_ALL
PE 0: MPICH_GNI_DMAPP_INTEROP = 1
PE 0: MPICH_GNI_DYNAMIC_CONN = 1
PE 0: MPICH_GNI_MAX_NUM_RETRIES = 16
PE 0: MPICH_GNI_FORK_MODE = PARTCOPY
PE 0: MPICH_GNI_MBOX_PLACEMENT = PROC
PE 0: MPICH_GNI_VC_MSG_PROTOCOL = MBOX
PE 0: MPICH_GNI_BTE_MULTI_CHANNEL = 1
PE 0: MPICH_GNI_LMT_GET_PATH = 1
PE 0: MPICH_GNI_LMT_PATH = 1
/----------------------------------------------------------\
| _ __ ______ __ __ ______ ____ ____ ____ |
| / | / // ____// //_/ / ____/ / __ \ / __ \ / __ \ |
| / |/ // __/ / ,< /___ \ / / / // / / // / / / |
| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |
| /_/ |_//_____//_/ |_|/_____/ \____/ \____/ \____/ |
| |
|----------------------------------------------------------|
| |
| NEK5000: Open Source Spectral Element Solver |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |
| Version: 1.0rc1 / SVN r618 |
| Web: http://nek5000.mcs.anl.gov |
| |
\----------------------------------------------------------/
Number of processors: 4096
REAL wdsize : 8
INTEGER wdsize : 4
Beginning session:
/cfs/klemming/nobackup/a/anoorani/pipe/Retau1000/new/pipe.rea
timer accuracy: 0.0000000E+00 sec
read .rea file
read .re2 file
byte swap: F 6.543210 -2.9312772E+35
nelgt/nelgv/lelt: 4495920 4495920 1100
lx1 /lx2 /lx3 : 4 2 2
mapping elements to processors
element load imbalance: 1 1097 1098
done :: mapping elements to processors
reading mesh
reading curved sides
reading bc for ifld 1
done :: read .re2 file
0 objects found
done :: read .rea file 114.58 sec
setup mesh topology
Right-handed check complete for 4495920 elements. OK.
setvert3d: 4 86046564 122013924 86046564 86046564
call usrsetvert
done :: usrsetvert
gs_setup: 13275258 unique labels shared
pairwise times (avg, min, max): 0.000561914 0.000431108 0.000673413
crystal router : 0.00190646 0.00186119 0.00197728
used all_to_all method: pairwise
setupds time 3.6859E-01 seconds 0 4 86046564 4495920
8 max multiplicity
done :: setup mesh topology
call usrdat
done :: usrdat
generate geomerty data
vol_t,vol_v: 78.53235632719681 78.53235632719681
done :: generate geomerty data
call usrdat2
done :: usrdat2
regenerate geomerty data 1
vol_t,vol_v: 78.53235632719681 78.53235632719681
done :: regenerate geomerty data 1
verify mesh topology
-1.000000000000000 1.000000000000000 Xrange
-1.000000000000000 1.000000000000000 Yrange
0.000000000000000 25.00000000000000 Zrange
done :: verify mesh topology
118 Parameters from file:/cfs/klemming/nobackup/a/anoorani/pipe/Retau1000/new/pipe.rea
1 1.00000 P001: DENSITY
2 -18850. P002: VISCOS
7 1.00000 P007: RHOCP
8 1.00000 P008: CONDUCT
11 100.0 P011: NSTEPS
12 -1.0000E-05 P012: DT
15 50.0000 P015: IOSTEP
17 1.00000 P017:
18 0.500000E-01 P018: GRID < 0 --> # cells on screen
19 -1.00000 P019: INTYPE
20 10.0000 P020: NORDER
21 0.100000E-05 P021: DIVERGENCE
22 0.100000E-06 P022: HELMHOLTZ
24 0.100000E-01 P024: TOLREL
25 0.100000E-01 P025: TOLABS
26 1.00000 P026: COURANT/NTAU
27 3.00000 P027: TORDER
28 0.00000 P028: TORDER: mesh velocity (0: p28=p27)
54 -3.00000 P054: fixed flow rate dir: |p54|=1,2,3=x,y,z
55 1.00000 P055: vol.flow rate (p54>0) or Ubar (p54<0)
65 1.00000 P065: #iofiles (eg, 0 or 64); <0 --> sep. dirs
66 4.00000 P066: output : <0=ascii, else binary
67 4.00000 P067: restart: <0=ascii, else binary
69 50000.0 P069: : : frequency of srf dump
93 20.0000 P093: Number of previous pressure solns saved
94 3.00000 P094: start projecting velocity after p94 step
95 5.00000 P095: start projecting pressure after p95 step
99 3.00000 P099: dealiasing: <0--> off/3--> old/4--> new
102 1.00000 P102: Dump out divergence at each time step
103 0.050000 P103: weight of stabilizing filter (.01)
IFTRAN = T
IFFLOW = T
IFHEAT = F
IFSPLIT = F
IFLOMACH = F
IFUSERVP = F
IFUSERMV = F
IFSTRS = F
IFCHAR = T
IFCYCLIC = F
IFAXIS = F
IFMVBD = F
IFMELT = F
IFMODEL = F
IFKEPS = F
IFMOAB = F
IFSYNC = T
IFVCOR = T
IFINTQ = F
IFCWUZ = F
IFSWALL = F
IFGEOM = F
IFSURT = F
IFWCNO = F
IFTMSH for field 1 = F
IFADVC for field 1 = T
IFNONL for field 1 = F
Dealiasing enabled, lxd= 6
Estimated eigenvalues
EIGAA = 1.650197855862141
EIGGA = 12385388.91135105
EIGAE = 1.5791367041742974E-002
EIGAS = 7.9744816586921851E-004
EIGGE = 12385388.91135105
EIGGS = 2.000000000000000
verify mesh topology
-1.000000000000000 1.000000000000000 Xrange
-1.000000000000000 1.000000000000000 Yrange
0.000000000000000 25.00000000000000 Zrange
done :: verify mesh topology
E-solver strategy: 1 itr
mg_nx: 1 3
mg_ny: 1 3
mg_nz: 1 3
call usrsetvert
done :: usrsetvert
gs_setup: 1393086 unique labels shared
pairwise times (avg, min, max): 0.000286875 0.000198293 0.000347185
crystal router : 0.000353284 0.000346708 0.000362897
used all_to_all method: pairwise
setupds time 3.3200E-02 seconds 1 2 4565612 4495920
setvert3d: 4 86046564 122013924 86046564 86046564
call usrsetvert
done :: usrsetvert
gs_setup: 13275258 unique labels shared
pairwise times (avg, min, max): 0.000565474 0.000431323 0.000666308
crystal router : 0.00192211 0.00188022 0.00198519
used all_to_all method: pairwise
setupds time 3.6968E-01 seconds 2 4 86046564 4495920
setup h1 coarse grid, nx_crs= 2
call usrsetvert
done :: usrsetvert
gs_setup: 1393086 unique labels shared
pairwise times (avg, min, max): 0.00027245 0.000212216 0.000325584
crystal router : 0.000303 0.000294399 0.000315809
used all_to_all method: crystal router
gs_setup: 1393086 unique labels shared
pairwise times (avg, min, max): 0.000276989 0.000202703 0.000333595
crystal router : 0.000296147 0.000291204 0.000304508
used all_to_all method: crystal router
Application 727540 exit codes: 143
Application 727540 resources: utime ~406010s, stime ~1268s
More information about the Nek5000-users
mailing list