TACC: Starting up job 1362981 TACC: Setting up parallel environment for OpenMPI mpirun. TACC: Setup complete. Running job script. TACC: starting parallel tasks... /----------------------------------------------------------\ | _ __ ______ __ __ ______ ____ ____ ____ | | / | / // ____// //_/ / ____/ / __ \ / __ \ / __ \ | | / |/ // __/ / ,< /___ \ / / / // / / // / / / | | / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / | | /_/ |_//_____//_/ |_|/_____/ \____/ \____/ \____/ | | | |----------------------------------------------------------| | | | NEK5000: Open Source Spectal Element Solver | | COPYRIGHT (c) 2008 UCHICAGO ARGONNE, LLC | | Version: 1.0rc1 / SVN r491 | | Web: http://nek5000.mcs.anl.gov | | | \----------------------------------------------------------/ Number of processors: 512 REAL wdsize : 8 INTEGER wdsize : 4 timer accuracy: 7.3909760E-07 sec read .rea file nelgt/nelgv/lelt: 23712 23712 50 lx1 /lx2 /lx3 : 11 11 11 mapping elements to processors RANK 0 IEG 948 949 951 952 2941 2942 3563 3976 4010 5569 6209 6337 18475 18828 20313 20786 20787 20788 20789 20790 20791 20792 22065 22443 22445 22446 22448 22609 22778 22780 22783 22784 22795 22796 22800 22874 22875 22876 22877 22878 22879 22880 22891 22893 22894 22896 done :: mapping elements to processors 0 objects found done :: read .rea file 22.927 sec setup mesh topology Right-handed check complete for 23712 elements. OK. setvert3d: 11 6526380 23812428 6526380 6526380 gs_setup: 2285823 unique labels shared pairwise times (avg, min, max): 0.00139187 0.00105052 0.00167079 crystal router : 0.00408029 0.00399039 0.00417299 used all_to_all method: pairwise setupds time 9.5818E-01 seconds 0 11 6526380 23712 10 max multiplicity done :: setup mesh topology call usrdat done :: usrdat generate geomerty data vol_t,vol_v: 5.614000551013292 5.614000551013292 done :: generate geomerty data call usrdat2 done :: usrdat2 regenerate geomerty data 1 vol_t,vol_v: 5.614000551013292 5.614000551013292 done :: regenerate geomerty data 1 verify mesh topology -5.700000000000000 1.950000000000000 Xrange -0.6787000000000001 1.500000000000000 Yrange 0.000000000000000 0.7500000000000000 Zrange done :: verify mesh topology 103 Parameters from file:/scratch/01420/s0j3095/LES_Run1/final.rea 1 1.00000 DENSITY 2 2.4E-05 VISCOS 7 1.00000 RHOCP 8 1.00000 CONDUCT 11 2000 NSTEPS 12 0.28E-03 DT 15 200.0 IOSTEP 18 -40.0000 GRID 19 -1.00000 INTYPE 20 5.00000 NORDER 21 0.100000E-05 DIVERGENCE 22 0.100000E-07 HELMHOLTZ 24 0.100000E-09 TOLREL 25 0.100000E-09 TOLABS 26 3.0000 COURANT 27 2.00000 TORDER 28 0.00000 TMESH 33 100.000 MXGEOM 34 100.000 MAXNEW 35 0.100000E-03 36 0.100000E-03 XMAGNET 37 0.100000E-01 NGRIDS 49 0.00000 65 1.00000 new file each dump (1) 66 4.00000 write form (0), unform(1), universal(2) 67 4.00000 read form (0), unform (1), universal(2) 93 30.0000 Number of pressure sol'ns saved 94 5.00000 start saving velocity soln's after this 95 5.00000 start saving pressure soln's after this 99 3.00000 p99=3 ==> dealiasing turned on 101 3.00000 p101 # add'l filter modes 102 1.00000 p102 103 0.500000E-01 p103 filter weight IFTRAN = T IFFLOW = T IFHEAT = F IFSPLIT = T IFLOMACH = F IFUSERVP = F IFUSERMV = F IFSTRS = F IFCHAR = T IFCYCLIC = F IFAXIS = F IFMVBD = F IFMELT = F IFMODEL = F IFKEPS = F IFMOAB = F IFSYNC = T IFVCOR = F IFINTQ = F IFCWUZ = F IFSWALL = F IFGEOM = F IFSURT = F IFWCNO = F IFTMSH for field 1 = F IFADVC for field 1 = T IFNONL for field 1 = F Dealiasing enabled, lxd= 16 Estimated eigenvalues EIGAA = 6.597950311047805 EIGGA = 389227649.1988692 EIGAE = 0.1686463223732642 EIGAS = 2.1300342584029187E-003 EIGGE = 389227649.1988692 EIGGS = 2.000000000000000 verify mesh topology -5.700000000000000 1.950000000000000 Xrange -0.6787000000000001 1.500000000000000 Yrange 0.000000000000000 0.7500000000000000 Zrange done :: verify mesh topology E-solver strategy: 0 itr setup h1 coarse grid, nx_crs= 2 gs_setup: 18021 unique labels shared pairwise times (avg, min, max): 0.000903117 0.000600815 0.00114081 crystal router : 0.000235383 0.000220394 0.000247693 all reduce : 0.0119909 0.0118397 0.012107 used all_to_all method: crystal router gs_setup: 17827 unique labels shared pairwise times (avg, min, max): 0.000907303 0.000612903 0.00113671 crystal router : 0.000361389 0.000323296 0.000400591 all reduce : 0.0112699 0.0112065 0.0113289 used all_to_all method: crystal router done :: setup h1 coarse grid 1.491070985794067 sec call usrdat3 done :: usrdat3 set initial conditions Checking restart options: restart0.f0007 Reading checkpoint data 0 0 OPEN: restart0.f0007 byte swap: F 6.543210 -2.9312772E+35 0 2.3322E+00 done :: Read checkpoint data avg data-throughput = 124.4MBps io-nodes = 1 perform dssum test ... xyz min -5.7000 -0.67870 0.0000 uvwpt min -1.4977 -2.1804 -1.6453 -0.44631 0.99999E-20 PS min 0.0000 0.99000E+22 xyz max 1.9500 1.5000 0.75000 uvwpt max 2.5660 2.5528 1.7819 5.1821 0.10000E-18 PS max 0.0000 -0.99000E+22 Restart: recompute geom. factors. regenerate geomerty data 1 vol_t,vol_v: 5.614000551013292 5.614000551013292 done :: regenerate geomerty data 1 done :: set initial conditions call userchk 0 2.3322E+00 2.5660E+00 2.5528E+00 2.5032E+00 2.5528E+00 umx div: davg: 1.0000E-01 3.3473E-02 4.4722E+05 0 2.33217E+00 1.50000E+00 6.67366E-02 3.37146E+01 3.37146E+01 cdiv 0 442848 0.000E+00 3.371E+01 0.000E+00 1.000E-01 3.371E+01 4.454E-03divmnmx done :: userchk dofs: 23644452 31560672 648108 Initial time: 0.2332169E+01 Initialization successfully completed 32.077 sec 1 Hmholtz VELZ: 12 1.9288E-09 6.5237E+00 1.0000E-08 L1/L2 DIV(V) : 1.9222E-01 2.3876E+00 L1/L2 QTL : 1.9180E-01 2.3014E+00 L1/L2 DIV(V)-QTL: 4.1645E-04 6.4129E-01 WARNING: DIV(V)-QTL too large! 1 2.3324E+00 1.6593E+01 Fluid done filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0031 0.0125 0.0281 0.0500 filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9969 0.9875 0.9719 0.9500 1 2.3324E+00 2.5671E+00 2.5438E+00 2.4989E+00 2.5438E+00 umx 1 2.33245E+00 1.50000E+00 6.67366E-02 3.37146E+01 3.37146E+01 cdiv 1 442848 0.000E+00 3.371E+01 0.000E+00 1.000E-01 3.371E+01 4.454E-03divmnmx Step 2, t= 2.3327287E+00, DT= 2.8000000E-04, C= 2.037 1.9022E+01 1.9022E+01 Solving for fluid 1.0000000000000000E-008 p22 2 1 2 PRES gmres: 302 9.8685E-07 1.0000E-06 2.8685E-01 3.9623E+00 1.5345E+01 1.0000000000000000E-008 p22 2 1 2 Hmholtz VELX: 10 4.5250E-09 9.4484E+00 1.0000E-08 1.0000000000000000E-008 p22 2 1 2 Hmholtz VELY: 10 4.6388E-09 8.6863E+00 1.0000E-08 1.0000000000000000E-008 p22 2 1 2 Hmholtz VELZ: 10 4.6651E-09 9.7576E+00 1.0000E-08 L1/L2 DIV(V) : 1.9224E-01 2.3908E+00 L1/L2 QTL : 1.9180E-01 2.3014E+00 L1/L2 DIV(V)-QTL: 4.3180E-04 6.5228E-01 WARNING: DIV(V)-QTL too large! 2 2.3327E+00 1.9137E+01 Fluid done 2 2.3327E+00 2.5682E+00 2.5340E+00 2.4937E+00 2.5340E+00 umx 2 2.33273E+00 1.50000E+00 6.67366E-02 3.37146E+01 3.37146E+01 cdiv 2 442848 0.000E+00 3.371E+01 0.000E+00 1.000E-01 3.371E+01 4.454E-03divmnmx Step 3, t= 2.3330087E+00, DT= 2.8000000E-04, C= 2.032 3.8927E+01 1.9904E+01 Solving for fluid 1.0000000000000000E-008 p22 3 1 3 PRES gmres: 282 9.8878E-07 1.0000E-06 6.4185E-02 3.7055E+00 1.4305E+01 1.0000000000000000E-008 p22 3 1 3 Hmholtz VELX: 10 4.5194E-09 9.4409E+00 1.0000E-08 1.0000000000000000E-008 p22 3 1 3 Hmholtz VELY: 10 4.6262E-09 8.6778E+00 1.0000E-08 1.0000000000000000E-008 p22 3 1 3 Hmholtz VELZ: 10 4.6529E-09 9.7524E+00 1.0000E-08 L1/L2 DIV(V) : 1.9224E-01 2.3923E+00 L1/L2 QTL : 1.9180E-01 2.3014E+00 L1/L2 DIV(V)-QTL: 4.3204E-04 6.5764E-01 WARNING: DIV(V)-QTL too large! 3 2.3330E+00 1.8092E+01 Fluid done 3 2.3330E+00 2.5688E+00 2.5380E+00 2.4880E+00 2.5380E+00 umx 3 2.33301E+00 1.50000E+00 6.67366E-02 3.37146E+01 3.37146E+01 cdiv 3 442848 0.000E+00 3.371E+01 0.000E+00 1.000E-01 3.371E+01 4.454E-03divmnmx ---- ---- ---- ---- Step 2000, t= 2.8921687E+00, DT= 2.8000000E-04, C= 1.798 2.1608E+04 1.1367E+01 Solving for fluid 2000 PRES alph1n 1.1133E+02 4.7563E-01 2.3407E+02 10 2000 PRES halpha 10 2.4595E-01 1.3095E-01 6.3471E-02 2.4926E-02 1.9414E-02 1.6537E-02 1.0284E-02 4.9269E-03 2.3130E-03 1.7983E-03 2000 PRES gmres: 113 9.9742E-07 1.0000E-06 3.3971E-05 1.5480E+00 6.1321E+00 2000 Hmholtz VELX: 10 5.3910E-09 8.8856E+00 1.0000E-08 2000 Hmholtz VELY: 10 5.3303E-09 8.3908E+00 1.0000E-08 2000 Hmholtz VELZ: 10 5.1876E-09 9.4266E+00 1.0000E-08 L1/L2 DIV(V) : 1.9224E-01 2.3948E+00 L1/L2 QTL : 1.9180E-01 2.3014E+00 L1/L2 DIV(V)-QTL: 4.3321E-04 6.6672E-01 WARNING: DIV(V)-QTL too large! 2000 2.8922E+00 1.0101E+01 Fluid done 2000 2.8922E+00 2.5997E+00 2.1239E+00 2.5997E+00 2.1239E+00 umx 2000 2.89217E+00 1.50000E+00 6.67366E-02 3.37146E+01 3.37146E+01 cdiv 2000 442848 0.000E+00 3.371E+01 0.000E+00 1.000E-01 3.371E+01 4.454E-03divmnmx