<div dir="ltr"><pre>This problem compiled with PPLIST="NEKNEK"<br><br>/----------------------------------------------------------\\<br>| _ __ ______ __ __ ______ ____ ____ ____ |<br>| / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |<br>| / |/ // __/ / ,< /___ \\ / / / // / / // / / / |<br>| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |<br>| /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ |<br>| |<br>|----------------------------------------------------------|<br>| |<br>| NEK5000: Open Source Spectral Element Solver |<br>| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |<br>| Version: 1.0rc1 / SVN r1115 |<br>| Web: <a href="http://nek5000.mcs.anl.gov">http://nek5000.mcs.anl.gov</a> |<br>| |<br>\\----------------------------------------------------------/<br> <br><br> Number of processors: 192<br> REAL wdsize : 8<br> INTEGER wdsize : 4<br> Timer accuracy : 7.87E-07<br>/----------------------------------------------------------\\<br>| _ __ ______ __ __ ______ ____ ____ ____ |<br>| / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |<br>| / |/ // __/ / ,< /___ \\ / / / // / / // / / / |<br>| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |<br>| /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ |<br>| |<br>|----------------------------------------------------------|<br>| |<br>| NEK5000: Open Source Spectral Element Solver |<br>| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |<br>| Version: 1.0rc1 / SVN r1115 |<br>| Web: <a href="http://nek5000.mcs.anl.gov">http://nek5000.mcs.anl.gov</a> |<br>| |<br>\\----------------------------------------------------------/<br> <br><br> Number of processors: 24<br> REAL wdsize : 8<br> INTEGER wdsize : 4<br> Timer accuracy : 5.25E-07<br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.rea <br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.re2 <br> byte swap: F 6.543210 -2.9312772E+35<br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.rea <br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.re2 <br> byte swap: F 6.543210 -2.9312772E+35<br><br> mapping elements to processors<br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe.map <br><br> mapping elements to processors<br> Reading /home/itp/ryzhenkov/nek5_calculate/annular_jet/swirl_0.0/G7/pipe_per.map <br> 0 213 213 5120 5120 NELV<br> 3 213 213 5120 5120 NELV<br> 4 213 213 5120 5120 NELV<br> 5 213 213 5120 5120 NELV<br> 1 213 213 5120 5120 NELV<br> 2 213 213 5120 5120 NELV<br> 6 213 213 5120 5120 NELV<br> 7 213 213 5120 5120 NELV<br> 8 213 213 5120 5120 NELV<br> 9 213 213 5120 5120 NELV<br> 10 213 213 5120 5120 NELV<br> 11 213 213 5120 5120 NELV<br> element load imbalance: 1 418 419<br> done :: mapping elements to processors<br> <br> reading mesh <br> 14 213 213 5120 5120 NELV<br> RANK 0 IEG 3873 3874 3875 3876 3877 3878 3879 3880<br> 3881 3882 3883 3884 3885 3886 3887 3888<br> 3889 3890 3891 3892 3893 3894 3895 3896<br> 3897 3898 3899 3900 3901 3902 3903 3904<br> 4129 4130 4131 4132 4133 4134 4135 4136<br> 4137 4138 4139 4140 4141 4142 4143 4144<br> 4145 4146 4147 4148 4149 4150 4151 4152<br> 16 214 214 5120 5120 NELV<br> 4153 4154 4155 4156 4157 4158 4159 4160<br> 4353 4354 4355 4356 4361 4362 4363 4364<br> 4369 4370 4371 4372 4377 4378 4379 4380<br> 4385 4386 4387 4388 4389 4390 4391 4392<br> 4393 4394 4395 4396 4397 4398 4399 4400<br> 4401 4402 4403 4404 4405 4406 4407 4408<br> 4409 4410 4411 4412 4413 4414 4415 4416<br> 4609 4610 4611 4612 4613 4617 4618 4619<br> 4620 4621 4625 4626 4627 4628 4633 4634<br> 4635 4636 4637 4638 4641 4642 4643 4644<br> 4645 4646 4647 4648 4649 4650 4651 4652<br> 17 214 214 5120 5120 NELV<br> 4653 4654 4655 4656 4657 4658 4659 4660<br> 4661 4662 4663 4664 4665 4666 4667 4668<br> 4669 4670 4671 4672 4865 4866 4867 4868<br> 4869 4873 4874 4875 4876 4877 4881 4882<br> 4883 4884 4889 4890 4891 4897 4898 4899<br> 4900 4901 4902 4903 4904 4905 4906 4907<br> 12 213 213 5120 5120 NELV<br> 4908 4909 4910 4911 4912 4913 4914 4915<br> 4916 4917 4918 4919 4920 4921 4922 4923<br> 4924 4925 4926 4927 4928<br> 15 213 213 5120 5120 NELV<br> element load imbalance: 1 213 214<br> done :: mapping elements to processors<br> <br> 18 214 214 5120 5120 NELV<br> reading mesh <br> 22 214 214 5120 5120 NELV<br> reading curved sides <br> 13 213 213 5120 5120 NELV<br> reading bc for ifld 1<br> 20 214 214 5120 5120 NELV<br> done :: read .re2 file<br> <br> 21 214 214 5120 5120 NELV<br> 0 objects found<br> 23 214 214 5120 5120 NELV<br> 118 Parameters from file:<br> 1 1.00000 p001 DENSITY<br> 2 -8900.00 p002 VISCOS<br> 19 214 214 5120 5120 NELV<br> 7 1.00000 p007 RHOCP<br> 8 -8900.00 p008 CONDUCT<br> 11 100000.0 p011 NSTEPS<br> 12 -3.0E-04 p012 DT<br> 15 1000.0 p015 IOSTEP<br> 18 5.0E-02 p018 GRID < 0 --> # cells on screen<br> 19 -1.00000 p019 INTYPE<br> 20 4.00000 p020 NORDER<br> 21 1.0E-06 p021 DIVERGENCE<br> 22 1.0E-08 p022 HELMHOLTZ<br> 24 1.0E-02 p024 TOLREL<br> 25 1.0E-02 p025 TOLABS<br> 26 0.50000 p026 COURANT/NTAU<br> 27 2.00000 p027 TORDER<br> 28 1.00000 p028 TORDER: mesh velocity (0: p28=p27)<br> 66 0.00000 p066 output : <0=ascii, else binary<br> 67 0.00000 p067 restart: <0=ascii, else binary<br> 93 20.0000 p093 Number of previous pressure solns saved<br> 95 5.00000 p095 start projecting pressure after p95 step<br> 99 3.00000 p099 dealiasing: <0--> off/3--> old/4--> new<br> 101 1.00000 p101 Number of additional modes to filter<br> 102 1.00000 p102 Dump out divergence at each time step<br> 103 9.500000E-01 p103 weight of stabilizing filter (.01)<br> <br> nelgt/nelgv/lelt: 5120 5120 4000<br> lx1 /lx2 /lx3 : 12 12 12<br> done :: read .rea file 0.54454 sec<br><br> setup mesh topology<br> Right-handed check complete for 5120 elements. OK.<br> setvert3d: 12 1772160 6892160 1772160 1772160<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 340773 unique labels shared<br> pairwise times (avg, min, max): 0.000373997 0.000350785 0.000392795<br> reading curved sides <br> crystal router : 0.00137192 0.00134859 0.0013859<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 2.16376e+06 2106108 2251852<br> buffer bytes (avg, min, max): 476643 410800 577024<br> setupds time 3.9776E-01 seconds 0 12 1772160 5120<br> 8 max multiplicity<br> done :: setup mesh topology<br> <br> call usrdat<br> done :: usrdat<br><br> generate geometry data<br> reading bc for ifld 1<br> done :: read .re2 file<br> <br> 0 objects found<br> 118 Parameters from file:<br> 1 1.00000 p001 DENSITY<br> 2 -8900.00 p002 VISCOS<br> 7 1.00000 p007 RHOCP<br> 8 -8900.00 p008 CONDUCT<br> 11 100000.0 p011 NSTEPS<br> 12 -3.0E-04 p012 DT<br> 15 1000.0 p015 IOSTEP<br> 18 5.0E-02 p018 GRID < 0 --> # cells on screen<br> 19 -1.00000 p019 INTYPE<br> 20 4.00000 p020 NORDER<br> 21 1.0E-06 p021 DIVERGENCE<br> 22 1.0E-08 p022 HELMHOLTZ<br> 24 1.0E-02 p024 TOLREL<br> 25 1.0E-02 p025 TOLABS<br> 26 0.50000 p026 COURANT/NTAU<br> 27 2.00000 p027 TORDER<br> 28 1.00000 p028 TORDER: mesh velocity (0: p28=p27)<br> 66 0.00000 p066 output : <0=ascii, else binary<br> 67 0.00000 p067 restart: <0=ascii, else binary<br> 93 20.0000 p093 Number of previous pressure solns saved<br> 95 5.00000 p095 start projecting pressure after p95 step<br> 99 3.00000 p099 dealiasing: <0--> off/3--> old/4--> new<br> 101 1.00000 p101 Number of additional modes to filter<br> 102 1.00000 p102 Dump out divergence at each time step<br> 103 9.500000E-01 p103 weight of stabilizing filter (.01)<br> <br> nelgt/nelgv/lelt: 80320 80320 4000<br> lx1 /lx2 /lx3 : 12 12 12<br> done :: read .rea file 1.1102 sec<br><br> setup mesh topology<br> Right-handed check complete for 80320 elements. OK.<br> setvert3d: 12 26906208 107226208 26906208 26906208<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 5553221 unique labels shared<br> pairwise times (avg, min, max): done :: generate geometry data<br> <br> call usrdat2<br>0.0016136 0.0012603 0.00186529<br> crystal router : 0.00763995 0.00749459 0.00784841<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 4.35947e+06 3848020 4681812<br> buffer bytes (avg, min, max): 976972 546496 1320848<br> setupds time 2.1430E+00 seconds 0 12 26906208 80320<br> 8 max multiplicity<br> done :: setup mesh topology<br> <br> call usrdat<br> done :: usrdat<br><br> generate geometry data<br> done :: generate geometry data<br> <br> call usrdat2<br> initializing intpts(), tol= 1.000000000000000E-013<br> initializing intpts(), tol= 1.000000000000000E-013<br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 1536 pipe_per <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> done :: usrdat2<br><br> done :: usrdat2<br><br> found 0 pipe <br> regenerate geometry data 1<br> regenerate geometry data 1<br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> found 0 pipe <br> done :: regenerate geometry data 1<br> <br> verify mesh topology<br> -1.00000000000000 1.00000000000000 Xrange<br> -1.00000000000000 1.00000000000000 Yrange<br> 0.000000000000000E+000 4.50000000000000 Zrange<br> done :: verify mesh topology<br> <br> IFTRAN = T<br> IFFLOW = T<br> IFHEAT = F<br> IFSPLIT = T<br> IFLOMACH = F<br> IFUSERVP = F<br> IFUSERMV = F<br> IFSTRS = T<br> IFCHAR = F<br> IFCYCLIC = F<br> IFAXIS = F<br> IFMVBD = F<br> IFMELT = F<br> IFMODEL = F<br> IFKEPS = F<br> IFMOAB = F<br> IFNEKNEK = T<br> IFSYNC = T<br> <br> IFVCOR = T<br> IFINTQ = F<br> IFCWUZ = F<br> IFSWALL = F<br> IFGEOM = F<br> IFSURT = F<br> IFWCNO = F<br> IFCMT = F<br> IFVISC = F<br> IFFLTR = F<br> <br> IFTMSH for field 1 = F<br> IFADVC for field 1 = T<br> IFNONL for field 1 = F<br> <br> Dealiasing enabled, lxd= 18<br> done :: regenerate geometry data 1<br> <br> verify mesh topology<br> -12.0000000000000 12.0000000000000 Xrange<br> -12.0000000000000 12.0000000000000 Yrange<br> 0.000000000000000E+000 36.0000000000000 Zrange<br> done :: verify mesh topology<br> <br> IFTRAN = T<br> IFFLOW = T<br> IFHEAT = F<br> IFSPLIT = T<br> IFLOMACH = F<br> IFUSERVP = F<br> IFUSERMV = F<br> IFSTRS = T<br> IFCHAR = F<br> IFCYCLIC = F<br> IFAXIS = F<br> IFMVBD = F<br> IFMELT = F<br> IFMODEL = F<br> IFKEPS = F<br> IFMOAB = F<br> IFNEKNEK = T<br> IFSYNC = T<br> <br> IFVCOR = F<br> IFINTQ = F<br> IFCWUZ = F<br> IFSWALL = F<br> IFGEOM = F<br> IFSURT = F<br> IFWCNO = F<br> IFCMT = F<br> IFVISC = F<br> IFFLTR = F<br> <br> IFTMSH for field 1 = F<br> IFADVC for field 1 = T<br> IFNONL for field 1 = F<br> <br> Dealiasing enabled, lxd= 18<br> <br> Estimated eigenvalues<br> EIGAA = 1.80739669073447 <br> EIGGA = 13884400.9520853 <br> EIGAE = 0.487387871658733 <br> EIGAS = 2.247191011235953E-002<br> EIGGE = 13884400.9520853 <br> EIGGS = 2.00000000000000 <br> <br> verify mesh topology<br> -1.00000000000000 1.00000000000000 Xrange<br> -1.00000000000000 1.00000000000000 Yrange<br> 0.000000000000000E+000 4.50000000000000 Zrange<br> done :: verify mesh topology<br> <br> E-solver strategy: 0 itr<br> mg_nx: 1 7 11<br> mg_ny: 1 7 11<br> mg_nz: 1 7 11<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 2723 unique labels shared<br> pairwise times (avg, min, max): 2.96265e-05 2.85149e-05 3.04937e-05<br> crystal router : 1.43409e-05 1.38998e-05 1.49012e-05<br> all reduce : 6.69807e-05 6.60896e-05 6.82116e-05<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 66207.7 54708 77100<br> buffer bytes (avg, min, max): 9284 7536 10800<br> setupds time 4.3750E-03 seconds 1 2 5760 5120<br> setvert3d: 4 103040 144000 103040 103040<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 25165 unique labels shared<br> pairwise times (avg, min, max): 2.84771e-05 2.72989e-05 2.95877e-05<br> crystal router : 4.75297e-05 4.67062e-05 4.81129e-05<br> all reduce : 0.00061613 0.000604296 0.000622106<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 171156 165372 179980<br> buffer bytes (avg, min, max): 39800 34608 47744<br> setupds time 2.7411E-02 seconds 2 4 103040 5120<br> setvert3d: 8 681600 1787520 681600 681600<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 137801 unique labels shared<br> pairwise times (avg, min, max): 8.8045e-05 8.38041e-05 9.10997e-05<br> crystal router : 0.000316096 0.00030961 0.000320601<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 886769 861564 925228<br> buffer bytes (avg, min, max): 197997 170992 239168<br> setupds time 1.0851E-01 seconds 3 8 681600 5120<br> setvert3d: 10 1162880 3784320 1162880 1162880<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 227995 unique labels shared<br> pairwise times (avg, min, max): 0.000147719 0.00013988 0.000153089<br> crystal router : 0.000699271 0.000687408 0.000709796<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 1.45509e+06 1415292 1515868<br> buffer bytes (avg, min, max): 322264 277968 389792<br> setupds time 1.9729E-01 seconds 4 10 1162880 5120<br> regenerate geometry data 1<br> NOTE: All elements deformed , param(59) ^=0<br> done :: regenerate geometry data 1<br> <br> h1_mg_nx: 1 3 11<br> h1_mg_ny: 1 3 11<br> h1_mg_nz: 1 3 11<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 2723 unique labels shared<br> pairwise times (avg, min, max): 2.57045e-05 2.50816e-05 2.63929e-05<br> crystal router : 1.31597e-05 1.27077e-05 1.35183e-05<br> all reduce : 6.57449e-05 6.45876e-05 6.69956e-05<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 66207.7 54708 77100<br> buffer bytes (avg, min, max): 9284 7536 10800<br> setupds time 3.6449E-03 seconds 5 2 5760 5120<br> setvert3d: 4 103040 144000 103040 103040<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 25165 unique labels shared<br> pairwise times (avg, min, max): 2.73794e-05 2.62976e-05 2.83003e-05<br> crystal router : 4.36117e-05 4.29153e-05 4.42028e-05<br> all reduce : 0.000558805 0.000551605 0.000561714<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 171156 165372 179980<br> buffer bytes (avg, min, max): 39800 34608 47744<br> setupds time 2.3185E-02 seconds 6 4 103040 5120<br> setvert3d: 4 103040 144000 103040 103040<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 25165 unique labels shared<br> pairwise times (avg, min, max): 3.27577e-05 3.13044e-05 3.38078e-05<br> crystal router : 4.4166e-05 4.32968e-05 4.48942e-05<br> all reduce : 0.000555371 0.00054791 0.000558496<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 171156 165372 179980<br> buffer bytes (avg, min, max): 39800 34608 47744<br> setupds time 2.2302E-02 seconds 7 4 103040 5120<br> setvert3d: 6 328320 656000 328320 328320<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 70191 unique labels shared<br> pairwise times (avg, min, max): 5.42253e-05 5.29051e-05 5.60999e-05<br> crystal router : 0.000130047 0.000128007 0.000131607<br> all reduce : 0.00147395 0.0014652 0.00147929<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 458791 444924 479932<br> buffer bytes (avg, min, max): 103843 89872 125152<br> setupds time 6.3579E-02 seconds 8 6 328320 5120<br> setvert3d: 12 1772160 6892160 1772160 1772160<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 340773 unique labels shared<br> pairwise times (avg, min, max): 0.000211759 0.000206304 0.00021801<br> crystal router : 0.00135557 0.00133109 0.00137<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 2.16376e+06 2106108 2251852<br> buffer bytes (avg, min, max): 476643 410800 577024<br> setupds time 3.5228E-01 seconds 9 12 1772160 5120<br> setvert3d: 14 2509440 11356800 2509440 2509440<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 476135 unique labels shared<br> pairwise times (avg, min, max): 0.000281647 0.000266504 0.000291586<br> crystal router : 0.00225826 0.0022172 0.00228479<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 3.01277e+06 2934012 3133180<br> buffer bytes (avg, min, max): 661133 569488 800864<br> setupds time 6.1860E-01 seconds 10 14 2509440 5120<br> setup h1 coarse grid, nx_crs= 2<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 2723 unique labels shared<br> pairwise times (avg, min, max): 1.48118e-05 1.41859e-05 1.58072e-05<br> crystal router : 1.33614e-05 1.21117e-05 1.38044e-05<br> all reduce : 6.4381e-05 6.33001e-05 6.53982e-05<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 66207.7 54708 77100<br> buffer bytes (avg, min, max): 9284 7536 10800<br> <br> Estimated eigenvalues<br> EIGAA = 1.396163174022414E-002<br> EIGGA = 26467996.3219686 <br> EIGAE = 7.615435494667707E-003<br> EIGAS = 4.545454545454543E-002<br> EIGGE = 26467996.3219686 <br> EIGGS = 2.00000000000000 <br> <br> verify mesh topology<br> -12.0000000000000 12.0000000000000 Xrange<br> -12.0000000000000 12.0000000000000 Yrange<br> 0.000000000000000E+000 36.0000000000000 Zrange<br> done :: setup h1 coarse grid 0.402811050415039 sec<br> <br> call usrdat3<br> done :: usrdat3<br><br> set initial conditions<br> Checking restart options: pipe_per0.f00010 <br> nekuic (1) for ifld 1<br> Reading checkpoint data<br> 0 0 OPEN: pipe_per0.f00010<br> byte swap: F 6.543210 -2.9312772E+35<br> done :: verify mesh topology<br> <br> E-solver strategy: 0 itr<br> mg_nx: 1 7 11<br> mg_ny: 1 7 11<br> mg_nz: 1 7 11<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 40651 unique labels shared<br> pairwise times (avg, min, max): 0.000218946 0.00018239 0.000236607<br> crystal router : 5.72741e-05 5.60999e-05 5.85079e-05<br> all reduce : 0.00148923 0.00145738 0.0015028<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 171689 112124 221476<br> buffer bytes (avg, min, max): 18153.2 12304 23232<br> setupds time 7.1854E-02 seconds 1 2 83038 80320<br> setvert3d: 4 1550072 2192632 1550072 1550072<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 400269 unique labels shared<br> pairwise times (avg, min, max): 0.000248798 0.000221109 0.000264406<br> crystal router : 0.000225847 0.000221992 0.000230503<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 1.41297e+06 878540 1853668<br> buffer bytes (avg, min, max): 151639 98096 195920<br> setupds time 7.4004E-02 seconds 2 4 1550072 80320<br> setvert3d: 8 10330540 27679660 10330540 10330540<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 2233849 unique labels shared<br> pairwise times (avg, min, max): 0.000514414 0.000456595 0.000552988<br> crystal router : 0.00232402 0.00227511 0.00238209<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 1.77983e+06 1567956 1917012<br> buffer bytes (avg, min, max): 405180 225984 547728<br> setupds time 4.5856E-01 seconds 3 8 10330540 80320<br> setvert3d: 10 17643974 58767814 17643974 17643974<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 3707811 unique labels shared<br> pairwise times (avg, min, max): 0.00074541 0.000669909 0.000782418<br><br> 0 2.8664E+02 done :: Read checkpoint data<br> avg data-throughput = 84.5MBps<br> io-nodes = 1<br><br> crystal router : xyz min -1.0000 -1.0000 0.0000 <br> uvwpt min -0.34051 -0.37479 -0.16102E-01 -0.90090E-01 0.0000 <br> PS min 0.0000 0.0000 0.99000E+22<br> xyz max 1.0000 1.0000 4.5000 <br> uvwpt max 0.32898 0.39774 1.3656 0.78843E-01 0.0000 <br> PS max 0.0000 0.0000 -0.99000E+22<br> Restart: recompute geom. factors.<br> regenerate geometry data 1<br>0.00430806 0.00423291 0.00441771<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 2.92727e+06 2581844 3147348<br> buffer bytes (avg, min, max): 660122 368832 892432<br> setupds time 1.0661E+00 seconds 4 10 17643974 80320<br> NOTE: All elements deformed , param(59) ^=0<br> done :: regenerate geometry data 1<br> <br> regenerate geometry data 1<br> done :: set initial conditions<br> <br> call userchk<br> NOTE: All elements deformed , param(59) ^=0<br> done :: regenerate geometry data 1<br> <br> h1_mg_nx: 1 3 11<br> h1_mg_ny: 1 3 11<br> h1_mg_nz: 1 3 11<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 40651 unique labels shared<br> pairwise times (avg, min, max): 0.000203656 0.000173688 0.000223613<br> crystal router : 5.95078e-05 5.80072e-05 6.10113e-05<br> all reduce : 0.00126102 0.0012404 0.00127389<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 171689 112124 221476<br> buffer bytes (avg, min, max): 18153.2 12304 23232<br> setupds time 2.9112E-02 seconds 5 2 83038 80320<br> setvert3d: 4 1550072 2192632 1550072 1550072<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 400269 unique labels shared<br> pairwise times (avg, min, max): 0.000249024 0.000214601 0.000264192<br> crystal router : 0.000207164 0.000203013 0.000212216<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 1.41297e+06 878540 1853668<br> buffer bytes (avg, min, max): 151639 98096 195920<br> setupds time 5.8844E-02 seconds 6 4 1550072 80320<br> setvert3d: 4 1550072 2192632 1550072 1550072<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 400269 unique labels shared<br> pairwise times (avg, min, max): 0.000250614 0.000216007 0.000266099<br> crystal router : 0.000206963 0.000201392 0.000213099<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 1.41297e+06 878540 1853668<br> buffer bytes (avg, min, max): 151639 98096 195920<br> setupds time 5.8632E-02 seconds 7 4 1550072 80320<br> setvert3d: 6 4965906 10106386 4965906 4965906<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 1131335 unique labels shared<br> pairwise times (avg, min, max): 0.000345911 0.000313807 0.000366807<br> crystal router : 0.000918429 0.000901699 0.000948811<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 917140 806356 990804<br> buffer bytes (avg, min, max): 212147 117952 286736<br> setupds time 1.7421E-01 seconds 8 6 4965906 80320<br> setvert3d: 12 26906208 107226208 26906208 26906208<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 5553221 unique labels shared<br> pairwise times (avg, min, max): 0.00103432 0.000936699 0.00110071<br> crystal router : 0.00765613 0.0074913 0.0078582<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 4.35947e+06 3848020 4681812<br> buffer bytes (avg, min, max): 976972 546496 1320848<br> setupds time 1.9455E+00 seconds 9 12 26906208 80320<br> setvert3d: 14 38117242 176910202 38117242 38117242<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 7770079 unique labels shared<br> pairwise times (avg, min, max): 0.00144147 0.00128691 0.0015306<br> crystal router : 0.0118797 0.011621 0.0121873<br> used all_to_all method: pairwise<br> handle bytes (avg, min, max): 6.07641e+06 5366484 6520404<br> buffer bytes (avg, min, max): 1.35573e+06 758976 1832976<br> setupds time 2.9645E+00 seconds 10 14 38117242 80320<br> setup h1 coarse grid, nx_crs= 2<br> call usrsetvert<br> done :: usrsetvert<br><br>gs_setup: 40651 unique labels shared<br> pairwise times (avg, min, max): 0.000223077 0.000198388 0.000240588<br> crystal router : 5.7518e-05 5.61953e-05 5.86987e-05<br> all reduce : 0.00124078 0.0012202 0.00125399<br> used all_to_all method: crystal router<br> handle bytes (avg, min, max): 171689 112124 221476<br> buffer bytes (avg, min, max): 18153.2 12304 23232<br> done :: setup h1 coarse grid 22.2913181781769 sec<br> <br> call usrdat3<br> done :: usrdat3<br><br> set initial conditions<br> Checking restart options: pipe0.f00010 <br> nekuic (1) for ifld 1<br> Reading checkpoint data<br> 0 0 OPEN: pipe0.f00010<br> byte swap: F 6.543210 -2.9312772E+35<br><br> 0 2.8664E+02 done :: Read checkpoint data<br> avg data-throughput = 98.5MBps<br> io-nodes = 1<br><br> xyz min -12.000 -12.000 0.0000 <br> uvwpt min -0.99492 -0.76080 -0.82980 -1.4131 0.0000 <br> PS min 0.0000 0.0000 0.99000E+22<br> xyz max 12.000 12.000 36.000 <br> uvwpt max 0.81959 0.91894 1.4289 0.99220 0.0000 <br> PS max 0.0000 0.0000 -0.99000E+22<br> Restart: recompute geom. factors.<br> regenerate geometry data 1<br> NOTE: All elements deformed , param(59) ^=0<br> done :: regenerate geometry data 1<br> <br> done :: set initial conditions<br> <br> call userchk<br> force z: 1.000000000000000E-005<br> done :: userchk<br><br>gridpoints unique/tot: 6892160 8847360<br> dofs: 6737280 6892160<br> <br> Initial time: 0.2866404E+03<br> Initialization successfully completed 71.437 sec<br><br>Starting time loop ...<br><br> DT/DTCFL/DTFS/DTINIT 0.300E-03 0.000E+00 0.233-309 0.300E-03<br> force z: 1.000000000000000E-005<br> dump history points<br> reading <a href="http://hpts.in">hpts.in</a><br> found 80 points<br> initializing intpts(), tol= 1.000000000000000E-013<br> done :: dump history points<br> Building mesh for avg<br> Avg mesh builded<br> initializing intpts(), tol= 1.000000000000000E-008<br> call intpts<br> call findpts<br> total number of points = 1920000<br> failed = 0<br> done :: intpts<br> Writting interp data avg<br> Interp data avg written<br> call intpts<br> call findpts<br> total number of points = 1920000<br> failed = 0<br> done :: intpts<br> Writting interp data avg<br> Interp data avg written<br> call intpts<br> call findpts<br> total number of points = 1920000<br> failed = 0<br> done :: intpts<br> Writting interp data avg<br> Interp data avg written<br> Building the mesh for pod1<br> Mesh for pod1 builded<br> call intpts<br> call findpts<br> total number of points = 21504000<br> failed = 0<br> done :: intpts<br> Writting pod1 data<br> pod1 data written<br> Building the mesh for pod2<br> Mesh for pod2 builded<br> call intpts<br> call findpts<br> total number of points = 21504000<br> failed = 0<br> done :: intpts<br> Writting pod2 data<br> pod2 data written<br> done :: userchk<br><br>gridpoints unique/tot: 107226208 138792960<br> dofs: 106989839 106823343<br> <br> Initial time: 0.2866404E+03<br> Initialization successfully completed 4728.4 sec<br><br>Starting time loop ...<br><br> DT/DTCFL/DTFS/DTINIT 0.300E-03 0.000E+00 0.236-309 0.300E-03<br>Step 1, t= 2.8664073E+02, DT= 3.0000000E-04, C= 0.149 0.0000E+00 0.0000E+00<br>Step 1, t= 2.8664073E+02, DT= 3.0000000E-04, C= 0.438 0.0000E+00 0.0000E+00<br> Solving for fluid T T T<br>filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2375 0.9500<br>filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.7625 0.0500<br> 1.000000000000000E-008 p22 1 1<br> 1 1.00000E-06 3.08669E-03 5.12889E-03 6.01823E-01 1 Divergence<br> 2 1.00000E-06 1.63507E-03 5.12889E-03 3.18795E-01 1 Divergence<br> 3 1.00000E-06 5.70865E-04 5.12889E-03 1.11304E-01 1 Divergence<br> 4 1.00000E-06 3.19087E-04 5.12889E-03 6.22137E-02 1 Divergence<br> 5 1.00000E-06 1.67594E-04 5.12889E-03 3.26764E-02 1 Divergence<br> 6 1.00000E-06 8.78601E-05 5.12889E-03 1.71304E-02 1 Divergence<br> 7 1.00000E-06 4.10072E-05 5.12889E-03 7.99534E-03 1 Divergence<br> 8 1.00000E-06 2.09226E-05 5.12889E-03 4.07936E-03 1 Divergence<br> 9 1.00000E-06 1.17707E-05 5.12889E-03 2.29498E-03 1 Divergence<br> 10 1.00000E-06 6.32563E-06 5.12889E-03 1.23333E-03 1 Divergence<br> 11 1.00000E-06 3.53710E-06 5.12889E-03 6.89642E-04 1 Divergence<br> 12 1.00000E-06 1.96804E-06 5.12889E-03 3.83716E-04 1 Divergence<br> 13 1.00000E-06 1.14795E-06 5.12889E-03 2.23821E-04 1 Divergence<br> 14 1.00000E-06 7.12487E-07 5.12889E-03 1.38916E-04 1 Divergence<br> 1 PRES gmres: 14 7.1249E-07 1.0000E-06 5.1289E-03 1.8504E+00 4.1575E+00 F<br> 0 1.0000E-08 3.4398E-07 1.0000E-08 tol,matmod<br> 1 Helmholtz3/fluid: 7 0.1612E-08 0.1000E-07 0.1453E-01<br> L1/L2 DIV(V) : -3.3387E-15 4.5580E-04<br> L1/L2 QTL : 0.0000E+00 0.0000E+00<br> L1/L2 DIV(V)-QTL: -3.3387E-15 4.5580E-04<br> 1 2.8664E+02 6.3868E+00 Fluid done<br> Solving for fluid T T T<br>filt amp 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2375 0.9500<br>filt trn 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.7625 0.0500<br><br><br>> Can you please post your logfile otherwise it’s hard to tell what’s going wrong.
<br>> On 20 Sep 2017, at 12:20, "<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">nek5000-users at lists.mcs.anl.gov</a> <mailto:<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">nek5000-users at lists.mcs.anl.gov</a>> " <<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">nek5000-users at lists.mcs.anl.gov</a> <mailto:<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">nek5000-users at lists.mcs.anl.gov</a>> > wrote:
> Dear Nek devs
> Usually I start some calculation with 100-150 processors and it runs with acceptable speed. But sometimes the speed of calculation is extremely low, something like 1 step per 10 hours with 50 million mesh points. It seems like all computational nodes fall on one processors, while the rest just stay without any job, despite that all requirements about nodes per processor are satisfied.
> Have you ever seen something like that ? And how it can be fixed ?
> Regards, Vatslav</pre><br><br><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">С уважением, Владимир Рыженьков</div></div></div></div></div>
</div>