TACC: Starting up job 3495330 TACC: Starting parallel tasks... [WARNING]: option 'log_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_monitor' does not exists in the database -- ignoring. -------------------== CASL Options Database ==------------------- List of entered options: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -niter_hodge 2 -pc_cell hypre -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing no-arg ----------------------------------------------------------------- Loaded solver state from ... /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 Parameters : Re = 500., mu = 0.004, rho = 1., grid is 8x4x4 cfl = 1., uniform_band = 16. Total runtime ... 0 KSP Residual norm 6.592949864638e+03 1 KSP Residual norm 6.077174452328e+02 2 KSP Residual norm 2.405616003539e+02 3 KSP Residual norm 8.768650742116e+01 4 KSP Residual norm 1.440078737034e+01 5 KSP Residual norm 6.834525985175e+00 6 KSP Residual norm 3.473292760695e+00 7 KSP Residual norm 1.486112135106e+00 8 KSP Residual norm 5.428447405558e-01 9 KSP Residual norm 2.145605274010e-01 10 KSP Residual norm 9.739688050446e-02 11 KSP Residual norm 4.077318302427e-02 12 KSP Residual norm 1.823539603279e-02 13 KSP Residual norm 7.584584423331e-03 14 KSP Residual norm 3.672006005395e-03 15 KSP Residual norm 1.556069828952e-03 16 KSP Residual norm 6.743555302030e-04 17 KSP Residual norm 2.996828128497e-04 18 KSP Residual norm 1.318635314398e-04 19 KSP Residual norm 5.655860818003e-05 20 KSP Residual norm 2.380349218968e-05 21 KSP Residual norm 1.010459099046e-05 22 KSP Residual norm 4.768829046563e-06 23 KSP Residual norm 1.995962983562e-06 24 KSP Residual norm 8.881469618326e-07 25 KSP Residual norm 3.822683385529e-07 26 KSP Residual norm 1.689474823574e-07 27 KSP Residual norm 7.562918899460e-08 28 KSP Residual norm 3.391397573292e-08 29 KSP Residual norm 1.562247921850e-08 30 KSP Residual norm 7.190790561650e-09 31 KSP Residual norm 3.078071241641e-09 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.387149061428e+02 1 KSP Residual norm 1.605979242342e+02 2 KSP Residual norm 5.281674743603e+01 3 KSP Residual norm 1.288659245377e+01 4 KSP Residual norm 4.244389193624e+00 5 KSP Residual norm 2.159260427143e+00 6 KSP Residual norm 9.553151188528e-01 7 KSP Residual norm 3.649294324930e-01 8 KSP Residual norm 1.536661640346e-01 9 KSP Residual norm 6.994704643813e-02 10 KSP Residual norm 3.013482892035e-02 11 KSP Residual norm 1.269208158177e-02 12 KSP Residual norm 5.749854728637e-03 13 KSP Residual norm 2.535438499129e-03 14 KSP Residual norm 1.158028026346e-03 15 KSP Residual norm 5.013520262007e-04 16 KSP Residual norm 2.353071672310e-04 17 KSP Residual norm 1.036933854452e-04 18 KSP Residual norm 4.648394597259e-05 19 KSP Residual norm 1.988761660118e-05 20 KSP Residual norm 8.557801408821e-06 21 KSP Residual norm 3.863042761898e-06 22 KSP Residual norm 1.735601215170e-06 23 KSP Residual norm 7.383842561311e-07 24 KSP Residual norm 3.398468767297e-07 25 KSP Residual norm 1.462667849594e-07 26 KSP Residual norm 6.577327824832e-08 27 KSP Residual norm 3.018452926849e-08 28 KSP Residual norm 1.345177748748e-08 29 KSP Residual norm 6.123516629738e-09 30 KSP Residual norm 2.790514355026e-09 31 KSP Residual norm 1.250998574752e-09 32 KSP Residual norm 5.437618693294e-10 33 KSP Residual norm 2.225676626218e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.425891483638e+02 1 KSP Residual norm 1.614535599579e+02 2 KSP Residual norm 5.275532353642e+01 3 KSP Residual norm 1.278441931036e+01 4 KSP Residual norm 4.224924193344e+00 5 KSP Residual norm 2.144666745661e+00 6 KSP Residual norm 9.487923144800e-01 7 KSP Residual norm 3.625932703878e-01 8 KSP Residual norm 1.522155238484e-01 9 KSP Residual norm 6.965767745242e-02 10 KSP Residual norm 3.010269844474e-02 11 KSP Residual norm 1.267407418092e-02 12 KSP Residual norm 5.766326090747e-03 13 KSP Residual norm 2.543300112669e-03 14 KSP Residual norm 1.162246239815e-03 15 KSP Residual norm 5.030334597125e-04 16 KSP Residual norm 2.359838740660e-04 17 KSP Residual norm 1.035899606919e-04 18 KSP Residual norm 4.606417675038e-05 19 KSP Residual norm 1.966470526481e-05 20 KSP Residual norm 8.421887320804e-06 21 KSP Residual norm 3.767677688281e-06 22 KSP Residual norm 1.694147364273e-06 23 KSP Residual norm 7.097594471949e-07 24 KSP Residual norm 3.203833647317e-07 25 KSP Residual norm 1.370118257609e-07 26 KSP Residual norm 6.073428302082e-08 27 KSP Residual norm 2.774070890764e-08 28 KSP Residual norm 1.233961566437e-08 29 KSP Residual norm 5.587358608612e-09 30 KSP Residual norm 2.581200924537e-09 31 KSP Residual norm 1.152778573106e-09 32 KSP Residual norm 5.259830259368e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 3.009927585321e+00 1 KSP Residual norm 1.579796158546e+00 2 KSP Residual norm 4.486245814129e-01 3 KSP Residual norm 1.661871660752e-01 4 KSP Residual norm 6.015447864242e-02 5 KSP Residual norm 1.269648610048e-02 6 KSP Residual norm 2.927807148725e-03 7 KSP Residual norm 7.704052773553e-04 8 KSP Residual norm 1.761664001259e-04 9 KSP Residual norm 4.480036879115e-05 10 KSP Residual norm 1.226095013173e-05 11 KSP Residual norm 3.234442148814e-06 12 KSP Residual norm 8.982837671212e-07 13 KSP Residual norm 2.601913686761e-07 14 KSP Residual norm 7.177650272468e-08 15 KSP Residual norm 1.880721914721e-08 16 KSP Residual norm 5.355145277610e-09 17 KSP Residual norm 1.411879949537e-09 18 KSP Residual norm 3.489598244873e-10 19 KSP Residual norm 8.873418171088e-11 20 KSP Residual norm 2.332246672296e-11 21 KSP Residual norm 6.555152105871e-12 22 KSP Residual norm 1.765455963599e-12 KSP Object: 4096 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 3.082825e-02 0 KSP Residual norm 1.180380389902e+02 1 KSP Residual norm 8.881999817606e-04 2 KSP Residual norm 2.785936810041e-04 3 KSP Residual norm 1.013793781382e-04 4 KSP Residual norm 4.146937520026e-05 5 KSP Residual norm 1.765958876362e-05 6 KSP Residual norm 7.334439179331e-06 7 KSP Residual norm 3.222486571227e-06 8 KSP Residual norm 1.348673094290e-06 9 KSP Residual norm 5.716432789174e-07 10 KSP Residual norm 2.558316285769e-07 11 KSP Residual norm 1.087101309152e-07 12 KSP Residual norm 4.836727455196e-08 13 KSP Residual norm 2.131492105693e-08 14 KSP Residual norm 9.109330361671e-09 15 KSP Residual norm 4.082647056270e-09 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.563065364364e+01 1 KSP Residual norm 1.374098793977e-03 2 KSP Residual norm 4.277469184084e-04 3 KSP Residual norm 1.546861749951e-04 4 KSP Residual norm 6.270902527740e-05 5 KSP Residual norm 2.659385079026e-05 6 KSP Residual norm 1.103141545262e-05 7 KSP Residual norm 4.851987154084e-06 8 KSP Residual norm 2.035630327731e-06 9 KSP Residual norm 8.625676236472e-07 10 KSP Residual norm 3.864615676195e-07 11 KSP Residual norm 1.623826495107e-07 12 KSP Residual norm 7.095293820414e-08 13 KSP Residual norm 3.051204855288e-08 14 KSP Residual norm 1.291763618928e-08 15 KSP Residual norm 5.839432624085e-09 16 KSP Residual norm 2.588374140946e-09 17 KSP Residual norm 1.169445918444e-09 18 KSP Residual norm 5.096887335070e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.565504375772e+01 1 KSP Residual norm 1.369986592338e-03 2 KSP Residual norm 4.276936920366e-04 3 KSP Residual norm 1.550163326355e-04 4 KSP Residual norm 6.314147345819e-05 5 KSP Residual norm 2.691667901418e-05 6 KSP Residual norm 1.123363142942e-05 7 KSP Residual norm 4.931503532496e-06 8 KSP Residual norm 2.029123051705e-06 9 KSP Residual norm 8.531070722026e-07 10 KSP Residual norm 3.823443822628e-07 11 KSP Residual norm 1.633455668922e-07 12 KSP Residual norm 7.291225653552e-08 13 KSP Residual norm 3.230035583595e-08 14 KSP Residual norm 1.395656012610e-08 15 KSP Residual norm 6.254056021963e-09 16 KSP Residual norm 2.661891448879e-09 17 KSP Residual norm 1.169391320904e-09 18 KSP Residual norm 5.120820563601e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 2.037653920145e+00 1 KSP Residual norm 1.194983831848e-04 2 KSP Residual norm 1.848003257110e-05 3 KSP Residual norm 4.626296675052e-06 4 KSP Residual norm 1.161710542212e-06 5 KSP Residual norm 3.172107274010e-07 6 KSP Residual norm 8.511670035672e-08 7 KSP Residual norm 2.519883644433e-08 8 KSP Residual norm 9.161526799432e-09 9 KSP Residual norm 3.687905265370e-09 10 KSP Residual norm 1.536740058911e-09 11 KSP Residual norm 5.037957618378e-10 12 KSP Residual norm 9.975769184669e-11 13 KSP Residual norm 2.004296653888e-11 14 KSP Residual norm 4.886132176380e-12 15 KSP Residual norm 1.337631546129e-12 KSP Object: 4096 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 8.017553e-07 Iteration #0000 : tn = 1.50161e+02, percent done : 0.0%, max_L2_norm_u = 1.16986e+00, number of leaves = 270588873 0 KSP Residual norm 7.213286005597e+03 1 KSP Residual norm 6.007008592234e+02 2 KSP Residual norm 8.595488943293e+01 3 KSP Residual norm 8.389386924437e+00 4 KSP Residual norm 2.438633928863e+00 5 KSP Residual norm 4.778355670223e-01 6 KSP Residual norm 1.027421855409e-01 7 KSP Residual norm 2.331136128401e-02 8 KSP Residual norm 6.011411368573e-03 9 KSP Residual norm 1.527212826226e-03 10 KSP Residual norm 3.866572829861e-04 11 KSP Residual norm 9.557179844707e-05 12 KSP Residual norm 2.428498147678e-05 13 KSP Residual norm 6.034897427104e-06 14 KSP Residual norm 1.551545078604e-06 15 KSP Residual norm 3.560157820200e-07 16 KSP Residual norm 8.307743518693e-08 17 KSP Residual norm 1.669120940229e-08 18 KSP Residual norm 3.833055115389e-09 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.451139431020e+02 1 KSP Residual norm 1.565398599606e+02 2 KSP Residual norm 1.466695352853e+01 3 KSP Residual norm 2.519150037253e+00 4 KSP Residual norm 5.649495559749e-01 5 KSP Residual norm 1.191637078600e-01 6 KSP Residual norm 2.915969017720e-02 7 KSP Residual norm 7.008548833941e-03 8 KSP Residual norm 1.785909863085e-03 9 KSP Residual norm 4.461772496176e-04 10 KSP Residual norm 9.985734700950e-05 11 KSP Residual norm 2.019761923863e-05 12 KSP Residual norm 4.137139751998e-06 13 KSP Residual norm 9.360976378687e-07 14 KSP Residual norm 2.328570659674e-07 15 KSP Residual norm 5.769334983487e-08 16 KSP Residual norm 1.534676523004e-08 17 KSP Residual norm 3.499994361542e-09 18 KSP Residual norm 8.675793615275e-10 19 KSP Residual norm 2.124014863637e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.470406394139e+02 1 KSP Residual norm 1.573697168646e+02 2 KSP Residual norm 1.476956590670e+01 3 KSP Residual norm 2.529450488725e+00 4 KSP Residual norm 5.645261849453e-01 5 KSP Residual norm 1.188609903430e-01 6 KSP Residual norm 2.936180977827e-02 7 KSP Residual norm 7.077147461163e-03 8 KSP Residual norm 1.803168887869e-03 9 KSP Residual norm 4.491042408433e-04 10 KSP Residual norm 9.827861616203e-05 11 KSP Residual norm 1.937721123362e-05 12 KSP Residual norm 3.862957173082e-06 13 KSP Residual norm 8.639889445625e-07 14 KSP Residual norm 2.152566889425e-07 15 KSP Residual norm 5.303714744609e-08 16 KSP Residual norm 1.406984337739e-08 17 KSP Residual norm 3.207544885398e-09 18 KSP Residual norm 7.940725950183e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 1.112437777306e+00 1 KSP Residual norm 6.755019585584e-01 2 KSP Residual norm 1.662037299681e-01 3 KSP Residual norm 5.475713029782e-02 4 KSP Residual norm 1.962690288761e-02 5 KSP Residual norm 4.845537551866e-03 6 KSP Residual norm 1.015241008351e-03 7 KSP Residual norm 2.369711418157e-04 8 KSP Residual norm 4.923209557708e-05 9 KSP Residual norm 1.259747425485e-05 10 KSP Residual norm 3.432644931652e-06 11 KSP Residual norm 8.701010197930e-07 12 KSP Residual norm 2.124486000729e-07 13 KSP Residual norm 5.715144084897e-08 14 KSP Residual norm 1.703702638445e-08 15 KSP Residual norm 5.126167763130e-09 16 KSP Residual norm 1.426698239644e-09 17 KSP Residual norm 3.531571529103e-10 18 KSP Residual norm 8.810219808508e-11 19 KSP Residual norm 2.237309836438e-11 20 KSP Residual norm 6.058298746725e-12 21 KSP Residual norm 1.652343078270e-12 22 KSP Residual norm 4.247596381077e-13 KSP Object: 4096 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 9.365452e-03 0 KSP Residual norm 1.198339738026e+02 1 KSP Residual norm 4.767058449685e-02 2 KSP Residual norm 7.730133259547e-03 3 KSP Residual norm 1.544819762950e-03 4 KSP Residual norm 3.723985891862e-04 5 KSP Residual norm 8.848003960739e-05 6 KSP Residual norm 2.234085726499e-05 7 KSP Residual norm 5.362764886062e-06 8 KSP Residual norm 1.205960918567e-06 9 KSP Residual norm 3.232509572888e-07 10 KSP Residual norm 8.247129336316e-08 11 KSP Residual norm 2.158235184170e-08 12 KSP Residual norm 5.750091621579e-09 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.764873125797e+01 1 KSP Residual norm 3.360474119783e-02 2 KSP Residual norm 5.295403674844e-03 3 KSP Residual norm 1.026251741898e-03 4 KSP Residual norm 2.413567343612e-04 5 KSP Residual norm 5.754707894613e-05 6 KSP Residual norm 1.414639623043e-05 7 KSP Residual norm 3.467414996212e-06 8 KSP Residual norm 7.736430056773e-07 9 KSP Residual norm 2.057824059519e-07 10 KSP Residual norm 5.376814049839e-08 11 KSP Residual norm 1.383429561062e-08 12 KSP Residual norm 3.699086516205e-09 13 KSP Residual norm 9.222536905125e-10 14 KSP Residual norm 2.402154902327e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.767287705475e+01 1 KSP Residual norm 3.377396246952e-02 2 KSP Residual norm 5.367404289751e-03 3 KSP Residual norm 1.046468326339e-03 4 KSP Residual norm 2.454470632962e-04 5 KSP Residual norm 5.765463444419e-05 6 KSP Residual norm 1.407435572590e-05 7 KSP Residual norm 3.440894105050e-06 8 KSP Residual norm 7.737475261228e-07 9 KSP Residual norm 2.060610200954e-07 10 KSP Residual norm 5.490724636180e-08 11 KSP Residual norm 1.430213917903e-08 12 KSP Residual norm 3.814811356407e-09 13 KSP Residual norm 9.411838959317e-10 14 KSP Residual norm 2.442138522498e-10 KSP Object: 4096 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 7.300842713789e-01 1 KSP Residual norm 7.675867944357e-03 2 KSP Residual norm 5.876582030692e-04 3 KSP Residual norm 2.055049532282e-04 4 KSP Residual norm 5.767090293502e-05 5 KSP Residual norm 1.726417496769e-05 6 KSP Residual norm 5.607131390463e-06 7 KSP Residual norm 1.744830259259e-06 8 KSP Residual norm 4.312218188109e-07 9 KSP Residual norm 1.056474398423e-07 10 KSP Residual norm 2.482746535376e-08 11 KSP Residual norm 7.159455914763e-09 12 KSP Residual norm 2.112888136733e-09 13 KSP Residual norm 5.479029236965e-10 14 KSP Residual norm 1.375329947795e-10 15 KSP Residual norm 3.609355263251e-11 16 KSP Residual norm 9.184581435562e-12 17 KSP Residual norm 2.097393021043e-12 18 KSP Residual norm 4.895669170922e-13 KSP Object: 4096 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4096 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 4096 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 6.935734e-06 Iteration #0001 : tn = 1.50163e+02, percent done : 0.0%, max_L2_norm_u = 1.16966e+00, number of leaves = 270631545 Mean computational time spent on viscosity step: 2.01154e+02 projection step: 1.92735e+02 computing velocities at nodes: 5.54243e+01 grid update: 8.02290e+01 full iteration (total): 5.39459e+02 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /work/04965/tg842642/stampede2/build/flow_past_sphere/test_petsc/flow_past_sphere on a skylake named c452-092.stampede2.tacc.utexas.edu with 4096 processors, by tg842642 Sun May 5 23:46:11 2019 Using Petsc Release Version 3.11.0, Mar, 29, 2019 Max Max/Min Avg Total Time (sec): 1.499e+03 1.000 1.499e+03 Objects: 1.343e+03 1.000 1.343e+03 Flop: 1.505e+09 2.752 1.198e+09 4.908e+12 Flop/sec: 1.004e+06 2.752 7.995e+05 3.275e+09 MPI Messages: 1.860e+05 18.387 4.318e+04 1.769e+08 MPI Message Lengths: 6.962e+08 4.755 8.811e+03 1.558e+12 MPI Reductions: 2.621e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 1.4987e+03 100.0% 4.9078e+12 100.0% 1.769e+08 100.0% 8.811e+03 100.0% 2.614e+03 99.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 21 1.0 7.3064e+01 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 VecDot 484 1.0 2.2925e+00 6.8 6.77e+07 1.1 0.0e+00 0.0e+00 4.8e+02 0 5 0 0 18 0 5 0 0 19 115294 VecDotNorm2 242 1.0 2.1908e+0011.2 6.77e+07 1.1 0.0e+00 0.0e+00 2.4e+02 0 5 0 0 9 0 5 0 0 9 120649 VecTDot 154 1.0 3.3815e-01 3.3 2.03e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 2 0 0 6 0 2 0 0 6 246485 VecNorm 343 1.0 3.2572e+00 2.6 4.73e+07 1.0 0.0e+00 0.0e+00 3.4e+02 0 4 0 0 13 0 4 0 0 13 57384 VecCopy 394 1.0 8.9514e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 136 1.0 6.1525e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 160 1.0 5.0737e-02 1.5 2.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 1707334 VecAYPX 75 1.0 1.1171e-01 7.6 9.78e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 358507 VecAXPBYCZ 484 1.0 1.7069e-01 1.2 1.35e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 0 11 0 0 0 3097058 VecWAXPY 484 1.0 1.6492e-01 1.1 6.77e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1602669 VecAssemblyBegin 11 1.0 5.6778e+0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecAssemblyEnd 11 1.0 4.6575e-0336.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecLoad 11 1.0 1.4023e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.1e+01 9 0 0 0 1 9 0 0 0 1 0 VecScatterBegin 1501 1.0 1.6540e+00 7.3 0.00e+00 0.0 1.3e+08 9.7e+03 0.0e+00 0 0 75 83 0 0 0 75 83 0 0 VecScatterEnd 1501 1.0 1.6084e+02 9.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 MatMult 569 1.0 2.0195e+0111.4 5.58e+0810.7 2.8e+07 6.4e+03 0.0e+00 0 33 16 11 0 0 33 16 11 0 81367 MatSOR 502 1.0 3.9757e+00 4.1 5.12e+08 6.5 0.0e+00 0.0e+00 0.0e+00 0 32 0 0 0 0 32 0 0 0 388874 MatConvert 2 1.0 4.8984e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 10 1.0 1.7193e+0112.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 10 1.0 6.0680e-01 1.0 0.00e+00 0.0 7.6e+05 1.6e+03 7.2e+01 0 0 0 0 3 0 0 0 0 3 0 MatGetRowIJ 4 1.0 3.0874e-021199.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 16 1.0 4.3585e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.6e+01 0 0 0 0 1 0 0 0 0 1 0 KSPSetUp 16 1.0 6.2274e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 16 1.0 3.1430e+02 1.1 1.51e+09 2.8 2.8e+07 6.4e+03 1.2e+03 20100 16 11 47 20100 16 11 47 15615 PCSetUp 8 1.0 1.0253e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 7 0 0 0 0 7 0 0 0 0 0 PCApply 585 1.0 2.0523e+02 1.1 5.12e+08 6.5 0.0e+00 0.0e+00 0.0e+00 13 32 0 0 0 13 32 0 0 0 7533 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 601 601 193306560 0. Index Set 356 356 13316584 0. IS L to G Mapping 159 159 52101108 0. Vec Scatter 178 178 6156432 0. Viewer 7 6 5040 0. Krylov Solver 8 8 11392 0. Matrix 26 26 75720872 0. Preconditioner 8 8 8528 0. ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 8.00133e-05 Average time for zero size MPI_Send(): 8.29081e-06 #PETSc Option Table entries: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -ksp_monitor -ksp_view -log_view -niter_hodge 2 -pc_cell hypre -pc_hypre_boomeramg_coarsen_type Falgout -pc_hypre_boomeramg_strong_threshold 0.5 -pc_hypre_boomeramg_truncfactor 0.1 -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing -vecload_block_size 1 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-x=0 -with-pic --with-make-np=8 --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" --with-packages-build-dir=/tmp/petsc-3.11/skylake --with-mpi=1 --with-cc=/opt/apps/intel18/impi/18.0.2/bin/mpicc --with-cxx=/opt/apps/intel18/impi/18.0.2/bin/mpicxx --with-fc=/opt/apps/intel18/impi/18.0.2/bin/mpif90 --with-mpi-include=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt/libmpi.so --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-plapack=1 --download-plapack --with-spai=1 --download-spai --with-sundials=1 --download-sundials --with-elemental=1 --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 --download-hdf5=1 --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 --with-debugging=no --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" LIBS= ----------------------------------------- Libraries compiled on 2019-04-02 03:10:34 on build.stampede2.tacc.utexas.edu Machine characteristics: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /home1/apps/intel18/impi18_0/petsc/3.11 Using PETSc arch: skylake ----------------------------------------- Using C compiler: /opt/apps/intel18/impi/18.0.2/bin/mpicc -fPIC -wd1572 -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g Using Fortran compiler: /opt/apps/intel18/impi/18.0.2/bin/mpif90 -fPIC -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g ----------------------------------------- Using include paths: -I/home1/apps/intel18/impi18_0/petsc/3.11/include -I/home1/apps/intel18/impi18_0/petsc/3.11/skylake/include -I/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include ----------------------------------------- Using C linker: /opt/apps/intel18/impi/18.0.2/bin/mpicc Using Fortran linker: /opt/apps/intel18/impi/18.0.2/bin/mpif90 Using libraries: -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -lpetsc -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -L/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib64 -L/opt/apps/gcc/6.3.0/lib64 -Wl,-rpath,/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -L/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -Wl,-rpath,/opt/apps/gcc/6.3.0/lib -L/opt/apps/gcc/6.3.0/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr -lspai -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lzoltan -lparmetis -lmetis -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lmpi -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lrt -lm -lpthread -lz -lstdc++ -ldl ----------------------------------------- TACC: Shutdown complete. Exiting.