TACC: Starting up job 3498378 TACC: Starting parallel tasks... [WARNING]: option 'log_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_monitor' does not exists in the database -- ignoring. -------------------== CASL Options Database ==------------------- List of entered options: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -niter_hodge 2 -pc_cell hypre -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing no-arg ----------------------------------------------------------------- Loaded solver state from ... /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 Parameters : Re = 500., mu = 0.004, rho = 1., grid is 8x4x4 cfl = 1., uniform_band = 16. Total runtime ... 0 KSP Residual norm 6.567660977440e+03 1 KSP Residual norm 6.037961362907e+02 2 KSP Residual norm 2.423996351470e+02 3 KSP Residual norm 9.489159293630e+01 4 KSP Residual norm 1.757817188036e+01 5 KSP Residual norm 7.731153114774e+00 6 KSP Residual norm 3.987528530507e+00 7 KSP Residual norm 1.772139806282e+00 8 KSP Residual norm 7.136707695383e-01 9 KSP Residual norm 2.838767905105e-01 10 KSP Residual norm 1.331595860905e-01 11 KSP Residual norm 6.154605951093e-02 12 KSP Residual norm 2.649005908686e-02 13 KSP Residual norm 1.089790220446e-02 14 KSP Residual norm 5.080649523815e-03 15 KSP Residual norm 2.209005055905e-03 16 KSP Residual norm 9.935305803306e-04 17 KSP Residual norm 4.383311085297e-04 18 KSP Residual norm 2.046801518836e-04 19 KSP Residual norm 9.422152982683e-05 20 KSP Residual norm 4.109674266118e-05 21 KSP Residual norm 1.798556106103e-05 22 KSP Residual norm 8.728596410226e-06 23 KSP Residual norm 3.781218913390e-06 24 KSP Residual norm 1.753603717992e-06 25 KSP Residual norm 7.770057555967e-07 26 KSP Residual norm 3.668548059986e-07 27 KSP Residual norm 1.705868243727e-07 28 KSP Residual norm 7.608570707001e-08 29 KSP Residual norm 3.478414385386e-08 30 KSP Residual norm 1.525767980063e-08 31 KSP Residual norm 6.854819962894e-09 32 KSP Residual norm 2.765932588957e-09 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.285212131448e+02 1 KSP Residual norm 1.581534794462e+02 2 KSP Residual norm 5.457209613713e+01 3 KSP Residual norm 1.469398208671e+01 4 KSP Residual norm 4.784640966258e+00 5 KSP Residual norm 2.469425062983e+00 6 KSP Residual norm 1.107858538765e+00 7 KSP Residual norm 4.328133358845e-01 8 KSP Residual norm 1.911537675151e-01 9 KSP Residual norm 8.889277604220e-02 10 KSP Residual norm 4.123320214995e-02 11 KSP Residual norm 1.749963925260e-02 12 KSP Residual norm 7.424300611287e-03 13 KSP Residual norm 3.269338368987e-03 14 KSP Residual norm 1.454536431012e-03 15 KSP Residual norm 6.272659357119e-04 16 KSP Residual norm 2.912396596179e-04 17 KSP Residual norm 1.299010648955e-04 18 KSP Residual norm 5.985559541031e-05 19 KSP Residual norm 2.584525239328e-05 20 KSP Residual norm 1.154092089119e-05 21 KSP Residual norm 5.246091600640e-06 22 KSP Residual norm 2.343284429403e-06 23 KSP Residual norm 1.030494226951e-06 24 KSP Residual norm 4.839538397960e-07 25 KSP Residual norm 2.182922364521e-07 26 KSP Residual norm 1.030153338789e-07 27 KSP Residual norm 4.709416641919e-08 28 KSP Residual norm 2.165948951883e-08 29 KSP Residual norm 9.935072491262e-09 30 KSP Residual norm 4.540574441975e-09 31 KSP Residual norm 2.080754388143e-09 32 KSP Residual norm 9.272402699965e-10 33 KSP Residual norm 4.097034489206e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.325318965282e+02 1 KSP Residual norm 1.589966905845e+02 2 KSP Residual norm 5.452420533314e+01 3 KSP Residual norm 1.456143520149e+01 4 KSP Residual norm 4.748225454081e+00 5 KSP Residual norm 2.452126197283e+00 6 KSP Residual norm 1.101671108111e+00 7 KSP Residual norm 4.298755328387e-01 8 KSP Residual norm 1.892412783554e-01 9 KSP Residual norm 8.829961775263e-02 10 KSP Residual norm 4.081874094161e-02 11 KSP Residual norm 1.728299121026e-02 12 KSP Residual norm 7.320686528313e-03 13 KSP Residual norm 3.216567540299e-03 14 KSP Residual norm 1.430032445705e-03 15 KSP Residual norm 6.154744040719e-04 16 KSP Residual norm 2.859100258453e-04 17 KSP Residual norm 1.279090903892e-04 18 KSP Residual norm 5.913126909817e-05 19 KSP Residual norm 2.571230176419e-05 20 KSP Residual norm 1.158018087990e-05 21 KSP Residual norm 5.288071621770e-06 22 KSP Residual norm 2.371373349908e-06 23 KSP Residual norm 1.047048530703e-06 24 KSP Residual norm 4.939386034602e-07 25 KSP Residual norm 2.235802246969e-07 26 KSP Residual norm 1.056121136001e-07 27 KSP Residual norm 4.857930758223e-08 28 KSP Residual norm 2.241977299804e-08 29 KSP Residual norm 1.032965336475e-08 30 KSP Residual norm 4.695797745260e-09 31 KSP Residual norm 2.143102060786e-09 32 KSP Residual norm 9.446768197572e-10 33 KSP Residual norm 4.133222683215e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 2.986152427277e+00 1 KSP Residual norm 1.574522744807e+00 2 KSP Residual norm 4.193022228994e-01 3 KSP Residual norm 1.558903130978e-01 4 KSP Residual norm 5.646632113583e-02 5 KSP Residual norm 1.354277198519e-02 6 KSP Residual norm 3.321175943111e-03 7 KSP Residual norm 8.296602408941e-04 8 KSP Residual norm 1.745304141750e-04 9 KSP Residual norm 4.768064859172e-05 10 KSP Residual norm 1.406916168350e-05 11 KSP Residual norm 4.207082968192e-06 12 KSP Residual norm 1.309660626662e-06 13 KSP Residual norm 4.653616613640e-07 14 KSP Residual norm 1.442455614107e-07 15 KSP Residual norm 3.900141820237e-08 16 KSP Residual norm 1.058751395254e-08 17 KSP Residual norm 2.588510884987e-09 18 KSP Residual norm 7.015519241784e-10 19 KSP Residual norm 2.139411012079e-10 20 KSP Residual norm 6.363403720049e-11 21 KSP Residual norm 1.780504858097e-11 22 KSP Residual norm 4.523609643226e-12 23 KSP Residual norm 1.141570236229e-12 KSP Object: 16384 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 3.082825e-02 0 KSP Residual norm 1.180380389847e+02 1 KSP Residual norm 9.125699899409e-04 2 KSP Residual norm 3.024769488172e-04 3 KSP Residual norm 1.159102507085e-04 4 KSP Residual norm 4.991511386658e-05 5 KSP Residual norm 2.126740918891e-05 6 KSP Residual norm 9.317038880685e-06 7 KSP Residual norm 4.183649027948e-06 8 KSP Residual norm 1.784630826716e-06 9 KSP Residual norm 8.046422388277e-07 10 KSP Residual norm 3.639889764647e-07 11 KSP Residual norm 1.323287021205e-07 12 KSP Residual norm 5.894998264876e-08 13 KSP Residual norm 1.326423195911e-08 14 KSP Residual norm 4.671106596705e-09 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.563065362024e+01 1 KSP Residual norm 1.406098298042e-03 2 KSP Residual norm 4.617168252354e-04 3 KSP Residual norm 1.756999035566e-04 4 KSP Residual norm 7.537200550227e-05 5 KSP Residual norm 3.235320690914e-05 6 KSP Residual norm 1.421486894700e-05 7 KSP Residual norm 6.442617885597e-06 8 KSP Residual norm 2.760671693861e-06 9 KSP Residual norm 1.272134312927e-06 10 KSP Residual norm 5.771808774744e-07 11 KSP Residual norm 2.539127024598e-07 12 KSP Residual norm 1.130271574688e-07 13 KSP Residual norm 4.674099709282e-08 14 KSP Residual norm 2.024289909177e-08 15 KSP Residual norm 8.647063111541e-09 16 KSP Residual norm 3.816149841530e-09 17 KSP Residual norm 1.745234046468e-09 18 KSP Residual norm 7.576016523709e-10 19 KSP Residual norm 2.792216854711e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.565504373596e+01 1 KSP Residual norm 1.400069077261e-03 2 KSP Residual norm 4.596838970682e-04 3 KSP Residual norm 1.749231838756e-04 4 KSP Residual norm 7.519511155846e-05 5 KSP Residual norm 3.237393316363e-05 6 KSP Residual norm 1.424745490365e-05 7 KSP Residual norm 6.460513291513e-06 8 KSP Residual norm 2.761194471498e-06 9 KSP Residual norm 1.265466794662e-06 10 KSP Residual norm 5.743817051079e-07 11 KSP Residual norm 2.536603692547e-07 12 KSP Residual norm 1.146602320423e-07 13 KSP Residual norm 4.830742274025e-08 14 KSP Residual norm 2.110662697016e-08 15 KSP Residual norm 8.753833964519e-09 16 KSP Residual norm 3.713754084393e-09 17 KSP Residual norm 1.728565185410e-09 18 KSP Residual norm 7.760272714513e-10 19 KSP Residual norm 3.487406701303e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 2.037653919160e+00 1 KSP Residual norm 1.203272874342e-04 2 KSP Residual norm 1.811441553099e-05 3 KSP Residual norm 4.767057704953e-06 4 KSP Residual norm 1.127905496271e-06 5 KSP Residual norm 2.931954660342e-07 6 KSP Residual norm 8.786323723995e-08 7 KSP Residual norm 3.085702418835e-08 8 KSP Residual norm 1.037608997316e-08 9 KSP Residual norm 3.089705260072e-09 10 KSP Residual norm 1.088955923994e-09 11 KSP Residual norm 4.210650301025e-10 12 KSP Residual norm 1.426615105486e-10 13 KSP Residual norm 4.854042884235e-11 14 KSP Residual norm 1.541950053873e-11 15 KSP Residual norm 4.305599687393e-12 16 KSP Residual norm 9.784571337953e-13 KSP Object: 16384 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 8.017552e-07 Iteration #0000 : tn = 1.50161e+02, percent done : 0.0%, max_L2_norm_u = 1.16986e+00, number of leaves = 270588873 0 KSP Residual norm 7.178496856949e+03 1 KSP Residual norm 6.058511195151e+02 2 KSP Residual norm 9.556746527770e+01 3 KSP Residual norm 1.031127398748e+01 4 KSP Residual norm 2.908020310847e+00 5 KSP Residual norm 6.630873715453e-01 6 KSP Residual norm 1.400792864938e-01 7 KSP Residual norm 3.037833038103e-02 8 KSP Residual norm 8.503475986084e-03 9 KSP Residual norm 2.088465116341e-03 10 KSP Residual norm 5.724480288053e-04 11 KSP Residual norm 1.550722740173e-04 12 KSP Residual norm 3.680458272384e-05 13 KSP Residual norm 9.416442230231e-06 14 KSP Residual norm 2.320954578740e-06 15 KSP Residual norm 5.305731052525e-07 16 KSP Residual norm 1.015587844683e-07 17 KSP Residual norm 1.803702608221e-08 18 KSP Residual norm 3.618044400731e-09 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.295417039962e+02 1 KSP Residual norm 1.586044118257e+02 2 KSP Residual norm 1.688960537164e+01 3 KSP Residual norm 3.080045335949e+00 4 KSP Residual norm 7.683414699032e-01 5 KSP Residual norm 1.723313971454e-01 6 KSP Residual norm 3.980008575780e-02 7 KSP Residual norm 1.043340792931e-02 8 KSP Residual norm 2.810495143756e-03 9 KSP Residual norm 7.151808217572e-04 10 KSP Residual norm 1.952912826248e-04 11 KSP Residual norm 4.485994480630e-05 12 KSP Residual norm 1.027872431567e-05 13 KSP Residual norm 2.506192969402e-06 14 KSP Residual norm 6.054161500592e-07 15 KSP Residual norm 1.493424870680e-07 16 KSP Residual norm 3.806178027100e-08 17 KSP Residual norm 9.481016080711e-09 18 KSP Residual norm 2.542628759787e-09 19 KSP Residual norm 6.605231116162e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.314451839524e+02 1 KSP Residual norm 1.594481305587e+02 2 KSP Residual norm 1.700914823587e+01 3 KSP Residual norm 3.090765981681e+00 4 KSP Residual norm 7.679507248976e-01 5 KSP Residual norm 1.723262851436e-01 6 KSP Residual norm 3.974522512048e-02 7 KSP Residual norm 1.033700677178e-02 8 KSP Residual norm 2.779973381143e-03 9 KSP Residual norm 7.077684537076e-04 10 KSP Residual norm 1.925073670015e-04 11 KSP Residual norm 4.394971358313e-05 12 KSP Residual norm 9.914509466392e-06 13 KSP Residual norm 2.404084845107e-06 14 KSP Residual norm 5.792936408872e-07 15 KSP Residual norm 1.429447609410e-07 16 KSP Residual norm 3.654871613657e-08 17 KSP Residual norm 9.112360380715e-09 18 KSP Residual norm 2.453603306992e-09 19 KSP Residual norm 6.407180779656e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 1.077218675627e+00 1 KSP Residual norm 6.620479628901e-01 2 KSP Residual norm 1.746628502428e-01 3 KSP Residual norm 6.277576454190e-02 4 KSP Residual norm 2.429920456862e-02 5 KSP Residual norm 6.340626338374e-03 6 KSP Residual norm 1.424377857206e-03 7 KSP Residual norm 3.363988738854e-04 8 KSP Residual norm 8.037845321522e-05 9 KSP Residual norm 2.686010744599e-05 10 KSP Residual norm 8.146783408043e-06 11 KSP Residual norm 2.120186758098e-06 12 KSP Residual norm 5.082140903932e-07 13 KSP Residual norm 1.400265096864e-07 14 KSP Residual norm 3.973779435185e-08 15 KSP Residual norm 1.039633800154e-08 16 KSP Residual norm 3.001850910227e-09 17 KSP Residual norm 8.001618657838e-10 18 KSP Residual norm 1.960472955866e-10 19 KSP Residual norm 4.799123661593e-11 20 KSP Residual norm 1.221838942841e-11 21 KSP Residual norm 3.330996894394e-12 22 KSP Residual norm 8.992653107914e-13 KSP Object: 16384 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 3.100886e-02 0 KSP Residual norm 1.198339325192e+02 1 KSP Residual norm 5.077605791138e-02 2 KSP Residual norm 9.040859657225e-03 3 KSP Residual norm 1.976302311761e-03 4 KSP Residual norm 4.942295431687e-04 5 KSP Residual norm 1.176017107054e-04 6 KSP Residual norm 3.030342472034e-05 7 KSP Residual norm 7.282599936494e-06 8 KSP Residual norm 1.848321725491e-06 9 KSP Residual norm 5.061613678429e-07 10 KSP Residual norm 1.253132797998e-07 11 KSP Residual norm 3.349053922549e-08 12 KSP Residual norm 8.292706953558e-09 13 KSP Residual norm 2.102309796990e-09 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.764869448851e+01 1 KSP Residual norm 3.563518815564e-02 2 KSP Residual norm 6.217896830000e-03 3 KSP Residual norm 1.320138075074e-03 4 KSP Residual norm 3.278446771013e-04 5 KSP Residual norm 7.872532354986e-05 6 KSP Residual norm 2.056763309626e-05 7 KSP Residual norm 5.235039152279e-06 8 KSP Residual norm 1.312815038800e-06 9 KSP Residual norm 3.573552989001e-07 10 KSP Residual norm 8.722360141195e-08 11 KSP Residual norm 2.259550836909e-08 12 KSP Residual norm 5.399954114973e-09 13 KSP Residual norm 1.304115290335e-09 14 KSP Residual norm 3.168623658490e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.767284132535e+01 1 KSP Residual norm 3.590512280275e-02 2 KSP Residual norm 6.319528594382e-03 3 KSP Residual norm 1.357160777826e-03 4 KSP Residual norm 3.373458486221e-04 5 KSP Residual norm 8.050695879112e-05 6 KSP Residual norm 2.093149298002e-05 7 KSP Residual norm 5.218612016115e-06 8 KSP Residual norm 1.322414837264e-06 9 KSP Residual norm 3.609258789046e-07 10 KSP Residual norm 8.802006986003e-08 11 KSP Residual norm 2.288536565033e-08 12 KSP Residual norm 5.470308498407e-09 13 KSP Residual norm 1.345384401663e-09 14 KSP Residual norm 3.232704145695e-10 KSP Object: 16384 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 7.300800716459e-01 1 KSP Residual norm 7.599905721982e-03 2 KSP Residual norm 6.354253586702e-04 3 KSP Residual norm 2.524608156352e-04 4 KSP Residual norm 8.632712981245e-05 5 KSP Residual norm 3.279065991192e-05 6 KSP Residual norm 1.118298160143e-05 7 KSP Residual norm 3.101006119658e-06 8 KSP Residual norm 6.254548939738e-07 9 KSP Residual norm 1.532735526786e-07 10 KSP Residual norm 4.160279913860e-08 11 KSP Residual norm 1.309560823195e-08 12 KSP Residual norm 3.698391134188e-09 13 KSP Residual norm 9.172120301703e-10 14 KSP Residual norm 2.324006371816e-10 15 KSP Residual norm 6.108200394839e-11 16 KSP Residual norm 1.637150796743e-11 17 KSP Residual norm 4.106807756334e-12 18 KSP Residual norm 1.048540004614e-12 KSP Object: 16384 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 16384 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 16384 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 6.935734e-06 Iteration #0001 : tn = 1.50163e+02, percent done : 0.0%, max_L2_norm_u = 1.16966e+00, number of leaves = 270631545 Mean computational time spent on viscosity step: 7.54276e+01 projection step: 3.11505e+02 computing velocities at nodes: 1.71697e+01 grid update: 2.46302e+01 full iteration (total): 4.32508e+02 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /work/04965/tg842642/stampede2/build/flow_past_sphere/test_petsc/flow_past_sphere on a skylake named c401-111.stampede2.tacc.utexas.edu with 16384 processors, by tg842642 Mon May 6 10:21:08 2019 Using Petsc Release Version 3.11.0, Mar, 29, 2019 Max Max/Min Avg Total Time (sec): 1.212e+03 1.000 1.212e+03 Objects: 1.343e+03 1.000 1.343e+03 Flop: 4.009e+08 3.015 3.041e+08 4.983e+12 Flop/sec: 3.308e+05 3.015 2.510e+05 4.112e+09 MPI Messages: 2.729e+05 26.806 4.558e+04 7.468e+08 MPI Message Lengths: 4.337e+08 7.221 3.726e+03 2.782e+12 MPI Reductions: 2.647e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 1.2118e+03 100.0% 4.9826e+12 100.0% 7.468e+08 100.0% 3.726e+03 100.0% 2.640e+03 99.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 21 1.0 5.9611e+0115.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecDot 494 1.0 1.4164e+00 4.0 1.77e+07 1.1 0.0e+00 0.0e+00 4.9e+02 0 5 0 0 19 0 5 0 0 19 190466 VecDotNorm2 247 1.0 1.1663e+00 5.0 1.77e+07 1.1 0.0e+00 0.0e+00 2.5e+02 0 5 0 0 9 0 5 0 0 9 231322 VecTDot 158 1.0 2.4957e-01 2.9 5.22e+06 1.0 0.0e+00 0.0e+00 1.6e+02 0 2 0 0 6 0 2 0 0 6 342634 VecNorm 350 1.0 3.9557e+02345.4 1.23e+07 1.1 0.0e+00 0.0e+00 3.5e+02 0 4 0 0 13 0 4 0 0 13 482 VecCopy 394 1.0 4.0224e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 138 1.0 6.4326e-0216.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 164 1.0 4.4140e-02 6.7 5.44e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 2011528 VecAYPX 77 1.0 2.8433e-0196.6 2.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 144664 VecAXPBYCZ 494 1.0 5.3091e-02 1.8 3.54e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 0 11 0 0 0 10163063 VecWAXPY 494 1.0 3.8152e-02 1.4 1.77e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 7071186 VecAssemblyBegin 11 1.0 5.2940e+01489.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecAssemblyEnd 11 1.0 5.9209e-0352.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecLoad 11 1.0 3.0705e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.1e+01 25 0 0 0 1 25 0 0 0 1 0 VecScatterBegin 1513 1.0 1.6795e+0014.9 0.00e+00 0.0 5.6e+08 4.0e+03 0.0e+00 0 0 75 81 0 0 0 75 81 0 0 VecScatterEnd 1513 1.0 5.3522e+01 7.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 MatMult 581 1.0 6.2042e+0014.5 1.52e+0815.8 1.1e+08 2.7e+03 0.0e+00 0 34 14 11 0 0 34 14 11 0 270437 MatSOR 512 1.0 1.2334e+00 5.7 1.36e+08 8.0 0.0e+00 0.0e+00 0.0e+00 0 31 0 0 0 0 31 0 0 0 1256500 MatConvert 2 1.0 7.0418e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 10 1.0 7.6426e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 10 1.0 6.2901e-01 1.2 0.00e+00 0.0 2.9e+06 6.9e+02 7.2e+01 0 0 0 0 3 0 0 0 0 3 0 MatGetRowIJ 4 1.0 4.6198e-022306.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 16 1.0 4.5780e-02 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.6e+01 0 0 0 0 1 0 0 0 0 1 0 KSPSetUp 16 1.0 3.3807e-02 7.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 16 1.0 6.0091e+02 1.0 4.01e+08 3.0 1.1e+08 2.7e+03 1.3e+03 49100 14 11 47 49100 14 11 48 8292 PCSetUp 8 1.0 1.9564e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 16 0 0 0 0 16 0 0 0 0 0 PCApply 597 1.0 4.0145e+02219.8 1.36e+08 8.0 0.0e+00 0.0e+00 0.0e+00 33 31 0 0 0 33 31 0 0 0 3860 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 601 601 54667136 0. Index Set 356 356 5436232 0. IS L to G Mapping 159 159 14754764 0. Vec Scatter 178 178 1724232 0. Viewer 7 6 5040 0. Krylov Solver 8 8 11392 0. Matrix 26 26 19078480 0. Preconditioner 8 8 8528 0. ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000104809 Average time for zero size MPI_Send(): 8.617e-06 #PETSc Option Table entries: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -ksp_monitor -ksp_view -log_view -niter_hodge 2 -pc_cell hypre -pc_hypre_boomeramg_coarsen_type Falgout -pc_hypre_boomeramg_strong_threshold 0.5 -pc_hypre_boomeramg_truncfactor 0.1 -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing -vecload_block_size 1 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-x=0 -with-pic --with-make-np=8 --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" --with-packages-build-dir=/tmp/petsc-3.11/skylake --with-mpi=1 --with-cc=/opt/apps/intel18/impi/18.0.2/bin/mpicc --with-cxx=/opt/apps/intel18/impi/18.0.2/bin/mpicxx --with-fc=/opt/apps/intel18/impi/18.0.2/bin/mpif90 --with-mpi-include=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt/libmpi.so --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-plapack=1 --download-plapack --with-spai=1 --download-spai --with-sundials=1 --download-sundials --with-elemental=1 --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 --download-hdf5=1 --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 --with-debugging=no --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" LIBS= ----------------------------------------- Libraries compiled on 2019-04-02 03:10:34 on build.stampede2.tacc.utexas.edu Machine characteristics: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /home1/apps/intel18/impi18_0/petsc/3.11 Using PETSc arch: skylake ----------------------------------------- Using C compiler: /opt/apps/intel18/impi/18.0.2/bin/mpicc -fPIC -wd1572 -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g Using Fortran compiler: /opt/apps/intel18/impi/18.0.2/bin/mpif90 -fPIC -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g ----------------------------------------- Using include paths: -I/home1/apps/intel18/impi18_0/petsc/3.11/include -I/home1/apps/intel18/impi18_0/petsc/3.11/skylake/include -I/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include ----------------------------------------- Using C linker: /opt/apps/intel18/impi/18.0.2/bin/mpicc Using Fortran linker: /opt/apps/intel18/impi/18.0.2/bin/mpif90 Using libraries: -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -lpetsc -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -L/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib64 -L/opt/apps/gcc/6.3.0/lib64 -Wl,-rpath,/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -L/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -Wl,-rpath,/opt/apps/gcc/6.3.0/lib -L/opt/apps/gcc/6.3.0/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr -lspai -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lzoltan -lparmetis -lmetis -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lmpi -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lrt -lm -lpthread -lz -lstdc++ -ldl ----------------------------------------- TACC: Shutdown complete. Exiting.