TACC: Starting up job 3486686 TACC: Starting parallel tasks... [WARNING]: option 'log_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_view' does not exists in the database -- ignoring. [WARNING]: option 'ksp_monitor' does not exists in the database -- ignoring. -------------------== CASL Options Database ==------------------- List of entered options: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -niter_hodge 2 -pc_cell hypre -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing no-arg ----------------------------------------------------------------- Loaded solver state from ... /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 Parameters : Re = 500., mu = 0.004, rho = 1., grid is 8x4x4 cfl = 1., uniform_band = 16. Total runtime ... 0 KSP Residual norm 6.581878233857e+03 1 KSP Residual norm 6.059778363509e+02 2 KSP Residual norm 2.414070145720e+02 3 KSP Residual norm 9.086173629925e+01 4 KSP Residual norm 1.583473816891e+01 5 KSP Residual norm 7.300202010437e+00 6 KSP Residual norm 3.756739034247e+00 7 KSP Residual norm 1.632260626999e+00 8 KSP Residual norm 6.193038171066e-01 9 KSP Residual norm 2.429522383818e-01 10 KSP Residual norm 1.108145018870e-01 11 KSP Residual norm 4.844381935594e-02 12 KSP Residual norm 2.121754812325e-02 13 KSP Residual norm 8.711694074971e-03 14 KSP Residual norm 4.182977224483e-03 15 KSP Residual norm 1.796821337819e-03 16 KSP Residual norm 7.935364994413e-04 17 KSP Residual norm 3.536006241392e-04 18 KSP Residual norm 1.604458697113e-04 19 KSP Residual norm 7.146375148773e-05 20 KSP Residual norm 3.060902044267e-05 21 KSP Residual norm 1.335408844225e-05 22 KSP Residual norm 6.494037182351e-06 23 KSP Residual norm 2.819176675740e-06 24 KSP Residual norm 1.305012678195e-06 25 KSP Residual norm 5.792018926453e-07 26 KSP Residual norm 2.685282721039e-07 27 KSP Residual norm 1.228584348899e-07 28 KSP Residual norm 5.475107132947e-08 29 KSP Residual norm 2.531246694554e-08 30 KSP Residual norm 1.122778047754e-08 31 KSP Residual norm 4.921670586745e-09 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.341462007664e+02 1 KSP Residual norm 1.595166378893e+02 2 KSP Residual norm 5.358858045127e+01 3 KSP Residual norm 1.372000883708e+01 4 KSP Residual norm 4.523077430927e+00 5 KSP Residual norm 2.336941029058e+00 6 KSP Residual norm 1.044585576086e+00 7 KSP Residual norm 3.998039291834e-01 8 KSP Residual norm 1.718520181918e-01 9 KSP Residual norm 7.897278516445e-02 10 KSP Residual norm 3.508835451554e-02 11 KSP Residual norm 1.484803941402e-02 12 KSP Residual norm 6.513078627962e-03 13 KSP Residual norm 2.868541141855e-03 14 KSP Residual norm 1.294176853513e-03 15 KSP Residual norm 5.525579084608e-04 16 KSP Residual norm 2.581652111907e-04 17 KSP Residual norm 1.142591002612e-04 18 KSP Residual norm 5.181302584265e-05 19 KSP Residual norm 2.219956046243e-05 20 KSP Residual norm 9.752366093706e-06 21 KSP Residual norm 4.455808328407e-06 22 KSP Residual norm 1.996802268180e-06 23 KSP Residual norm 8.705490727179e-07 24 KSP Residual norm 4.067579783495e-07 25 KSP Residual norm 1.788799015417e-07 26 KSP Residual norm 8.256002856172e-08 27 KSP Residual norm 3.777219471012e-08 28 KSP Residual norm 1.703609767304e-08 29 KSP Residual norm 7.772304117841e-09 30 KSP Residual norm 3.524928113364e-09 31 KSP Residual norm 1.595682010492e-09 32 KSP Residual norm 7.014139964559e-10 33 KSP Residual norm 2.972274538462e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 5.380135798611e+02 1 KSP Residual norm 1.603824476479e+02 2 KSP Residual norm 5.351521790485e+01 3 KSP Residual norm 1.358848631924e+01 4 KSP Residual norm 4.484947201134e+00 5 KSP Residual norm 2.314818863438e+00 6 KSP Residual norm 1.035752475985e+00 7 KSP Residual norm 3.966178127869e-01 8 KSP Residual norm 1.698060096678e-01 9 KSP Residual norm 7.822730755716e-02 10 KSP Residual norm 3.466664186866e-02 11 KSP Residual norm 1.462550544143e-02 12 KSP Residual norm 6.429769262814e-03 13 KSP Residual norm 2.831405970058e-03 14 KSP Residual norm 1.282593506700e-03 15 KSP Residual norm 5.496546099111e-04 16 KSP Residual norm 2.576962428262e-04 17 KSP Residual norm 1.141750893495e-04 18 KSP Residual norm 5.159454627729e-05 19 KSP Residual norm 2.204532534900e-05 20 KSP Residual norm 9.653641147083e-06 21 KSP Residual norm 4.380001588030e-06 22 KSP Residual norm 1.954638748329e-06 23 KSP Residual norm 8.382090732158e-07 24 KSP Residual norm 3.856953385653e-07 25 KSP Residual norm 1.688576327839e-07 26 KSP Residual norm 7.742791131064e-08 27 KSP Residual norm 3.550373464449e-08 28 KSP Residual norm 1.604588772262e-08 29 KSP Residual norm 7.327932793246e-09 30 KSP Residual norm 3.373930970628e-09 31 KSP Residual norm 1.527208193146e-09 32 KSP Residual norm 7.101590140229e-10 33 KSP Residual norm 3.126946039278e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 3.099074484706e+00 1 KSP Residual norm 1.581820210071e+00 2 KSP Residual norm 4.365706636902e-01 3 KSP Residual norm 1.550581158722e-01 4 KSP Residual norm 5.358114951397e-02 5 KSP Residual norm 1.327685961259e-02 6 KSP Residual norm 3.301213921441e-03 7 KSP Residual norm 7.148118751383e-04 8 KSP Residual norm 1.570847327016e-04 9 KSP Residual norm 4.834973427726e-05 10 KSP Residual norm 1.365554028512e-05 11 KSP Residual norm 3.629587730729e-06 12 KSP Residual norm 9.862637689428e-07 13 KSP Residual norm 2.772493523132e-07 14 KSP Residual norm 7.419199635310e-08 15 KSP Residual norm 1.977490749106e-08 16 KSP Residual norm 5.453033889595e-09 17 KSP Residual norm 1.330222814478e-09 18 KSP Residual norm 3.363097209201e-10 19 KSP Residual norm 8.843582275698e-11 20 KSP Residual norm 2.001112309449e-11 21 KSP Residual norm 4.984936331861e-12 22 KSP Residual norm 1.400066868481e-12 KSP Object: 8192 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 3.082825e-02 0 KSP Residual norm 1.180380389876e+02 1 KSP Residual norm 9.000325426163e-04 2 KSP Residual norm 2.905562436697e-04 3 KSP Residual norm 1.085654625198e-04 4 KSP Residual norm 4.591435261878e-05 5 KSP Residual norm 1.964024975101e-05 6 KSP Residual norm 8.406464718702e-06 7 KSP Residual norm 3.756402913819e-06 8 KSP Residual norm 1.582116711717e-06 9 KSP Residual norm 6.957171995424e-07 10 KSP Residual norm 3.130461705105e-07 11 KSP Residual norm 1.153111147174e-07 12 KSP Residual norm 5.762222552301e-08 13 KSP Residual norm 1.341762809178e-08 14 KSP Residual norm 4.991761326576e-09 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=272717884, cols=272717884 total: nonzeros=1580066459, allocated nonzeros=1585007707 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.563065363330e+01 1 KSP Residual norm 1.387607204443e-03 2 KSP Residual norm 4.427703648646e-04 3 KSP Residual norm 1.638627182030e-04 4 KSP Residual norm 6.840463142092e-05 5 KSP Residual norm 2.924518201166e-05 6 KSP Residual norm 1.246856417465e-05 7 KSP Residual norm 5.612670602874e-06 8 KSP Residual norm 2.386632770460e-06 9 KSP Residual norm 1.068828153244e-06 10 KSP Residual norm 4.886654658015e-07 11 KSP Residual norm 2.101778012844e-07 12 KSP Residual norm 9.297863908824e-08 13 KSP Residual norm 3.845563764370e-08 14 KSP Residual norm 1.653946802185e-08 15 KSP Residual norm 7.198180683947e-09 16 KSP Residual norm 3.061227318279e-09 17 KSP Residual norm 1.368348267537e-09 18 KSP Residual norm 5.948838523799e-10 19 KSP Residual norm 2.622260823778e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273191780, cols=273191780 total: nonzeros=1580943620, allocated nonzeros=1585884868 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.565504374782e+01 1 KSP Residual norm 1.383768189716e-03 2 KSP Residual norm 4.431881397437e-04 3 KSP Residual norm 1.644855235312e-04 4 KSP Residual norm 6.920958320528e-05 5 KSP Residual norm 2.980531999983e-05 6 KSP Residual norm 1.282952736658e-05 7 KSP Residual norm 5.816583380827e-06 8 KSP Residual norm 2.480686000245e-06 9 KSP Residual norm 1.107438505795e-06 10 KSP Residual norm 5.030953152096e-07 11 KSP Residual norm 2.190566209000e-07 12 KSP Residual norm 9.829928786326e-08 13 KSP Residual norm 4.050001422651e-08 14 KSP Residual norm 1.688386451685e-08 15 KSP Residual norm 6.786439258605e-09 16 KSP Residual norm 2.802144388146e-09 17 KSP Residual norm 1.261150452074e-09 18 KSP Residual norm 5.351981189104e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273185288, cols=273185288 total: nonzeros=1580922782, allocated nonzeros=1585864030 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 2.037653919628e+00 1 KSP Residual norm 1.201502345336e-04 2 KSP Residual norm 1.809425724241e-05 3 KSP Residual norm 4.542580780946e-06 4 KSP Residual norm 1.109932744495e-06 5 KSP Residual norm 2.886998572779e-07 6 KSP Residual norm 8.154663651327e-08 7 KSP Residual norm 2.569123881133e-08 8 KSP Residual norm 9.622292798847e-09 9 KSP Residual norm 3.880459781460e-09 10 KSP Residual norm 1.496866534802e-09 11 KSP Residual norm 4.623451968588e-10 12 KSP Residual norm 8.907562857192e-11 13 KSP Residual norm 1.794034214505e-11 14 KSP Residual norm 4.166741070566e-12 15 KSP Residual norm 1.091372142677e-12 KSP Object: 8192 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=270588873, cols=270588873 total: nonzeros=1577191465, allocated nonzeros=1582468633 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 8.017552e-07 Iteration #0000 : tn = 1.50161e+02, percent done : 0.0%, max_L2_norm_u = 1.16986e+00, number of leaves = 270588873 0 KSP Residual norm 7.198185897602e+03 1 KSP Residual norm 6.030135557056e+02 2 KSP Residual norm 9.021174012005e+01 3 KSP Residual norm 9.322965335372e+00 4 KSP Residual norm 2.689222309890e+00 5 KSP Residual norm 5.641667930998e-01 6 KSP Residual norm 1.204702060135e-01 7 KSP Residual norm 2.652869251001e-02 8 KSP Residual norm 7.205642302919e-03 9 KSP Residual norm 1.821344679222e-03 10 KSP Residual norm 4.788789048571e-04 11 KSP Residual norm 1.230162180790e-04 12 KSP Residual norm 2.956399555607e-05 13 KSP Residual norm 7.458942583330e-06 14 KSP Residual norm 1.869534297166e-06 15 KSP Residual norm 4.226565668433e-07 16 KSP Residual norm 8.838001474405e-08 17 KSP Residual norm 1.642900726915e-08 18 KSP Residual norm 3.617222815744e-09 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.381704824396e+02 1 KSP Residual norm 1.573922893485e+02 2 KSP Residual norm 1.567066186845e+01 3 KSP Residual norm 2.811626614135e+00 4 KSP Residual norm 6.654447173096e-01 5 KSP Residual norm 1.434608104127e-01 6 KSP Residual norm 3.403467185715e-02 7 KSP Residual norm 8.576834606752e-03 8 KSP Residual norm 2.272528541648e-03 9 KSP Residual norm 5.754984155396e-04 10 KSP Residual norm 1.407521316074e-04 11 KSP Residual norm 2.915762021307e-05 12 KSP Residual norm 6.006442022129e-06 13 KSP Residual norm 1.383746701634e-06 14 KSP Residual norm 3.300444124275e-07 15 KSP Residual norm 8.016509074229e-08 16 KSP Residual norm 2.054062081701e-08 17 KSP Residual norm 4.860754691566e-09 18 KSP Residual norm 1.272516365786e-09 19 KSP Residual norm 3.244800678468e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 8.400502030044e+02 1 KSP Residual norm 1.582664074774e+02 2 KSP Residual norm 1.579780497533e+01 3 KSP Residual norm 2.820624172632e+00 4 KSP Residual norm 6.625947450932e-01 5 KSP Residual norm 1.426753133653e-01 6 KSP Residual norm 3.405971217023e-02 7 KSP Residual norm 8.560079659324e-03 8 KSP Residual norm 2.267948954386e-03 9 KSP Residual norm 5.722820335673e-04 10 KSP Residual norm 1.378996332152e-04 11 KSP Residual norm 2.822197221142e-05 12 KSP Residual norm 5.735271290486e-06 13 KSP Residual norm 1.321920901160e-06 14 KSP Residual norm 3.182603146435e-07 15 KSP Residual norm 7.720353836312e-08 16 KSP Residual norm 1.977062591514e-08 17 KSP Residual norm 4.649366734326e-09 18 KSP Residual norm 1.210923001415e-09 19 KSP Residual norm 3.099839733458e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 1.086520264889e+00 1 KSP Residual norm 6.624822790933e-01 2 KSP Residual norm 1.674730462964e-01 3 KSP Residual norm 6.326754122546e-02 4 KSP Residual norm 2.470441330512e-02 5 KSP Residual norm 5.951913585570e-03 6 KSP Residual norm 1.344909165961e-03 7 KSP Residual norm 3.269178339694e-04 8 KSP Residual norm 6.647861040473e-05 9 KSP Residual norm 1.751805311916e-05 10 KSP Residual norm 4.971428617400e-06 11 KSP Residual norm 1.193851325667e-06 12 KSP Residual norm 2.739694393146e-07 13 KSP Residual norm 8.247561403066e-08 14 KSP Residual norm 2.833622803359e-08 15 KSP Residual norm 9.522481664197e-09 16 KSP Residual norm 3.141580447334e-09 17 KSP Residual norm 9.049666814940e-10 18 KSP Residual norm 2.452092506612e-10 19 KSP Residual norm 6.973286147452e-11 20 KSP Residual norm 2.045878216006e-11 21 KSP Residual norm 5.910675890169e-12 22 KSP Residual norm 1.612737195578e-12 23 KSP Residual norm 4.436906255258e-13 KSP Object: 8192 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #0, error = 6.023750e-03 0 KSP Residual norm 1.198339556456e+02 1 KSP Residual norm 4.905780985249e-02 2 KSP Residual norm 8.329740175917e-03 3 KSP Residual norm 1.741077125254e-03 4 KSP Residual norm 4.292023560397e-04 5 KSP Residual norm 1.020265286546e-04 6 KSP Residual norm 2.595300637170e-05 7 KSP Residual norm 6.134031290417e-06 8 KSP Residual norm 1.452029448754e-06 9 KSP Residual norm 3.914821215956e-07 10 KSP Residual norm 9.742821777031e-08 11 KSP Residual norm 2.604581105543e-08 12 KSP Residual norm 6.865431462836e-09 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=272758414, cols=272758414 total: nonzeros=1580326338, allocated nonzeros=1585267586 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.764871583392e+01 1 KSP Residual norm 3.451494051924e-02 2 KSP Residual norm 5.709999921288e-03 3 KSP Residual norm 1.157259606517e-03 4 KSP Residual norm 2.806630355745e-04 5 KSP Residual norm 6.703867596659e-05 6 KSP Residual norm 1.695187996929e-05 7 KSP Residual norm 4.188272539025e-06 8 KSP Residual norm 9.771773018707e-07 9 KSP Residual norm 2.620108119412e-07 10 KSP Residual norm 6.615994628341e-08 11 KSP Residual norm 1.736555655573e-08 12 KSP Residual norm 4.520878621313e-09 13 KSP Residual norm 1.127299701380e-09 14 KSP Residual norm 2.881137100623e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273234290, cols=273234290 total: nonzeros=1581208530, allocated nonzeros=1586149778 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 6.767286074569e+01 1 KSP Residual norm 3.472311329078e-02 2 KSP Residual norm 5.788682229250e-03 3 KSP Residual norm 1.181030992047e-03 4 KSP Residual norm 2.863402532193e-04 5 KSP Residual norm 6.785857056593e-05 6 KSP Residual norm 1.716879253107e-05 7 KSP Residual norm 4.236561020564e-06 8 KSP Residual norm 9.983121450465e-07 9 KSP Residual norm 2.680119242645e-07 10 KSP Residual norm 6.765097803197e-08 11 KSP Residual norm 1.786885378920e-08 12 KSP Residual norm 4.712542736160e-09 13 KSP Residual norm 1.198029383129e-09 14 KSP Residual norm 3.119116779531e-10 KSP Object: 8192 MPI processes type: bcgs maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=273226670, cols=273226670 total: nonzeros=1581185598, allocated nonzeros=1586126846 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines 0 KSP Residual norm 7.300743797576e-01 1 KSP Residual norm 7.492556612764e-03 2 KSP Residual norm 8.393495971767e-04 3 KSP Residual norm 3.623677572282e-04 4 KSP Residual norm 1.204073873631e-04 5 KSP Residual norm 3.643845284796e-05 6 KSP Residual norm 9.032407383696e-06 7 KSP Residual norm 2.052975340382e-06 8 KSP Residual norm 4.717430563824e-07 9 KSP Residual norm 1.309159290423e-07 10 KSP Residual norm 3.601228141185e-08 11 KSP Residual norm 1.314395913513e-08 12 KSP Residual norm 4.235373902754e-09 13 KSP Residual norm 1.091056116881e-09 14 KSP Residual norm 2.859041424027e-10 15 KSP Residual norm 7.455010383703e-11 16 KSP Residual norm 2.072091497837e-11 17 KSP Residual norm 5.940501575632e-12 18 KSP Residual norm 1.710969819620e-12 19 KSP Residual norm 4.516375175150e-13 KSP Object: 8192 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8192 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.5 Interpolation truncation factor 0.1 Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 8192 MPI processes type: mpiaij rows=270631545, cols=270631545 total: nonzeros=1577473397, allocated nonzeros=1582750565 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines hodge iteration #1, error = 6.935734e-06 Iteration #0001 : tn = 1.50163e+02, percent done : 0.0%, max_L2_norm_u = 1.16966e+00, number of leaves = 270631545 Mean computational time spent on viscosity step: 1.27363e+02 projection step: 2.81819e+02 computing velocities at nodes: 2.93914e+01 grid update: 4.40841e+01 full iteration (total): 4.89256e+02 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /work/04965/tg842642/stampede2/build/flow_past_sphere/test_petsc/flow_past_sphere on a skylake named c413-032.stampede2.tacc.utexas.edu with 8192 processors, by tg842642 Fri May 3 22:57:38 2019 Using Petsc Release Version 3.11.0, Mar, 29, 2019 Max Max/Min Avg Total Time (sec): 1.062e+03 1.000 1.062e+03 Objects: 1.343e+03 1.000 1.343e+03 Flop: 7.844e+08 2.973 6.036e+08 4.944e+12 Flop/sec: 7.388e+05 2.973 5.685e+05 4.657e+09 MPI Messages: 2.413e+05 23.743 4.448e+04 3.644e+08 MPI Message Lengths: 5.430e+08 6.088 5.670e+03 2.066e+12 MPI Reductions: 2.635e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 1.0617e+03 100.0% 4.9444e+12 100.0% 3.644e+08 100.0% 5.670e+03 100.0% 2.628e+03 99.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 21 1.0 1.3350e+01 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecDot 488 1.0 2.1281e+00 3.1 3.47e+07 1.1 0.0e+00 0.0e+00 4.9e+02 0 5 0 0 19 0 5 0 0 19 125229 VecDotNorm2 244 1.0 1.9516e+00 3.4 3.47e+07 1.1 0.0e+00 0.0e+00 2.4e+02 0 5 0 0 9 0 5 0 0 9 136561 VecTDot 158 1.0 2.8651e-01 2.9 1.04e+07 1.0 0.0e+00 0.0e+00 1.6e+02 0 2 0 0 6 0 2 0 0 6 298455 VecNorm 347 1.0 2.6291e+00 2.1 2.42e+07 1.1 0.0e+00 0.0e+00 3.5e+02 0 4 0 0 13 0 4 0 0 13 71920 VecCopy 394 1.0 6.1877e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 138 1.0 1.2153e+00142.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 164 1.0 6.6064e-02 4.2 1.09e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 1344000 VecAYPX 77 1.0 1.2812e-0118.1 5.02e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 321049 VecAXPBYCZ 488 1.0 1.0370e-01 1.5 6.94e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 0 11 0 0 0 5140055 VecWAXPY 488 1.0 8.2127e-02 1.2 3.47e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 3245053 VecAssemblyBegin 11 1.0 3.1321e+0053.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 11 1.0 4.7266e-0341.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecLoad 11 1.0 5.4419e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.1e+01 5 0 0 0 1 5 0 0 0 1 0 VecScatterBegin 1507 1.0 1.5547e+00 9.9 0.00e+00 0.0 2.7e+08 6.2e+03 0.0e+00 0 0 75 82 0 0 0 75 82 0 0 VecScatterEnd 1507 1.0 9.9323e+01 8.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 MatMult 575 1.0 1.1670e+0112.5 2.95e+0815.1 5.4e+07 4.2e+03 0.0e+00 0 34 15 11 0 0 34 15 11 0 142290 MatSOR 506 1.0 2.2620e+00 5.2 2.65e+08 7.8 0.0e+00 0.0e+00 0.0e+00 0 31 0 0 0 0 31 0 0 0 683828 MatConvert 2 1.0 1.3187e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 10 1.0 1.0943e+01 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 10 1.0 6.9632e-01 1.0 0.00e+00 0.0 1.5e+06 1.1e+03 7.2e+01 0 0 0 0 3 0 0 0 0 3 0 MatGetRowIJ 4 1.0 4.1761e-021607.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 16 1.0 5.6503e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.6e+01 0 0 0 0 1 0 0 0 0 1 0 KSPSetUp 16 1.0 1.2379e-0111.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 16 1.0 5.2181e+02 1.0 7.84e+08 3.0 5.4e+07 4.2e+03 1.2e+03 49100 15 11 47 49100 15 11 47 9475 PCSetUp 8 1.0 1.6026e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 15 0 0 0 0 15 0 0 0 0 0 PCApply 591 1.0 3.5565e+02 1.0 2.65e+08 7.8 0.0e+00 0.0e+00 0.0e+00 33 31 0 0 0 33 31 0 0 0 4349 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 601 601 101052168 0. Index Set 356 356 8297260 0. IS L to G Mapping 159 159 27253992 0. Vec Scatter 178 178 3189208 0. Viewer 7 6 5040 0. Krylov Solver 8 8 11392 0. Matrix 26 26 37961504 0. Preconditioner 8 8 8528 0. ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 Average time for MPI_Barrier(): 9.70364e-05 Average time for zero size MPI_Send(): 8.20801e-06 #PETSc Option Table entries: -cell_solver cg -cfl 1.0 -hodge_tol 0.0000001 -ksp_monitor -ksp_view -log_view -niter_hodge 2 -pc_cell hypre -pc_hypre_boomeramg_coarsen_type Falgout -pc_hypre_boomeramg_strong_threshold 0.5 -pc_hypre_boomeramg_truncfactor 0.1 -restart /work/04965/tg842642/stampede2/flow_past_sphere/3D/Re_500.00/lmin_6_lmax_11/backup_0 -timing -vecload_block_size 1 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-x=0 -with-pic --with-make-np=8 --with-packages-search-path="[/admin/build/admin/rpms/stampede2/SOURCES/petsc-packages]" --with-packages-build-dir=/tmp/petsc-3.11/skylake --with-mpi=1 --with-cc=/opt/apps/intel18/impi/18.0.2/bin/mpicc --with-cxx=/opt/apps/intel18/impi/18.0.2/bin/mpicxx --with-fc=/opt/apps/intel18/impi/18.0.2/bin/mpif90 --with-mpi-include=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt/libmpi.so --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-chaco=1 --download-chaco --with-hypre=1 --download-hypre --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-plapack=1 --download-plapack --with-spai=1 --download-spai --with-sundials=1 --download-sundials --with-elemental=1 --download-elemental --with-cxx-dialect=C++11 --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 --download-hdf5=1 --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-suitesparse=1 --download-suitesparse --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-zoltan=1 --download-zoltan=1 --download-ptscotch=1 --with-debugging=no --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl COPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" FOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" CXXOPTFLAGS="-xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g" LIBS= ----------------------------------------- Libraries compiled on 2019-04-02 03:10:34 on build.stampede2.tacc.utexas.edu Machine characteristics: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /home1/apps/intel18/impi18_0/petsc/3.11 Using PETSc arch: skylake ----------------------------------------- Using C compiler: /opt/apps/intel18/impi/18.0.2/bin/mpicc -fPIC -wd1572 -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g Using Fortran compiler: /opt/apps/intel18/impi/18.0.2/bin/mpif90 -fPIC -xCORE-AVX2 -axMIC-AVX512,COMMON-AVX512 -O2 -g ----------------------------------------- Using include paths: -I/home1/apps/intel18/impi18_0/petsc/3.11/include -I/home1/apps/intel18/impi18_0/petsc/3.11/skylake/include -I/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include ----------------------------------------- Using C linker: /opt/apps/intel18/impi/18.0.2/bin/mpicc Using Fortran linker: /opt/apps/intel18/impi/18.0.2/bin/mpif90 Using libraries: -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -lpetsc -Wl,-rpath,/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -L/home1/apps/intel18/impi18_0/petsc/3.11/skylake/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/release_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -L/opt/apps/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -Wl,-rpath,/opt/apps/gcc/6.3.0/lib64 -L/opt/apps/gcc/6.3.0/lib64 -Wl,-rpath,/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -L/opt/apps/gcc/6.3.0/x86_64-pc-linux-gnu/lib -Wl,-rpath,/opt/apps/gcc/6.3.0/lib -L/opt/apps/gcc/6.3.0/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr -lspai -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lzoltan -lparmetis -lmetis -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lmpi -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lrt -lm -lpthread -lz -lstdc++ -ldl ----------------------------------------- TACC: Shutdown complete. Exiting.