0 KSP Residual norm 1.495651392161e+03 1 KSP Residual norm 3.649891107619e+02 2 KSP Residual norm 2.117146192828e+02 3 KSP Residual norm 1.445856885170e+02 4 KSP Residual norm 1.077424410387e+02 5 KSP Residual norm 9.198111477688e+01 6 KSP Residual norm 7.533215136725e+01 7 KSP Residual norm 6.451868462772e+01 8 KSP Residual norm 5.872899850046e+01 9 KSP Residual norm 5.061580432653e+01 10 KSP Residual norm 4.708451245418e+01 11 KSP Residual norm 4.237917691640e+01 12 KSP Residual norm 3.901940233527e+01 13 KSP Residual norm 3.646993301702e+01 14 KSP Residual norm 3.332041450528e+01 15 KSP Residual norm 3.194496692785e+01 16 KSP Residual norm 2.928883693435e+01 17 KSP Residual norm 2.800449787474e+01 18 KSP Residual norm 2.659073917787e+01 19 KSP Residual norm 2.468772425132e+01 20 KSP Residual norm 2.408386623759e+01 21 KSP Residual norm 2.259733320445e+01 22 KSP Residual norm 2.165089065314e+01 23 KSP Residual norm 2.087552323842e+01 24 KSP Residual norm 1.983208395628e+01 25 KSP Residual norm 1.921481237253e+01 26 KSP Residual norm 1.836751251265e+01 27 KSP Residual norm 1.772999436226e+01 28 KSP Residual norm 1.717025579150e+01 29 KSP Residual norm 1.648753232895e+01 30 KSP Residual norm 1.595404607384e+01 31 KSP Residual norm 1.553529314488e+01 32 KSP Residual norm 1.488033633220e+01 33 KSP Residual norm 1.450317697522e+01 34 KSP Residual norm 1.410137231648e+01 35 KSP Residual norm 1.350442954302e+01 36 KSP Residual norm 1.332756202308e+01 37 KSP Residual norm 1.275864495790e+01 38 KSP Residual norm 1.242357295332e+01 39 KSP Residual norm 1.222135293906e+01 40 KSP Residual norm 1.162533517126e+01 41 KSP Residual norm 1.158693300904e+01 42 KSP Residual norm 1.113234572782e+01 43 KSP Residual norm 1.082139597868e+01 44 KSP Residual norm 1.078038754391e+01 45 KSP Residual norm 1.020082046247e+01 46 KSP Residual norm 1.021689722133e+01 47 KSP Residual norm 9.907150556837e+00 48 KSP Residual norm 9.547388096064e+00 49 KSP Residual norm 9.600154708139e+00 50 KSP Residual norm 9.110924039435e+00 51 KSP Residual norm 9.065506894706e+00 52 KSP Residual norm 8.878368576554e+00 53 KSP Residual norm 8.505425004955e+00 54 KSP Residual norm 8.581392859885e+00 55 KSP Residual norm 8.148007421767e+00 56 KSP Residual norm 8.065927739730e+00 57 KSP Residual norm 7.990533170413e+00 58 KSP Residual norm 7.556927487891e+00 59 KSP Residual norm 7.675404973915e+00 60 KSP Residual norm 7.333803913416e+00 61 KSP Residual norm 7.166317051638e+00 62 KSP Residual norm 7.198844469611e+00 63 KSP Residual norm 6.774954193917e+00 64 KSP Residual norm 6.872066969908e+00 65 KSP Residual norm 6.617813188020e+00 66 KSP Residual norm 6.412411326410e+00 67 KSP Residual norm 6.491178230546e+00 68 KSP Residual norm 6.101212156660e+00 69 KSP Residual norm 6.182217362982e+00 70 KSP Residual norm 6.103620464104e+00 71 KSP Residual norm 6.095418939334e+00 72 KSP Residual norm 6.775974052758e+00 73 KSP Residual norm 7.527726872709e+00 74 KSP Residual norm 9.072043988077e+00 75 KSP Residual norm 9.636613703974e+00 76 KSP Residual norm 7.657551468431e+00 77 KSP Residual norm 5.481479203936e+00 78 KSP Residual norm 4.259497268317e+00 79 KSP Residual norm 4.436470054159e+00 80 KSP Residual norm 4.564357893234e+00 81 KSP Residual norm 3.720705638968e+00 82 KSP Residual norm 3.456690901373e+00 83 KSP Residual norm 3.453431604562e+00 84 KSP Residual norm 3.151142098664e+00 85 KSP Residual norm 2.882452789367e+00 86 KSP Residual norm 2.798244762166e+00 87 KSP Residual norm 2.572182250139e+00 88 KSP Residual norm 2.394429603093e+00 89 KSP Residual norm 2.315722094218e+00 90 KSP Residual norm 2.101641503104e+00 91 KSP Residual norm 2.011937794513e+00 92 KSP Residual norm 1.868463333889e+00 93 KSP Residual norm 1.741962828026e+00 94 KSP Residual norm 1.650905494042e+00 95 KSP Residual norm 1.515305169977e+00 96 KSP Residual norm 1.421981579767e+00 97 KSP Residual norm 1.317045136017e+00 98 KSP Residual norm 1.215586504381e+00 99 KSP Residual norm 1.126380158677e+00 100 KSP Residual norm 1.045448228934e+00 101 KSP Residual norm 9.390091098393e-01 102 KSP Residual norm 8.829404315108e-01 103 KSP Residual norm 7.760121248092e-01 104 KSP Residual norm 7.253597449597e-01 105 KSP Residual norm 6.586582757537e-01 106 KSP Residual norm 6.051736039622e-01 107 KSP Residual norm 5.790585710076e-01 108 KSP Residual norm 5.259594077655e-01 109 KSP Residual norm 4.982302792693e-01 110 KSP Residual norm 4.593048900932e-01 111 KSP Residual norm 4.168158507746e-01 112 KSP Residual norm 3.969629457262e-01 113 KSP Residual norm 3.546379850023e-01 114 KSP Residual norm 3.332453713647e-01 115 KSP Residual norm 3.068925104294e-01 116 KSP Residual norm 2.756944445656e-01 117 KSP Residual norm 2.635375966688e-01 118 KSP Residual norm 2.325001353311e-01 119 KSP Residual norm 2.199234046339e-01 120 KSP Residual norm 1.994580647155e-01 121 KSP Residual norm 1.812120424979e-01 122 KSP Residual norm 1.683880795172e-01 123 KSP Residual norm 1.507657264996e-01 124 KSP Residual norm 1.376966981436e-01 125 KSP Residual norm 1.258652583185e-01 126 KSP Residual norm 1.113645108302e-01 127 KSP Residual norm 1.026205995037e-01 128 KSP Residual norm 9.068139854964e-02 129 KSP Residual norm 8.119198385262e-02 130 KSP Residual norm 7.350479129364e-02 131 KSP Residual norm 6.334173405612e-02 132 KSP Residual norm 5.811559484006e-02 133 KSP Residual norm 4.952193458274e-02 134 KSP Residual norm 4.427509896691e-02 135 KSP Residual norm 3.842091471301e-02 136 KSP Residual norm 3.277284939040e-02 137 KSP Residual norm 2.889849060988e-02 138 KSP Residual norm 2.391014409595e-02 139 KSP Residual norm 2.080839323584e-02 140 KSP Residual norm 1.726070845998e-02 141 KSP Residual norm 1.450952641061e-02 Residual norm 3.26933e-05 Summary of Memory Usage in PETSc Maximum (over computational time) process memory: total 1.9399e+10 max 9.7000e+09 min 9.6992e+09 Current process memory: total 1.8596e+09 max 9.3022e+08 min 9.2937e+08 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex45 on a named glados.dl.ac.uk with 2 processors, by kchockalingam Wed Nov 24 09:19:33 2021 Using Petsc Release Version 3.15.3, Aug 06, 2021 Max Max/Min Avg Total Time (sec): 2.222e+02 1.000 2.222e+02 Objects: 5.800e+01 1.000 5.800e+01 Flop: 2.963e+11 1.000 2.963e+11 5.925e+11 Flop/sec: 1.333e+09 1.000 1.333e+09 2.666e+09 MPI Messages: 2.960e+02 1.000 2.960e+02 5.920e+02 MPI Message Lengths: 3.191e+08 1.000 1.078e+06 6.381e+08 MPI Reductions: 5.360e+02 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 2.2224e+02 100.0% 5.9253e+11 100.0% 5.920e+02 100.0% 1.078e+06 100.0% 5.180e+02 96.6% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors) CpuToGpu Count: total number of CPU to GPU copies per processor CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor) GpuToCpu Count: total number of GPU to CPU copies per processor GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor) GPU %F: percent flops on GPU in this event ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F --------------------------------------------------------------------------------------------------------------------------------------------------------------- --- Event Stage 0: Main Stage BuildTwoSided 3 1.0 8.7330e-02 4.3 0.00e+00 0.0 2.0e+00 4.0e+00 3.0e+00 0 0 0 0 1 0 0 0 0 1 0 0 0 0.00e+00 0 0.00e+00 0 BuildTwoSidedF 2 1.0 8.7315e-02 4.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 MatMult 294 1.0 2.8526e+00 1.3 9.50e+10 1.0 5.9e+02 1.1e+06 1.0e+00 1 32100100 0 1 32100100 0 66605 109985 2 2.19e+03 0 0.00e+00 100 MatSOR 295 1.0 1.8572e+02 1.0 1.03e+11 1.0 0.0e+00 0.0e+00 0.0e+00 83 35 0 0 0 83 35 0 0 0 1105 0 0 0.00e+00 576 1.15e+05 0 MatAssemblyBegin 2 1.0 8.7371e-02 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 MatAssemblyEnd 2 1.0 6.8931e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 1 0 0 0 0 1 0 0 0 0.00e+00 0 0.00e+00 0 MatCUSPARSCopyTo 2 1.0 3.6613e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 2 2.19e+03 0 0.00e+00 0 KSPSetUp 2 1.0 9.2705e+00 1.0 1.39e+10 1.0 2.4e+01 9.5e+05 6.6e+01 4 5 4 4 12 4 5 4 4 13 3006 116114 36 4.38e+03 31 1.99e+03 72 KSPSolve 1 1.0 1.9028e+02 1.0 2.82e+11 1.0 5.7e+02 1.1e+06 4.2e+02 86 95 96 96 79 86 95 96 96 82 2963 103602 1417 5.68e+04 990 1.13e+05 65 KSPGMRESOrthog 10 1.0 4.0958e-01 1.1 5.48e+09 1.0 0.0e+00 0.0e+00 1.0e+01 0 2 0 0 2 0 2 0 0 2 26768 129969 20 1.99e+03 10 5.32e-02 100 DMCreateMat 1 1.0 1.7783e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.0e+00 8 0 0 0 1 8 0 0 0 1 0 0 0 0.00e+00 0 0.00e+00 0 SFSetGraph 2 1.0 1.2732e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 SFSetUp 1 1.0 2.4075e-03 1.0 0.00e+00 0.0 4.0e+00 2.7e+05 1.0e+00 0 0 1 0 0 0 0 1 0 0 0 0 0 0.00e+00 0 0.00e+00 0 SFPack 294 1.0 9.5077e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 SFUnpack 294 1.0 6.5334e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 VecMDot 10 1.0 3.5582e-01 1.2 2.74e+09 1.0 0.0e+00 0.0e+00 1.0e+01 0 1 0 0 2 0 1 0 0 2 15407 177096 10 1.99e+03 10 5.32e-02 100 VecTDot 282 1.0 1.8488e-01 1.0 1.41e+10 1.0 0.0e+00 0.0e+00 2.8e+02 0 5 0 0 53 0 5 0 0 54 152030 156002 0 0.00e+00 282 2.26e-03 100 VecNorm 154 1.0 7.9338e-01 9.0 7.67e+09 1.0 0.0e+00 0.0e+00 1.5e+02 0 3 0 0 29 0 3 0 0 30 19347 260968 1 1.99e+02 154 1.23e-03 100 VecScale 11 1.0 7.4799e-03 1.0 2.74e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 73289 73338 11 8.80e-05 0 0.00e+00 100 VecCopy 287 1.0 2.6882e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 VecSet 313 1.0 1.0696e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 VecAXPY 284 1.0 3.1031e-01 1.0 1.42e+10 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 91223 101939 285 1.99e+02 0 0.00e+00 100 VecAYPX 424 1.0 4.6610e+00 1.0 2.11e+10 1.0 0.0e+00 0.0e+00 0.0e+00 2 7 0 0 0 2 7 0 0 0 9067 59976 567 2.85e+04 0 0.00e+00 100 VecAXPBYCZ 142 1.0 4.2748e+00 1.0 3.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 2 12 0 0 0 2 12 0 0 0 16554 187273 568 2.83e+04 0 0.00e+00 100 VecMAXPY 11 1.0 6.3508e-02 1.0 3.24e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 102014 102026 11 5.20e-04 0 0.00e+00 100 VecScatterBegin 294 1.0 6.4442e-02 1.0 0.00e+00 0.0 5.9e+02 1.1e+06 1.0e+00 0 0100100 0 0 0100100 0 0 0 0 0.00e+00 0 0.00e+00 0 VecScatterEnd 294 1.0 7.4767e-0138.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 VecNormalize 11 1.0 4.3984e-02 1.2 8.22e+08 1.0 0.0e+00 0.0e+00 1.1e+01 0 0 0 0 2 0 0 0 0 2 37391 141046 12 1.99e+02 11 8.80e-05 100 VecCUDACopyTo 297 1.0 8.1843e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 0 297 5.92e+04 0 0.00e+00 0 VecCUDACopyFrom 576 1.0 1.7463e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 8 0 0 0 0 0 0 0 0.00e+00 576 1.15e+05 0 PCSetUp 1 1.0 9.0794e+00 1.0 1.39e+10 1.0 2.4e+01 9.5e+05 6.0e+01 4 5 4 4 11 4 5 4 4 12 3069 116796 36 4.38e+03 31 1.99e+03 72 PCApply 142 1.0 1.8867e+02 1.0 1.94e+11 1.0 2.8e+02 1.1e+06 0.0e+00 85 66 48 48 0 85 66 48 48 0 2058 97012 995 5.68e+04 566 1.13e+05 49 --------------------------------------------------------------------------------------------------------------------------------------------------------------- Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 3 3 34072 0. DMKSP interface 1 1 664 0. Matrix 3 3 3683542300 0. Distributed Mesh 2 2 10608 0. Index Set 4 4 100759064 0. IS L to G Mapping 1 1 100214440 0. Star Forest Graph 6 6 7056 0. Discrete System 2 2 1808 0. Weak Form 2 2 1648 0. Vector 29 29 4984738072 0. Preconditioner 3 3 3040 0. Viewer 2 1 848 0. ======================================================================================================================== Average time to get PetscTime(): 3.12924e-08 Average time for MPI_Barrier(): 8.24034e-07 Average time for zero size MPI_Send(): 4.88758e-06 #PETSc Option Table entries: -da_grid_x 368 -da_grid_y 368 -da_grid_z 368 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_monitor -ksp_type cg -log_view -malloc_log -memory_view -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 --download-hypre-configure-arguments=--enable-gpu-profiling=yes,--enable-cusparse=yes,--enable-cublas=yes,--enable-curand=yes,HYPRE_CUDA_SM=70 --with-debugging=no PETSC_ARCH=arch-ci-linux-cuda11-hypre-double ----------------------------------------- Libraries compiled on 2021-11-18 14:19:41 on glados.dl.ac.uk Machine characteristics: Linux-4.18.0-193.6.3.el8_2.x86_64-x86_64-with-centos-8.2.2004-Core Using PETSc directory: /home/kchockalingam/tools/petsc-3.15.3 Using PETSc arch: ----------------------------------------- Using C compiler: /apps/packages/gcc/7.3.0/openmpi/3.1.2/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O3 -fno-omit-frame-pointer Using Fortran compiler: /apps/packages/gcc/7.3.0/openmpi/3.1.2/bin/mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -O3 -fno-omit-frame-pointer ----------------------------------------- Using include paths: -I/home/kchockalingam/tools/petsc-3.15.3/include -I/home/kchockalingam/tools/petsc-3.15.3/arch-ci-linux-cuda11-hypre-double/include -I/home/kchockalingam/petsc-hash-pkgs/194329/include -I/apps/packages/gcc/7.3.0/openmpi/3.1.2/include -I/apps/packages/cuda/10.1/include ----------------------------------------- Using C linker: /apps/packages/gcc/7.3.0/openmpi/3.1.2/bin/mpicc Using Fortran linker: /apps/packages/gcc/7.3.0/openmpi/3.1.2/bin/mpif90 Using libraries: -Wl,-rpath,/home/kchockalingam/tools/petsc-3.15.3/lib -L/home/kchockalingam/tools/petsc-3.15.3/lib -lpetsc -Wl,-rpath,/home/kchockalingam/petsc-hash-pkgs/194329/lib -L/home/kchockalingam/petsc-hash-pkgs/194329/lib -Wl,-rpath,/apps/packages/cuda/10.1/lib64 -L/apps/packages/cuda/10.1/lib64 -Wl,-rpath,/apps/packages/gcc/7.3.0/openmpi/3.1.2/lib -L/apps/packages/gcc/7.3.0/openmpi/3.1.2/lib -Wl,-rpath,/apps/packages/compilers/gcc/7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -L/apps/packages/compilers/gcc/7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -Wl,-rpath,/apps/packages/compilers/gcc/7.3.0/lib64 -L/apps/packages/compilers/gcc/7.3.0/lib64 -Wl,-rpath,/apps/packages/compilers/gcc/7.3.0/lib -L/apps/packages/compilers/gcc/7.3.0/lib -lHYPRE -llapack -lblas -lcufft -lcublas -lcudart -lcusparse -lcusolver -lcurand -lX11 -lstdc++ -ldl -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lutil -lrt -lz -lgfortran -lm -lgfortran -lgcc_s -lquadmath -lpthread -lquadmath -lstdc++ -ldl -----------------------------------------