rank: 1, xmin = 43 0 0, xmax = 88 41 41 divisions: 2 1 1 rank: 0, xmin = -1 0 0, xmax = 44 41 41 lxSim: 88 42 42 3d LBM -- MPI version Initialize ContactMult Initialize Lattice Dir. Vectors Initialize MPI Variables Initialize Link Map # Grid data for a very simple unstructured grid of tetrhedrals Number of grid vertices: 1597 Number of Surface vertices: 645 Processor 0 assigned 799 vertices Processor 1 assigned 798 vertices Vertices assigned to processor 0 Vertices assigned to processor 1 Number of grid elements 7337 Processor 0 assigned 3669 elements Processor 1 assigned 3668 elements Elements assigned to processor 0 Elements assigned to processor 1 Element neighbors on processor 0 Element neighbors on processor 1 Reading node-to-element mapping for processor 0 Reading node-to-element mapping for processor 1 Initialize FLUID Using BGK fluid Update with tau != 1 Navier-Stokes flow: p_feq_3d=feq_3d Initialize WALLS Initialize FEA Initialize SOLID lbm rank:0 nlocal-350 nghost-295 lbm rank:1 nlocal-295 nghost-350 Total Memory requrements for local variables is 1185200 bytes Total Memory requrements for local variables is 1159496 bytes Which is 1.1303e+00 MB Which is 1.1058e+00 MB Set up FSI and Link data Starting a new simulation Loading 0 small particles and 1 large particles Loading 0 FEAtypes Time: 1 Number multi links: 332 Number multi links: 476 Convergence for initial acceleration: 8 1 time = 1.000e+00, : Convergence: : 9 Time: 2 2 time = 2.000e+00, : Convergence: : 9 Time: 3 3 time = 3.000e+00, : Convergence: : 9 Time: 4 4 time = 4.000e+00, : Convergence: : 9 Time: 5 5 time = 5.000e+00, : Convergence: : 9 Time: 6 6 time = 6.000e+00, : Convergence: : 9 Time: 7 7 time = 7.000e+00, : Convergence: : 9 Time: 8 8 time = 8.000e+00, : Convergence: : 9 Time: 9 9 time = 9.000e+00, : Convergence: : 9 Time: 10 10 time = 1.000e+01, : Convergence: : 10 Time: 11 11 time = 1.100e+01, : Convergence: : 10 Time: 12 12 time = 1.200e+01, : Convergence: : 10 Time: 13 13 time = 1.300e+01, : Convergence: : 10 Time: 14 14 time = 1.400e+01, : Convergence: : 10 Time: 15 15 time = 1.500e+01, : Convergence: : 10 Time: 16 16 time = 1.600e+01, : Convergence: : 10 Time: 17 17 time = 1.700e+01, : Convergence: : 10 Time: 18 18 time = 1.800e+01, : Convergence: : 10 Time: 19 19 time = 1.900e+01, : Convergence: : 10 Time: 20 20 time = 2.000e+01, : Convergence: : 10 Time: 21 21 time = 2.100e+01, : Convergence: : 10 Time: 22 22 time = 2.200e+01, : Convergence: : 10 Time: 23 23 time = 2.300e+01, : Convergence: : 10 Time: 24 24 time = 2.400e+01, : Convergence: : 10 Time: 25 25 time = 2.500e+01, : Convergence: : 10 Time: 26 26 time = 2.600e+01, : Convergence: : 10 Time: 27 27 time = 2.700e+01, : Convergence: : 10 Time: 28 28 time = 2.800e+01, : Convergence: : 10 Time: 29 29 time = 2.900e+01, : Convergence: : 10 Time: 30 30 time = 3.000e+01, : Convergence: : 10 Time: 31 31 time = 3.100e+01, : Convergence: : 10 Time: 32 32 time = 3.200e+01, : Convergence: : 10 Time: 33 33 time = 3.300e+01, : Convergence: : 10 Time: 34 34 time = 3.400e+01, : Convergence: : 10 Time: 35 35 time = 3.500e+01, : Convergence: : 10 Time: 36 36 time = 3.600e+01, : Convergence: : 10 Time: 37 37 time = 3.700e+01, : Convergence: : 10 Time: 38 38 time = 3.800e+01, : Convergence: : 10 Time: 39 39 time = 3.900e+01, : Convergence: : 10 Time: 40 40 time = 4.000e+01, : Convergence: : 10 Time: 41 41 time = 4.100e+01, : Convergence: : 10 Time: 42 42 time = 4.200e+01, : Convergence: : 10 Time: 43 43 time = 4.300e+01, : Convergence: : 10 Time: 44 44 time = 4.400e+01, : Convergence: : 10 Time: 45 45 time = 4.500e+01, : Convergence: : 10 Time: 46 46 time = 4.600e+01, : Convergence: : 10 Time: 47 47 time = 4.700e+01, : Convergence: : 10 Time: 48 48 time = 4.800e+01, : Convergence: : 10 Time: 49 49 time = 4.900e+01, : Convergence: : 10 Time: 50 50 time = 5.000e+01, : Convergence: : 10 Time: 51 51 time = 5.100e+01, : Convergence: : 10 Time: 52 52 time = 5.200e+01, : Convergence: : 10 Time: 53 53 time = 5.300e+01, : Convergence: : 10 Time: 54 54 time = 5.400e+01, : Convergence: : 10 Time: 55 55 time = 5.500e+01, : Convergence: : 10 Time: 56 56 time = 5.600e+01, : Convergence: : 10 Time: 57 57 time = 5.700e+01, : Convergence: : 10 Time: 58 58 time = 5.800e+01, : Convergence: : 10 Time: 59 59 time = 5.900e+01, : Convergence: : 10 Time: 60 60 time = 6.000e+01, : Convergence: : 10 Time: 61 61 time = 6.100e+01, : Convergence: : 10 Time: 62 62 time = 6.200e+01, : Convergence: : 10 Time: 63 63 time = 6.300e+01, : Convergence: : 10 Time: 64 64 time = 6.400e+01, : Convergence: : 10 Time: 65 65 time = 6.500e+01, : Convergence: : 10 Time: 66 66 time = 6.600e+01, : Convergence: : 10 Time: 67 67 time = 6.700e+01, : Convergence: : 10 Time: 68 68 time = 6.800e+01, : Convergence: : 10 Time: 69 69 time = 6.900e+01, : Convergence: : 10 Time: 70 70 time = 7.000e+01, : Convergence: : 10 Time: 71 71 time = 7.100e+01, : Convergence: : 10 Time: 72 72 time = 7.200e+01, : Convergence: : 10 Time: 73 73 time = 7.300e+01, : Convergence: : 10 Time: 74 74 time = 7.400e+01, : Convergence: : 10 Time: 75 75 time = 7.500e+01, : Convergence: : 10 Time: 76 76 time = 7.600e+01, : Convergence: : 10 Time: 77 77 time = 7.700e+01, : Convergence: : 10 Time: 78 78 time = 7.800e+01, : Convergence: : 10 Time: 79 79 time = 7.900e+01, : Convergence: : 10 Time: 80 80 time = 8.000e+01, : Convergence: : 10 Time: 81 81 time = 8.100e+01, : Convergence: : 10 Time: 82 82 time = 8.200e+01, : Convergence: : 10 Time: 83 83 time = 8.300e+01, : Convergence: : 10 Time: 84 84 time = 8.400e+01, : Convergence: : 10 Time: 85 85 time = 8.500e+01, : Convergence: : 10 Time: 86 86 time = 8.600e+01, : Convergence: : 10 Time: 87 87 time = 8.700e+01, : Convergence: : 10 Time: 88 88 time = 8.800e+01, : Convergence: : 10 Time: 89 89 time = 8.900e+01, : Convergence: : 10 Time: 90 90 time = 9.000e+01, : Convergence: : 10 735 errors and warnings output to error logs 1109 errors and warnings output to error logs ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./lbm3dp_prof on a linux-mpi named ptb341 with 4 processors, by irfan Tue Nov 24 19:05:42 2009 Using Petsc Release Version 3.0.0, Patch 7, Mon Jul 6 11:33:34 CDT 2009 Max Max/Min Avg Total Time (sec): 2.540e+01 1.00159 2.538e+01 Objects: 1.560e+02 7.80000 8.800e+01 Flops: 4.104e+08 0.00000 2.002e+08 8.009e+08 Flops/sec: 1.616e+07 0.00000 7.883e+06 3.153e+07 Memory: 7.400e+06 34.59875 1.513e+07 MPI Messages: 1.834e+03 3.34763 1.191e+03 4.765e+03 MPI Message Lengths: 3.556e+06 1.04830 2.918e+03 1.390e+07 MPI Reductions: 4.554e+03 113.85000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.3739e+00 25.1% 1.1073e+07 1.4% 1.030e+02 2.2% 1.424e+02 4.9% 1.455e+02 363.8% 1: Fluid: 1.2538e+01 49.4% 0.0000e+00 0.0% 1.080e+03 22.7% 1.466e+03 50.2% 0.000e+00 0.0% 2: Solid: 6.4682e+00 25.5% 7.8985e+08 98.6% 3.582e+03 75.2% 1.309e+03 44.9% 2.102e+03 5253.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run config/configure.py # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage AOPetscToApplication 1 0.0 2.1615e-02 0.0 0.00e+00 0.0 5.0e+00 3.1e+04 0.0e+00 0 0 0 1 0 0 0 5 23 0 0 AOApplicationToPetsc 1 0.0 1.7262e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 2 0 0 0 0 1 0 DAGlobalToLocal 1 0.0 4.8969e-03 0.0 0.00e+00 0.0 6.0e+00 5.8e+04 5.5e+00 0 0 0 3 14 0 0 6 52 4 0 DALocalToGlobal 1 0.0 2.0885e-04 0.0 0.00e+00 0.0 1.0e+00 2.0e+02 0.0e+00 0 0 0 0 0 0 0 1 0 0 0 DALocalADFunc 1 0.0 4.2467e-02 0.0 0.00e+00 0.0 1.2e+01 2.6e+03 9.0e+00 0 0 0 0 22 0 0 12 5 6 0 VecView 1 0.0 4.9114e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMin 8 0.0 3.8359e-03 0.0 1.48e+06 0.0 1.6e+01 8.5e+02 0.0e+00 0 0 0 0 0 0 27 16 2 0 769 VecTDot 9 0.0 3.5830e-03 0.0 1.66e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 30 0 0 0 920 VecSet 1 0.0 8.1062e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecWAXPY 1 0.0 8.3020e-03 0.0 2.28e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 39 0 0 0 526 VecAssemblyEnd 1 0.0 1.4091e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 4 0 0 0 0 1 0 VecScatterBegin 12 0.0 1.0369e-01 0.0 0.00e+00 0.0 6.0e+00 1.3e+04 9.0e+00 0 0 0 1 22 0 0 6 11 6 0 VecScatterEnd 12 0.0 2.0105e-02 0.0 0.00e+00 0.0 1.6e+01 2.2e+02 3.2e+01 0 0 0 0 79 0 0 16 1 22 0 VecReduceBarrier 11985 0.0 2.6606e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecReduceComm 1 0.0 4.0531e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1 0.0 8.4496e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 5 0 0 0 0 1 0 1 0.0 1.6003e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e-01 0 0 0 0 1 0 0 0 0 0 0 2 0.0 1.9150e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 16 0.0 2.3031e-04 0.0 7.67e+04 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 20 0 1 0 0 5 666 9 0.0 2.7657e-04 0.0 4.31e+04 0.0 0.0e+00 0.0e+00 4.5e+00 0 0 0 0 11 0 1 0 0 3 312 2 0.0 2.5034e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 22 0.0 1.0037e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 16 0.0 1.8239e-04 0.0 7.67e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 841 7 0.0 9.5844e-05 0.0 3.36e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 700 15 0.0 4.4250e-04 0.0 0.00e+00 0.0 2.6e+01 1.0e+04 0.0e+00 0 0 1 2 0 0 0 25 40 0 0 15 0.0 9.5034e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 2 0.0 1.6999e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1 0.0 1.9301e-02 0.0 5.65e+06 0.0 1.6e+01 8.5e+02 2.1e+01 0 1 0 0 52 0100 16 2 14 574 2 0.0 1.0994e-02 0.0 2.28e+06 0.0 0.0e+00 0.0e+00 4.5e+00 0 1 0 0 11 0 39 0 0 3 397 1 0.0 1.0787e-02 0.0 2.28e+06 0.0 0.0e+00 0.0e+00 3.5e+00 0 1 0 0 9 0 39 0 0 2 405 9 0.0 3.8106e-03 0.0 1.66e+06 0.0 0.0e+00 0.0e+00 4.5e+00 0 0 0 0 11 0 30 0 0 3 865 --- Event Stage 1: Fluid VecScatterBegin 360 0.0 4.3619e-03 0.0 0.00e+00 0.0 1.1e+03 6.5e+03 0.0e+00 0 0 23 50 0 0 0100100 0 0 VecScatterEnd 360 0.0 1.3239e+01 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 26 0 0 0 0 53 0 0 0 0 0 --- Event Stage 2: Solid VecMin 1071 0.0 6.2610e-01 0.0 1.95e+08 0.0 2.1e+03 8.5e+02 0.0e+00 1 48 45 13 0 5 48 60 29 0 608 VecTDot 981 0.0 4.3788e-01 0.0 1.77e+08 0.0 0.0e+00 0.0e+00 0.0e+00 1 43 0 0 0 3 44 0 0 0 785 VecWAXPY 1 0.0 9.1310e-03 0.0 2.34e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 469 VecAssemblyEnd 1 0.0 1.0579e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+00 0 0 0 0 4 0 0 0 0 0 0 VecReduceComm 1 0.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1 0.0 1.2941e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 5 0 0 0 0 0 0 1782 0.0 5.4054e-02 0.0 8.54e+06 0.0 0.0e+00 0.0e+00 8.9e+02 0 2 0 02228 0 2 0 0 42 316 981 0.0 7.8067e-02 0.0 4.70e+06 0.0 0.0e+00 0.0e+00 4.9e+02 0 1 0 01226 1 1 0 0 23 120 360 0.0 3.2594e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1431 0.0 5.8234e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 2862 0.0 3.4366e-02 0.0 1.35e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 785 801 0.0 1.0798e-02 0.0 3.84e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 711 270 0.0 4.1471e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 2061 0.0 1.3270e-02 0.0 0.00e+00 0.0 3.5e+03 1.8e+03 0.0e+00 0 0 73 45 0 0 0 97100 0 0 2061 0.0 6.5395e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 13 0 0 0 0 50 0 0 0 0 0 2 0.0 3.3593e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 90 0.0 1.0660e+00 0.0 3.66e+08 0.0 1.8e+03 8.5e+02 1.9e+03 2 89 37 114801 8 90 50 24 91 669 2 0.0 1.1409e-02 0.0 2.34e+06 0.0 0.0e+00 0.0e+00 4.5e+00 0 1 0 0 11 0 1 0 0 0 375 90 0.0 1.1321e-02 0.0 2.34e+06 0.0 0.0e+00 0.0e+00 3.5e+00 0 1 0 0 9 0 1 0 0 0 378 981 0.0 4.6253e-01 0.0 1.77e+08 0.0 0.0e+00 0.0e+00 4.9e+02 1 43 0 01226 3 44 0 0 23 743 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. --- Event Stage 0: Main Stage Application Order 2 0 0 0 Vec 6 0 0 0 Vec Scatter 3 0 0 0 Index Set 6 6 22068 0 IS L to G Mapping 3 0 0 0 --- Event Stage 1: Fluid Application Order 0 2 11264 0 Vec 0 6 54072 0 Vec Scatter 0 3 2604 0 IS L to G Mapping 0 3 24432 0 --- Event Stage 2: Solid ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 2.6226e-06 Average time for zero size MPI_Send(): 3.75509e-06 #PETSc Option Table entries: -ksp_type cg -log_summary -mat_partitioning_type parmetis #End o PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 Configure run at: Tue Nov 17 12:30:02 2009 Configure options: --download-c-blas-lapack=1 --with-mpi-dir=/usr/local/mpich2.12 --download-parmetis=1 --with-shared=0 ----------------------------------------- Libraries compiled on Tue Nov 17 12:31:07 EST 2009 on ptb341 Machine characteristics: Linux ptb341 2.6.24-25-generic #1 SMP Tue Oct 20 06:49:12 UTC 2009 x86_64 GNU/Linux Using PETSc directory: /home/irfan/downloads/petsc-3.0.0-p7 Using PETSc arch: linux-mpich2-debug ----------------------------------------- Using C compiler: /usr/local/mpich2.12/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Using Fortran compiler: ----------------------------------------- Using include paths: -I/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/include -I/home/irfan/downloads/petsc-3.0.0-p7/include -I/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/include -I/usr/local/mpich2.12/include ------------------------------------------ Using C linker: /usr/local/mpich2.12/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Using Fortran linker: Using libraries: -Wl,-rpath,/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/lib -L/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/lib -lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/lib -L/home/irfan/downloads/petsc-3.0.0-p7/linux-mpich2-debug/lib -lf2clapack -lf2cblas -lm -lparmetis -lmetis -lnsl -lrt -lm -ldl ------------------------------------------ WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-ksp_type value: cg Option left: name:-mat_partitioning_type value: parmetis calculate completed normally.calculate completed normally.calculate completed normally. calculate completed normally.