On Mon, Apr 25, 2011 at 9:30 AM, Eugene Kozlov <span dir="ltr"><<a href="mailto:neoveneficus@gmail.com">neoveneficus@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I never use the maillist before this. How can I better show logs?<br>
Attach file to mail?</blockquote><div><br></div><div>1) You attached 3 runs with 1 processor</div><div><br></div><div>2) This matrix looks pretty small</div><div><br></div><div>3) You are still getting 2 GF/s</div><div><br>
</div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">
Norm of error 0.00012322 iterations 448<br>
Norm of error 0.00012322 iterations 448<br>
Norm of error 0.00012322 iterations 448<br>
</div>************************************************************************************************************************<br>
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r<br>
-fCourier9' to print this document ***<br>
************************************************************************************************************************<br>
<br>
---------------------------------------------- PETSc Performance<br>
Summary: ----------------------------------------------<br>
<br>
ex2 on a arch-linu named cn11 with 1 processor, by kukushkinav Mon Apr<br>
25 18:24:15 2011<br>
Using Petsc Development HG revision:<br>
d3e10315d68b1dd5481adb2889c7d354880da362 HG Date: Wed Apr 20 21:03:56<br>
2011 -0500<br>
<br>
Max Max/Min Avg Total<br>
Time (sec): 3.891e+01 1.00000 3.891e+01<br>
Objects: 2.500e+01 1.00000 2.500e+01<br>
Flops: 9.474e+09 1.00000 9.474e+09 9.474e+09<br>
Flops/sec: 2.435e+08 1.00000 2.435e+08 2.435e+08<br>
Memory: 1.562e+08 1.00000 1.562e+08<br>
MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Reductions: 5.881e+03 1.00000<br>
<br>
Flop counting convention: 1 flop = 1 real number operation of type<br>
(multiply/divide/add/subtract)<br>
e.g., VecAXPY() for real vectors of length<br>
N --> 2N flops<br>
and VecAXPY() for complex vectors of<br>
length N --> 8N flops<br>
<br>
Summary of Stages: ----- Time ------ ----- Flops ----- ---<br>
Messages --- -- Message Lengths -- -- Reductions --<br>
Avg %Total Avg %Total counts<br>
%Total Avg %Total counts %Total<br>
0: Main Stage: 3.8368e+01 98.6% 9.4739e+09 100.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 5.855e+03 99.6%<br>
1: Assembly: 5.3823e-01 1.4% 0.0000e+00 0.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 1.200e+01 0.2%<br>
<br>
------------------------------------------------------------------------------------------------------------------------<br>
See the 'Profiling' chapter of the users' manual for details on<br>
interpreting output.<br>
Phase summary info:<br>
Count: number of times phase was executed<br>
Time and Flops: Max - maximum over all processors<br>
Ratio - ratio of maximum to minimum over all processors<br>
Mess: number of messages sent<br>
Avg. len: average message length<br>
Reduct: number of global reductions<br>
Global: entire computation<br>
Stage: stages of a computation. Set stages with PetscLogStagePush()<br>
and PetscLogStagePop().<br>
%T - percent time in this phase %F - percent flops in this phase<br>
%M - percent messages in this phase %L - percent message<br>
lengths in this phase<br>
%R - percent reductions in this phase<br>
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time<br>
over all processors)<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option, #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
Event Count Time (sec) Flops<br>
--- Global --- --- Stage --- Total<br>
Max Ratio Max Ratio Max Ratio Mess Avg<br>
len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
MatMult 449 1.0 1.4191e+00 1.0 2.87e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 4 30 0 0 0 4 30 0 0 0 2023<br>
MatSolve 449 1.0 1.3127e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 34 27 0 0 0 34 27 0 0 0 197<br>
MatCholFctrNum 1 1.0 1.7007e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4<br>
MatICCFactorSym 1 1.0 1.1560e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetRowIJ 1 1.0 1.1921e-06 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetOrdering 1 1.0 1.0204e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatCUSPCopyTo 2 1.0 2.0546e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecDot 896 1.0 1.0086e+00 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 3 12 0 0 30 3 12 0 0 31 1137<br>
VecNorm 450 1.0 8.4599e-01 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 9.0e+02 2 12 0 0 15 2 12 0 0 15 1362<br>
VecCopy 2 1.0 2.6081e-03 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecSet 451 1.0 2.7629e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
VecAXPY 897 1.0 2.1131e-01 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 1 12 0 0 0 1 12 0 0 0 5434<br>
VecAYPX 447 1.0 1.0049e-01 1.0 5.72e+08 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 5694<br>
VecScatterBegin 449 1.0 6.8694e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
VecCUSPCopyTo 1346 1.0 1.2865e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0<br>
VecCUSPCopyFrom 1346 1.0 2.2437e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0<br>
KSPSetup 2 1.0 3.1233e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
KSPSolve 1 1.0 1.8947e+01 1.0 9.46e+09 1.0 0.0e+00<br>
0.0e+00 5.8e+03 49100 0 0 99 49100 0 0100 499<br>
PCSetUp 2 1.0 3.8846e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 7.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCSetUpOnBlocks 1 1.0 3.8828e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 5.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCApply 449 1.0 1.4987e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 39 27 0 0 31 39 27 0 0 31 172<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
MatAssemblyBegin 1 1.0 2.3842e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 17 0<br>
MatAssemblyEnd 1 1.0 7.0319e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+01 0 0 0 0 0 13 0 0 0 83 0<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
Memory usage is given in bytes:<br>
<br>
Object Type Creations Destructions Memory Descendants' Mem.<br>
Reports information only for process 0.<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
Matrix 4 4 120310620 0<br>
Vec 8 9 20493272 0<br>
Vec Scatter 0 1 596 0<br>
Index Set 3 3 2562160 0<br>
Krylov Solver 2 2 2048 0<br>
Preconditioner 2 2 1688 0<br>
Viewer 1 0 0 0<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
Vec 2 1 1496 0<br>
Vec Scatter 1 0 0 0<br>
Index Set 2 2 1432 0<br>
========================================================================================================================<br>
Average time to get PetscTime(): 9.53674e-08<br>
#PETSc Option Table entries:<br>
-ksp_type cg<br>
-log_summary<br>
-m 800<br>
-mat_type mpiaijcusp<br>
-n 800<br>
-vec_type mpicusp<br>
#End of PETSc Option Table entries<br>
Compiled without FORTRAN kernels<br>
Compiled with full precision matrices (default)<br>
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8<br>
sizeof(PetscScalar) 8<br>
Configure run at: Mon Apr 25 12:42:34 2011<br>
Configure options: --prefix=/home/kukushkinav<br>
--with-blas-lapack-dir=/opt/intel/composerxe-2011.0.084/mkl<br>
--with-mpi-dir=/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin" target="_blank">4.0.1.007/intel64/bin</a> --with-cuda=1<br>
--with-cusp=1 --with-thrust=1<br>
--with-thrust-dir=/home/kukushkinav/include<br>
--with-cusp-dir=/home/kukushkinav/include --with-cuda-arch=sm_13<br>
-----------------------------------------<br>
Libraries compiled on Mon Apr 25 12:42:34 2011 on manager<br>
Machine characteristics: Linux-2.6.18-238.5.1.el5-x86_64-with-redhat-5.6-Tikanga<br>
Using PETSc directory: /export/home/kukushkinav/soft/petsc-dev<br>
Using PETSc arch: arch-linux2-c-debug<br>
-----------------------------------------<br>
<br>
Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing<br>
-Wno-unknown-pragmas -g3 ${COPTFLAGS} ${CFLAGS}<br>
Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -g<br>
${FOPTFLAGS} ${FFLAGS}<br>
-----------------------------------------<br>
<br>
Using include paths:<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/usr/local/cuda/include -I/home/kukushkinav/include/<br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin/include" target="_blank">4.0.1.007/intel64/bin/include</a><br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/include" target="_blank">4.0.1.007/intel64/include</a><br>
-----------------------------------------<br>
<br>
Using C linker: mpicc<br>
Using Fortran linker: mpif90<br>
Using libraries:<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-L/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec<br>
-lpetscsys -lX11 -Wl,-rpath,/usr/local/cuda/lib64<br>
-L/usr/local/cuda/lib64 -lcufft -lcublas -lcudart<br>
-Wl,-rpath,/opt/intel/composerxe-2011.0.084/mkl<br>
-L/opt/intel/composerxe-2011.0.084/mkl -lmkl_intel_lp64<br>
-lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl<br>
-L/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-L/opt/intel/composerxe-2011.0.084/compiler/lib/intel64<br>
-L/opt/intel/composerxe-2011.0.084/mkl/lib/intel64<br>
-L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpi -lmpigf -lmpigi<br>
-lpthread -lrt -lgcc_s<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/-Xlinker -lmpi_dbg<br>
-lgfortran -lm -Wl,-rpath,/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-Wl,-rpath,/opt/intel/mpi-rt/4.0.1 -lm -lmpigc4 -lmpi_dbg -lstdc++<br>
-lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lpthread -lrt<br>
-lgcc_s -ldl<br>
-----------------------------------------<br>
<br>
************************************************************************************************************************<br>
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r<br>
-fCourier9' to print this document ***<br>
************************************************************************************************************************<br>
<br>
---------------------------------------------- PETSc Performance<br>
Summary: ----------------------------------------------<br>
<br>
ex2 on a arch-linu named cn11 with 1 processor, by kukushkinav Mon Apr<br>
25 18:24:15 2011<br>
Using Petsc Development HG revision:<br>
d3e10315d68b1dd5481adb2889c7d354880da362 HG Date: Wed Apr 20 21:03:56<br>
2011 -0500<br>
<br>
Max Max/Min Avg Total<br>
Time (sec): 3.892e+01 1.00000 3.892e+01<br>
Objects: 2.500e+01 1.00000 2.500e+01<br>
Flops: 9.474e+09 1.00000 9.474e+09 9.474e+09<br>
Flops/sec: 2.434e+08 1.00000 2.434e+08 2.434e+08<br>
Memory: 1.562e+08 1.00000 1.562e+08<br>
MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Reductions: 5.881e+03 1.00000<br>
<br>
Flop counting convention: 1 flop = 1 real number operation of type<br>
(multiply/divide/add/subtract)<br>
e.g., VecAXPY() for real vectors of length<br>
N --> 2N flops<br>
and VecAXPY() for complex vectors of<br>
length N --> 8N flops<br>
<br>
Summary of Stages: ----- Time ------ ----- Flops ----- ---<br>
Messages --- -- Message Lengths -- -- Reductions --<br>
Avg %Total Avg %Total counts<br>
%Total Avg %Total counts %Total<br>
0: Main Stage: 3.8368e+01 98.6% 9.4739e+09 100.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 5.855e+03 99.6%<br>
1: Assembly: 5.4742e-01 1.4% 0.0000e+00 0.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 1.200e+01 0.2%<br>
<br>
------------------------------------------------------------------------------------------------------------------------<br>
See the 'Profiling' chapter of the users' manual for details on<br>
interpreting output.<br>
Phase summary info:<br>
Count: number of times phase was executed<br>
Time and Flops: Max - maximum over all processors<br>
Ratio - ratio of maximum to minimum over all processors<br>
Mess: number of messages sent<br>
Avg. len: average message length<br>
Reduct: number of global reductions<br>
Global: entire computation<br>
Stage: stages of a computation. Set stages with PetscLogStagePush()<br>
and PetscLogStagePop().<br>
%T - percent time in this phase %F - percent flops in this phase<br>
%M - percent messages in this phase %L - percent message<br>
lengths in this phase<br>
%R - percent reductions in this phase<br>
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time<br>
over all processors)<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option, #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
Event Count Time (sec) Flops<br>
--- Global --- --- Stage --- Total<br>
Max Ratio Max Ratio Max Ratio Mess Avg<br>
len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
MatMult 449 1.0 1.4400e+00 1.0 2.87e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 4 30 0 0 0 4 30 0 0 0 1994<br>
MatSolve 449 1.0 1.3012e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 33 27 0 0 0 34 27 0 0 0 199<br>
MatCholFctrNum 1 1.0 1.7184e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4<br>
MatICCFactorSym 1 1.0 1.1631e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetRowIJ 1 1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetOrdering 1 1.0 1.0190e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatCUSPCopyTo 2 1.0 2.1061e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecDot 896 1.0 1.0891e+00 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 3 12 0 0 30 3 12 0 0 31 1053<br>
VecNorm 450 1.0 1.1033e+00 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 9.0e+02 3 12 0 0 15 3 12 0 0 15 1044<br>
VecCopy 2 1.0 2.7471e-03 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecSet 451 1.0 2.5600e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
VecAXPY 897 1.0 1.9726e-01 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 1 12 0 0 0 1 12 0 0 0 5821<br>
VecAYPX 447 1.0 1.0141e-01 1.0 5.72e+08 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 5642<br>
VecScatterBegin 449 1.0 6.9110e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
VecCUSPCopyTo 1346 1.0 1.5747e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0<br>
VecCUSPCopyFrom 1346 1.0 2.0444e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0<br>
KSPSetup 2 1.0 4.0770e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
KSPSolve 1 1.0 1.8957e+01 1.0 9.46e+09 1.0 0.0e+00<br>
0.0e+00 5.8e+03 49100 0 0 99 49100 0 0100 499<br>
PCSetUp 2 1.0 3.9091e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 7.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCSetUpOnBlocks 1 1.0 3.9070e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 5.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCApply 449 1.0 1.4649e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 38 27 0 0 31 38 27 0 0 31 176<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
MatAssemblyBegin 1 1.0 3.9101e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 17 0<br>
MatAssemblyEnd 1 1.0 7.0405e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+01 0 0 0 0 0 13 0 0 0 83 0<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
Memory usage is given in bytes:<br>
<br>
Object Type Creations Destructions Memory Descendants' Mem.<br>
Reports information only for process 0.<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
Matrix 4 4 120310620 0<br>
Vec 8 9 20493272 0<br>
Vec Scatter 0 1 596 0<br>
Index Set 3 3 2562160 0<br>
Krylov Solver 2 2 2048 0<br>
Preconditioner 2 2 1688 0<br>
Viewer 1 0 0 0<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
Vec 2 1 1496 0<br>
Vec Scatter 1 0 0 0<br>
Index Set 2 2 1432 0<br>
========================================================================================================================<br>
Average time to get PetscTime(): 9.53674e-08<br>
#PETSc Option Table entries:<br>
-ksp_type cg<br>
-log_summary<br>
-m 800<br>
-mat_type mpiaijcusp<br>
-n 800<br>
-vec_type mpicusp<br>
#End of PETSc Option Table entries<br>
Compiled without FORTRAN kernels<br>
Compiled with full precision matrices (default)<br>
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8<br>
sizeof(PetscScalar) 8<br>
Configure run at: Mon Apr 25 12:42:34 2011<br>
Configure options: --prefix=/home/kukushkinav<br>
--with-blas-lapack-dir=/opt/intel/composerxe-2011.0.084/mkl<br>
--with-mpi-dir=/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin" target="_blank">4.0.1.007/intel64/bin</a> --with-cuda=1<br>
--with-cusp=1 --with-thrust=1<br>
--with-thrust-dir=/home/kukushkinav/include<br>
--with-cusp-dir=/home/kukushkinav/include --with-cuda-arch=sm_13<br>
-----------------------------------------<br>
Libraries compiled on Mon Apr 25 12:42:34 2011 on manager<br>
Machine characteristics: Linux-2.6.18-238.5.1.el5-x86_64-with-redhat-5.6-Tikanga<br>
Using PETSc directory: /export/home/kukushkinav/soft/petsc-dev<br>
Using PETSc arch: arch-linux2-c-debug<br>
-----------------------------------------<br>
<br>
Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing<br>
-Wno-unknown-pragmas -g3 ${COPTFLAGS} ${CFLAGS}<br>
Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -g<br>
${FOPTFLAGS} ${FFLAGS}<br>
-----------------------------------------<br>
<br>
Using include paths:<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/usr/local/cuda/include -I/home/kukushkinav/include/<br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin/include" target="_blank">4.0.1.007/intel64/bin/include</a><br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/include" target="_blank">4.0.1.007/intel64/include</a><br>
-----------------------------------------<br>
<br>
Using C linker: mpicc<br>
Using Fortran linker: mpif90<br>
Using libraries:<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-L/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec<br>
-lpetscsys -lX11 -Wl,-rpath,/usr/local/cuda/lib64<br>
-L/usr/local/cuda/lib64 -lcufft -lcublas -lcudart<br>
-Wl,-rpath,/opt/intel/composerxe-2011.0.084/mkl<br>
-L/opt/intel/composerxe-2011.0.084/mkl -lmkl_intel_lp64<br>
-lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl<br>
-L/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-L/opt/intel/composerxe-2011.0.084/compiler/lib/intel64<br>
-L/opt/intel/composerxe-2011.0.084/mkl/lib/intel64<br>
-L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpi -lmpigf -lmpigi<br>
-lpthread -lrt -lgcc_s<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/-Xlinker -lmpi_dbg<br>
-lgfortran -lm -Wl,-rpath,/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-Wl,-rpath,/opt/intel/mpi-rt/4.0.1 -lm -lmpigc4 -lmpi_dbg -lstdc++<br>
-lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lpthread -lrt<br>
-lgcc_s -ldl<br>
-----------------------------------------<br>
<br>
************************************************************************************************************************<br>
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r<br>
-fCourier9' to print this document ***<br>
************************************************************************************************************************<br>
<br>
---------------------------------------------- PETSc Performance<br>
Summary: ----------------------------------------------<br>
<br>
ex2 on a arch-linu named cn11 with 1 processor, by kukushkinav Mon Apr<br>
25 18:24:16 2011<br>
Using Petsc Development HG revision:<br>
d3e10315d68b1dd5481adb2889c7d354880da362 HG Date: Wed Apr 20 21:03:56<br>
2011 -0500<br>
<br>
Max Max/Min Avg Total<br>
Time (sec): 3.946e+01 1.00000 3.946e+01<br>
Objects: 2.500e+01 1.00000 2.500e+01<br>
Flops: 9.474e+09 1.00000 9.474e+09 9.474e+09<br>
Flops/sec: 2.401e+08 1.00000 2.401e+08 2.401e+08<br>
Memory: 1.562e+08 1.00000 1.562e+08<br>
MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00<br>
MPI Reductions: 5.881e+03 1.00000<br>
<br>
Flop counting convention: 1 flop = 1 real number operation of type<br>
(multiply/divide/add/subtract)<br>
e.g., VecAXPY() for real vectors of length<br>
N --> 2N flops<br>
and VecAXPY() for complex vectors of<br>
length N --> 8N flops<br>
<br>
Summary of Stages: ----- Time ------ ----- Flops ----- ---<br>
Messages --- -- Message Lengths -- -- Reductions --<br>
Avg %Total Avg %Total counts<br>
%Total Avg %Total counts %Total<br>
0: Main Stage: 3.8927e+01 98.6% 9.4739e+09 100.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 5.855e+03 99.6%<br>
1: Assembly: 5.3425e-01 1.4% 0.0000e+00 0.0% 0.000e+00<br>
0.0% 0.000e+00 0.0% 1.200e+01 0.2%<br>
<br>
------------------------------------------------------------------------------------------------------------------------<br>
See the 'Profiling' chapter of the users' manual for details on<br>
interpreting output.<br>
Phase summary info:<br>
Count: number of times phase was executed<br>
Time and Flops: Max - maximum over all processors<br>
Ratio - ratio of maximum to minimum over all processors<br>
Mess: number of messages sent<br>
Avg. len: average message length<br>
Reduct: number of global reductions<br>
Global: entire computation<br>
Stage: stages of a computation. Set stages with PetscLogStagePush()<br>
and PetscLogStagePop().<br>
%T - percent time in this phase %F - percent flops in this phase<br>
%M - percent messages in this phase %L - percent message<br>
lengths in this phase<br>
%R - percent reductions in this phase<br>
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time<br>
over all processors)<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
<br>
##########################################################<br>
# #<br>
# WARNING!!! #<br>
# #<br>
# This code was compiled with a debugging option, #<br>
# To get timing results run ./configure #<br>
# using --with-debugging=no, the performance will #<br>
# be generally two or three times faster. #<br>
# #<br>
##########################################################<br>
<br>
<br>
Event Count Time (sec) Flops<br>
--- Global --- --- Stage --- Total<br>
Max Ratio Max Ratio Max Ratio Mess Avg<br>
len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
MatMult 449 1.0 1.4559e+00 1.0 2.87e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 4 30 0 0 0 4 30 0 0 0 1972<br>
MatSolve 449 1.0 1.2787e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 32 27 0 0 0 33 27 0 0 0 202<br>
MatCholFctrNum 1 1.0 1.6867e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4<br>
MatICCFactorSym 1 1.0 1.1588e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetRowIJ 1 1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatGetOrdering 1 1.0 1.0179e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
MatCUSPCopyTo 2 1.0 1.9272e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecDot 896 1.0 1.1960e+00 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 3 12 0 0 30 3 12 0 0 31 959<br>
VecNorm 450 1.0 1.0563e+00 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 9.0e+02 3 12 0 0 15 3 12 0 0 15 1091<br>
VecCopy 2 1.0 2.4669e-03 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecSet 451 1.0 2.5303e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
VecAXPY 897 1.0 1.9844e-01 1.0 1.15e+09 1.0 0.0e+00<br>
0.0e+00 0.0e+00 1 12 0 0 0 1 12 0 0 0 5786<br>
VecAYPX 447 1.0 1.0348e-01 1.0 5.72e+08 1.0 0.0e+00<br>
0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 5529<br>
VecScatterBegin 449 1.0 7.1386e-01 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
VecCUSPCopyTo 1346 1.0 1.6784e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0<br>
VecCUSPCopyFrom 1346 1.0 2.2173e+00 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0<br>
KSPSetup 2 1.0 4.5061e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
KSPSolve 1 1.0 1.8962e+01 1.0 9.46e+09 1.0 0.0e+00<br>
0.0e+00 5.8e+03 48100 0 0 99 49100 0 0100 499<br>
PCSetUp 2 1.0 3.8716e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 7.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCSetUpOnBlocks 1 1.0 3.8694e-01 1.0 6.40e+05 1.0 0.0e+00<br>
0.0e+00 5.0e+00 1 0 0 0 0 1 0 0 0 0 2<br>
PCApply 449 1.0 1.4576e+01 1.0 2.58e+09 1.0 0.0e+00<br>
0.0e+00 1.8e+03 37 27 0 0 31 37 27 0 0 31 177<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
MatAssemblyBegin 1 1.0 5.1022e-05 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 17 0<br>
MatAssemblyEnd 1 1.0 6.9999e-02 1.0 0.00e+00 0.0 0.0e+00<br>
0.0e+00 1.0e+01 0 0 0 0 0 13 0 0 0 83 0<br>
------------------------------------------------------------------------------------------------------------------------<br>
<br>
Memory usage is given in bytes:<br>
<br>
Object Type Creations Destructions Memory Descendants' Mem.<br>
Reports information only for process 0.<br>
<br>
--- Event Stage 0: Main Stage<br>
<br>
Matrix 4 4 120310620 0<br>
Vec 8 9 20493272 0<br>
Vec Scatter 0 1 596 0<br>
Index Set 3 3 2562160 0<br>
Krylov Solver 2 2 2048 0<br>
Preconditioner 2 2 1688 0<br>
Viewer 1 0 0 0<br>
<br>
--- Event Stage 1: Assembly<br>
<br>
Vec 2 1 1496 0<br>
Vec Scatter 1 0 0 0<br>
Index Set 2 2 1432 0<br>
========================================================================================================================<br>
Average time to get PetscTime(): 0<br>
#PETSc Option Table entries:<br>
-ksp_type cg<br>
-log_summary<br>
-m 800<br>
-mat_type mpiaijcusp<br>
-n 800<br>
-vec_type mpicusp<br>
#End of PETSc Option Table entries<br>
Compiled without FORTRAN kernels<br>
Compiled with full precision matrices (default)<br>
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8<br>
sizeof(PetscScalar) 8<br>
Configure run at: Mon Apr 25 12:42:34 2011<br>
Configure options: --prefix=/home/kukushkinav<br>
--with-blas-lapack-dir=/opt/intel/composerxe-2011.0.084/mkl<br>
--with-mpi-dir=/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin" target="_blank">4.0.1.007/intel64/bin</a> --with-cuda=1<br>
--with-cusp=1 --with-thrust=1<br>
--with-thrust-dir=/home/kukushkinav/include<br>
--with-cusp-dir=/home/kukushkinav/include --with-cuda-arch=sm_13<br>
-----------------------------------------<br>
Libraries compiled on Mon Apr 25 12:42:34 2011 on manager<br>
Machine characteristics: Linux-2.6.18-238.5.1.el5-x86_64-with-redhat-5.6-Tikanga<br>
Using PETSc directory: /export/home/kukushkinav/soft/petsc-dev<br>
Using PETSc arch: arch-linux2-c-debug<br>
-----------------------------------------<br>
<br>
Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing<br>
-Wno-unknown-pragmas -g3 ${COPTFLAGS} ${CFLAGS}<br>
Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -g<br>
${FOPTFLAGS} ${FFLAGS}<br>
-----------------------------------------<br>
<br>
Using include paths:<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/include<br>
-I/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/include<br>
-I/usr/local/cuda/include -I/home/kukushkinav/include/<br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/bin/include" target="_blank">4.0.1.007/intel64/bin/include</a><br>
-I/opt/intel/impi/<a href="http://4.0.1.007/intel64/include" target="_blank">4.0.1.007/intel64/include</a><br>
-----------------------------------------<br>
<br>
Using C linker: mpicc<br>
Using Fortran linker: mpif90<br>
Using libraries:<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-L/export/home/kukushkinav/soft/petsc-dev/arch-linux2-c-debug/lib<br>
-lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec<br>
-lpetscsys -lX11 -Wl,-rpath,/usr/local/cuda/lib64<br>
-L/usr/local/cuda/lib64 -lcufft -lcublas -lcudart<br>
-Wl,-rpath,/opt/intel/composerxe-2011.0.084/mkl<br>
-L/opt/intel/composerxe-2011.0.084/mkl -lmkl_intel_lp64<br>
-lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl<br>
-L/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-L/opt/intel/composerxe-2011.0.084/compiler/lib/intel64<br>
-L/opt/intel/composerxe-2011.0.084/mkl/lib/intel64<br>
-L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpi -lmpigf -lmpigi<br>
-lpthread -lrt -lgcc_s<br>
-Wl,-rpath,/export/home/kukushkinav/soft/petsc-dev/-Xlinker -lmpi_dbg<br>
-lgfortran -lm -Wl,-rpath,/opt/intel/impi/<a href="http://4.0.1.007/intel64/lib" target="_blank">4.0.1.007/intel64/lib</a><br>
-Wl,-rpath,/opt/intel/mpi-rt/4.0.1 -lm -lmpigc4 -lmpi_dbg -lstdc++<br>
-lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lpthread -lrt<br>
-lgcc_s -ldl<br>
-----------------------------------------<br>
<br>
<br>
2011/4/25 Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>>:<br>
<div><div></div><div class="h5">> On Mon, Apr 25, 2011 at 9:06 AM, Eugene Kozlov <<a href="mailto:neoveneficus@gmail.com">neoveneficus@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hello,<br>
><br>
> To answer any kind of question about performance, we need the full output of<br>
> -log_summary.<br>
> Matt<br>
><br>
>><br>
>> I am trying to test PETSc capability of solving sparse linear systems<br>
>> in parallel with GPU.<br>
>><br>
>> I compiled and tried to run example<br>
>> src/ksp/ksp/examples/tutorials/ex2.c, which can be executed in<br>
>> parallel.<br>
>><br>
>> In this example matrix and vectors created using VecSetFromOptions()<br>
>> and MatSetFromOptions().<br>
>><br>
>> According to the page<br>
>> <a href="http://www.mcs.anl.gov/petsc/petsc-2/features/gpus.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-2/features/gpus.html</a> , I execute<br>
>> the program with keys<br>
>><br>
>> -vec_type mpicusp -mat_type mpiaijcusp<br>
>><br>
>> in parallel on the different number of GPUs. Full command:<br>
>><br>
>> cleo-submit -np 1 ex2 -ksp_type cg -vec_type mpicusp -mat_type<br>
>> mpiaijcusp -m 800 -n 800<br>
>><br>
>> Where 'cleo-submit' is a batch manager utility.<br>
>><br>
>> I tested the program on 1, 2 and 3 GPUs. As a result I have output as<br>
>> (for 3 GPUs):<br>
>><br>
>> Norm of error 0.00012322 iterations 448<br>
>> Norm of error 0.00012322 iterations 448<br>
>> Norm of error 0.00012322 iterations 448<br>
>><br>
>> and run times: 30, 40 and 46 seconds respectively.<br>
>><br>
>> What can be a cause of these results?<br>
>><br>
>> --<br>
>> Best regards,<br>
>> Eugene<br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments<br>
> is infinitely more interesting than any results to which their experiments<br>
> lead.<br>
> -- Norbert Wiener<br>
><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>