<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>I'm no expert on Sieve but I would probably rerun these simulations with the code compiled without<div>'-g' which is the debugging option. See the following:</div><div><br><div><div><blockquote type="cite"><div><div><br></div><div> ##########################################################</div><div> # #</div><div> # WARNING!!! #</div><div> # #</div><div> # This code was compiled with a debugging option, #</div><div> # To get timing results run config/configure.py #</div><div> # using --with-debugging=no, the performance will #</div><div> # be generally two or three times faster. #</div><div> # #</div><div> ##########################################################</div></div></blockquote><div><br></div><div>From my experience it is good to run with no debugging to get a true timing of the simulation and</div><div>much better than just trying to estimate that no debugging will be 2-3x faster.</div><div><br></div><div>Travis</div><br><blockquote type="cite"><div><div><br></div><div><br></div><div>Event Count Time (sec) Flops --- Global --- --- Stage --- Total</div><div> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>--- Event Stage 0: Main Stage</div><div><br></div><div>VecMDot 3195 1.0 8.1583e-01 2.7 2.23e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 7 0 0 8 0 7 0 0 8 1480</div><div>VecNorm 4335 1.0 1.2828e+00 2.0 1.13e+07 1.4 0.0e+00 0.0e+00 7.8e+02 0 4 0 0 18 0 4 0 0 18 475</div><div>VecScale 4192 1.0 2.3778e-02 1.3 5.47e+06 1.4 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 12415</div><div>VecCopy 995 1.0 5.0113e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecSet 2986 1.0 6.1189e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecAXPY 1140 1.0 8.8003e-03 1.3 2.78e+06 1.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 17310</div><div>VecAYPX 71 1.0 8.8720e-04 1.7 8.09e+04 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5286</div><div>VecWAXPY 2 1.0 2.3029e-05 1.7 2.28e+03 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5736</div><div>VecMAXPY 4192 1.0 5.6833e-02 1.4 3.08e+07 1.4 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 29299</div><div>VecAssemblyBegin 3 1.0 5.8301e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecAssemblyEnd 3 1.0 1.3023e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecScatterBegin 2279 1.0 5.6063e-02 3.6 0.00e+00 0.0 5.1e+05 3.7e+02 0.0e+00 0 0 96 88 0 0 0 96 88 0 0</div><div>VecScatterEnd 2279 1.0 4.8437e-0122.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecNormalize 4118 1.0 8.3514e-01 1.9 1.61e+07 1.4 0.0e+00 0.0e+00 5.7e+02 0 5 0 0 13 0 5 0 0 13 1043</div><div>MatMult 3482 1.0 1.0133e+00 2.1 1.18e+08 1.4 2.1e+05 2.8e+02 0.0e+00 0 37 39 27 0 0 37 39 27 0 6271</div><div>MatMultAdd 71 1.0 9.5340e-02 3.9 1.04e+06 1.3 2.3e+04 1.8e+02 0.0e+00 0 0 4 2 0 0 0 4 2 0 611</div><div>MatMultTranspose 142 1.0 2.2453e-01 1.6 2.09e+06 1.3 4.6e+04 1.8e+02 2.8e+02 0 1 9 4 7 0 1 9 4 7 519</div><div>MatSolve 3550 1.0 5.7862e-01 1.4 1.26e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 39 0 0 0 0 39 0 0 0 11693</div><div>MatLUFactorNum 2 1.0 4.7321e-03 1.5 3.25e+05 1.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3655</div><div>MatILUFactorSym 2 1.0 1.1258e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatAssemblyBegin 5 1.0 1.6813e-0120.3 0.00e+00 0.0 1.4e+03 2.3e+03 6.0e+00 0 0 0 2 0 0 0 0 2 0 0</div><div>MatAssemblyEnd 5 1.0 2.3137e-02 1.3 0.00e+00 0.0 1.9e+03 6.1e+01 2.8e+01 0 0 0 0 1 0 0 0 0 1 0</div><div>MatGetRowIJ 2 1.0 4.9662e-06 4.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatGetSubMatrice 2 1.0 2.5637e-0132.3 0.00e+00 0.0 3.2e+03 2.1e+03 1.0e+01 0 0 1 3 0 0 0 1 3 0 0</div><div>MatGetOrdering 2 1.0 1.2449e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatIncreaseOvrlp 2 1.0 9.9950e-03 1.1 0.00e+00 0.0 1.3e+03 1.8e+02 6.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatZeroEntries 2 1.0 8.5980e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MeshView 4 1.0 5.9615e+00 1.0 0.00e+00 0.0 1.8e+03 3.1e+03 0.0e+00 2 0 0 3 0 2 0 0 3 0 0</div><div>MeshGetGlobalScatter 3 1.0 4.1654e-02 1.2 0.00e+00 0.0 9.7e+02 6.0e+01 1.8e+01 0 0 0 0 0 0 0 0 0 0 0</div><div>MeshAssembleMatrix 1606 1.1 6.7121e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MeshUpdateOperator 2168 1.1 2.7389e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>SectionRealView 2 1.0 5.9061e-01199.5 0.00e+00 0.0 2.5e+02 4.1e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>PCSetUp 5 1.0 2.8859e-01 7.4 3.25e+05 1.5 5.8e+03 1.2e+03 4.6e+01 0 0 1 3 1 0 0 1 3 1 60</div><div>PCSetUpOnBlocks 284 1.0 8.3234e-03 1.3 3.25e+05 1.5 0.0e+00 0.0e+00 2.6e+01 0 0 0 0 1 0 0 0 0 1 2078</div><div>PCApply 71 1.0 4.8040e+00 1.0 3.13e+08 1.4 4.8e+05 3.8e+02 4.0e+03 1 97 91 84 92 1 97 91 84 94 3503</div><div>KSPGMRESOrthog 3195 1.0 8.5857e-01 2.5 4.46e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 14 0 0 8 0 14 0 0 8 2814</div><div>KSPSetup 6 1.0 2.9785e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0</div><div>KSPSolve 1 1.0 5.0004e+00 1.0 3.20e+08 1.4 5.1e+05 3.7e+02 4.2e+03 1100 96 87 95 1100 96 87 97 3449</div><div>MeshDestroy 5 1.0 3.1958e-011357.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>DistributeMesh 1 1.0 4.5183e+00 1.1 0.00e+00 0.0 5.0e+02 9.5e+03 0.0e+00 1 0 0 2 0 1 0 0 2 0 0</div><div>PartitionCreate 2 1.0 3.5427e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>PartitionClosure 2 1.0 1.2162e+0011594.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>DistributeCoords 2 1.0 8.2849e-01 2.8 0.00e+00 0.0 5.0e+02 3.0e+03 0.0e+00 0 0 0 1 0 0 0 0 1 0 0</div><div>DistributeLabels 2 1.0 1.6425e+00 1.0 0.00e+00 0.0 3.8e+02 6.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>CreateOverlap 2 1.0 1.2166e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 1 0 0 0 0 1 0 0</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>Memory usage is given in bytes:</div><div><br></div><div>Object Type Creations Destructions Memory Descendants' Mem.</div><div>Reports information only for process 0.</div><div><br></div><div>--- Event Stage 0: Main Stage</div><div><br></div><div> Viewer 4 4 1344 0</div><div> Index Set 29 29 89664 0</div><div> Vec 132 131 1098884 0</div><div> Vec Scatter 8 8 4320 0</div><div> Matrix 13 13 1315884 0</div><div> Mesh 5 5 1680 0</div><div> SectionReal 7 5 1320 0</div><div> Preconditioner 7 7 3132 0</div><div> Krylov Solver 7 7 88364 0</div><div>========================================================================================================================</div><div>Average time to get PetscTime(): 1.90735e-07</div><div>Average time for MPI_Barrier(): 0.000211</div><div>Average time for zero size MPI_Send(): 1.4998e-05</div><div>#PETSc Option Table entries:</div><div>-log_summary</div><div>#End of PETSc Option Table entries</div><div>Compiled without FORTRAN kernels</div><div>Compiled with full precision matrices (default)</div><div>sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4 sizeof(PetscScalar) 8</div><div>Configure run at: Wed Mar 9 20:22:08 2011</div><div>Configure options: --with-clanguage=cxx --with-shared=1 --with-dynamic=1 --download-f-blas-lapack=1 --with-mpi-dir=/bwfs/software/ictce3.2/impi/3.2.0.011 --download-boost=1 --download-fiat=1 --download-generator=1 --download-triangle=1 --download-tetgen=1 --download-chaco=1 --download-parmetis=1 --download-zoltan=1 --with-sieve=1 --with-opt-sieve=1 --with-exodusii-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75 --with-netcdf-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1</div><div>-----------------------------------------</div><div>Libraries compiled on Wed Mar 9 20:22:27 CST 2011 on console </div><div>Machine characteristics: Linux console 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux </div><div>Using PETSc directory: /bwfs/home/fdkong/petsc/petsc-3.1-p7</div><div>Using PETSc arch: linux-gnu-c-debug</div><div>-----------------------------------------</div><div>Using C compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC </div><div>Using Fortran compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g </div><div>-----------------------------------------</div><div>Using include paths: -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/export/ictce3.2/impi/3.2.0.011/include/gfortran/4.1.0 -I/export/ictce3.2/impi/3.2.0.011/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include/sieve -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/Boost/ -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/include -I/bwfs/software/ictce3.2/impi/3.2.0.011/include </div><div>------------------------------------------</div><div>Using C linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g </div><div>Using Fortran linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g </div><div>Using libraries: -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lzoltan -ltriangle -lX11 -lchaco -lparmetis -lmetis -ltetgen -lflapack -lfblas -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -lexoIIv2for -lexoIIv2c -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -lnetcdf -Wl,-rpath,/export/ictce3.2/impi/3.2.0.011/lib -L/export/ictce3.2/impi/3.2.0.011/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/-Xlinker -lmpi_dbg -lgfortran -lm -Wl,-rpath,/opt/intel/mpi-rt/3.2 -lm -lmpigc4 -lmpi_dbg -lstdc++ -lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -ldl </div><div>------------------------------------------</div></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "><br></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "><br></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "><br></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "><sign signid="2"><div style="color:#909090;font-family:Arial Narrow;font-size:12px">------------------</div><div style="font-size:14px;font-family:Verdana;color:#000;"><div>Fande Kong</div><div>ShenZhen Institutes of Advanced Technology</div><div>Chinese Academy of Sciences</div>
</div></sign></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "> </div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; "><includetail><div> </div><div> </div><div style="font:Verdana normal 14px;color:#000;"><div style="FONT-SIZE: 12px;FONT-FAMILY: Arial Narrow;padding:2px 0 2px 0;">------------------ Original ------------------</div><div style="FONT-SIZE: 12px;background:#efefef;padding:8px;"><div id="menu_sender"><b>From: </b> "knepley"<<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>>;</div><div><b>Date: </b> Mon, Mar 28, 2011 02:19 PM</div><div><b>To: </b> "PETSc users list"<<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>; <wbr></div><div><b>Cc: </b> "fdkong"<<a href="mailto:fd.kong@siat.ac.cn">fd.kong@siat.ac.cn</a>>; <wbr></div><div><b>Subject: </b> Re: [petsc-users] Generation, refinement of the mesh (Sieve mesh) isvery slow!</div></div><div> </div>
<div dir="ltr">1) Always send the output of -log_summary when asking a performance question<div><br></div><div>2) There are implementations that are optimized for different things. Its possible to</div><div>�1�7�1�7 �1�7optimize mesh handling for a cells-vertices mesh, but not if you need edges and</div>
<div>�1�7�1�7 �1�7faces generated.</div><div><br></div><div>3) I am out of the country. I can look at the performance when I get back.</div><div><br></div><div>�1�7�1�7 Matt</div><div><br><div class="gmail_quote">On Mon, Mar 28, 2011 at 1:06 AM, fdkong <span dir="ltr"><<a href="mailto:fd.kong@siat.ac.cn">fd.kong@siat.ac.cn</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="color:rgb(0, 0, 0);font-family:Verdana;font-size:14px">Hi everyone</div><div style="color:rgb(0, 0, 0);font-family:Verdana;font-size:14px">
�1�7�1�7 I have developed my application based on the sieve mesh object in the Pestc. And now,�1�7I encountered some �1�7serious problems.�1�7</div><div style="color:rgb(0, 0, 0);font-family:Verdana;font-size:14px">�1�71. The generation of mesh�1�7takes a lot of time, run very slowly. The following code is used:</div>
<div>�1�7�1�7�1�7�1�7�1�7 �1�7double lower[2] = {-1.0, -1.0};</div><div>�1�7�1�7 �1�7 �1�7 �1�7double upper[2] = {1.0, 1.0};</div><div>�1�7�1�7 �1�7 �1�7 �1�7int �1�7 �1�7edges[2] = {256,256};</div><div>�1�7�1�7 �1�7 �1�7 �1�7mB = ALE::MeshBuilder<ALE::Mesh>::createSquareBoundary(comm, lower, upper, edges, debug);</div>
<div><div>�1�7�1�7ALE::ISieveConverter::convertMesh(*mB, *meshBd, renumbering, false);</div><div>�1�7�1�7 �1�7 �1�7ierr = PetscPrintf(PETSC_COMM_WORLD," End build convertMesh �1�7\n");CHKERRQ(ierr);</div><div>�1�7�1�7 �1�7 �1�7ierr = MeshSetMesh(boundary, meshBd);CHKERRQ(ierr);</div>
<div>�1�7�1�7 �1�7 �1�7ierr = PetscPrintf(PETSC_COMM_WORLD," Begin build MeshGenerate �1�7\n");CHKERRQ(ierr);</div><div><br></div><div>�1�7�1�7 �1�7 �1�7ierr = MeshGenerate(boundary,interpolate, &mesh);CHKERRQ(ierr);</div></div><div>�1�7�1�7�1�7</div>
<div>�1�7�1�7 2. The refinement of mesh is also very slow. Th code:</div><div>�1�7�1�7 �1�7�1�7refinementLimit=0.0001;</div><div>�1�7�1�7 �1�7if (refinementLimit > 0.0)�1�7</div><div>�1�7�1�7 �1�7{</div><div>�1�7�1�7 �1�7 �1�7Mesh refinedMesh;</div><div><br></div><div>
�1�7�1�7 �1�7 �1�7ierr = MeshRefine(mesh, refinementLimit,interpolate, &refinedMesh);CHKERRQ(ierr);</div><div>�1�7�1�7 �1�7 �1�7ierr = MeshDestroy(mesh);CHKERRQ(ierr);</div><div>�1�7�1�7 �1�7 �1�7mesh = refinedMesh;</div><div>�1�7�1�7 �1�7}</div><div><br></div><div>
�1�7�1�7 �1�73. The distribution of mesh is also very slow. The code:</div><div>�1�7�1�7 �1�7�1�7if (size > 1)�1�7</div><div>�1�7�1�7 �1�7{</div><div>�1�7�1�7 �1�7 �1�7Mesh parallelMesh;</div><div><br></div><div>�1�7�1�7 �1�7 �1�7//ierr = DistributeMeshnew(mesh, "chao", &parallelMesh);CHKERRQ(ierr);</div>
<div>�1�7�1�7 �1�7 �1�7ierr = DistributeMeshnew(mesh, "parmetis", &parallelMesh);CHKERRQ(ierr);</div><div>�1�7�1�7 �1�7 �1�7ierr = MeshDestroy(mesh);CHKERRQ(ierr);</div><div>�1�7�1�7 �1�7 �1�7mesh = parallelMesh;</div><div>�1�7�1�7 �1�7}.</div><div>�1�7�1�7�1�7</div>
<div>�1�7�1�7 �1�7Does anyone encounter these similar problem? If anyone can help, thank you very much! �1�7</div><div>�1�7�1�7</div><div>�1�7�1�7 And I wonder to consult which parallel mesh can work �1�7with Petsc very well, when we develop some complex problem?�1�7</div>
<div>�1�7�1�7�1�7</div><div>�1�7�1�7 �1�7�1�7</div><div style="font-size:14px;font-family:Verdana;color:rgb(0, 0, 0)"><div style="color:#909090;font-family:Arial Narrow;font-size:12px">------------------</div><div style="font-size:14px;font-family:Verdana;color:#000">
<div>Fande Kong</div><div>ShenZhen Institutes of Advanced Technology</div><div>Chinese Academy of Sciences</div>
</div></div><div style="color:rgb(0, 0, 0);font-family:Verdana;font-size:14px">�1�7</div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</div></div>
</div></includetail></div></blockquote></div><br></div></div></body></html>