<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Satish :<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I don't see any changes hzhang/fix-submat_samerowdist in past few<br>
days. So how did this error get fixed?<br></blockquote><div> </div><div>I've never seen valgrind error for ex56 from nightly tests. </div><div>Mark told me about Valgrind error on his mac, and I reproduced it on our linux machine with my own configure,</div><div>which directs to lapack. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
please do not merge yet.<br></blockquote><div>Sure.</div><div><br></div><div>Hong </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
<br>
On Fri, 10 Nov 2017, Mark Adams wrote:<br>
<br>
> I don't see a problem with this. As far as what branch to merge it into,<br>
> that is a Barry/Satish/etc decision.<br>
><br>
> On Fri, Nov 10, 2017 at 5:17 PM, Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br>
><br>
> > Mark:<br>
> ><br>
> >> I would add:<br>
> >><br>
> >> '--download-fblaslapack=1',<br>
> >><br>
> >> This is what I have on my Linux machine (cg at ANL) and it runs clean.<br>
> >><br>
> > I do not see any error report from nightly tests for ex56. Should I merge<br>
> > this branch to maint?<br>
> > Hong<br>
> ><br>
> >><br>
> >> On Fri, Nov 10, 2017 at 11:03 AM, Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br>
> >><br>
> >>> I use<br>
> >>> Using configure Options: --download-metis --download-mpich<br>
> >>> --download-mumps --download-parmetis --download-scalapack<br>
> >>> --download-superlu --download-superlu_dist --download-suitesparse<br>
> >>> --download-hypre --download-ptscotch --download-chaco --with-ctable=1<br>
> >>> --download-cmake --with-cc=gcc --with-cxx=g++ --with-debugging=1<br>
> >>> --with-visibility=0 --with-fc=gfortran<br>
> >>> Hong<br>
> >>><br>
> >>> On Fri, Nov 10, 2017 at 9:59 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> >>><br>
> >>>> This must be a configure issue. I don't see these warning:<br>
> >>>><br>
> >>>> #!/usr/bin/python<br>
> >>>> if __name__ == '__main__':<br>
> >>>> import sys<br>
> >>>> import os<br>
> >>>> sys.path.insert(0, os.path.abspath('config'))<br>
> >>>> import configure<br>
> >>>> configure_options = [<br>
> >>>> '--with-cc=clang',<br>
> >>>> '--with-cc++=clang++',<br>
> >>>> '--download-mpich=1',<br>
> >>>> '--download-metis=1',<br>
> >>>> '--download-superlu=1',<br>
> >>>> '--download-superlu_dist=1',<br>
> >>>> '--download-parmetis=1',<br>
> >>>> '--download-fblaslapack=1',<br>
> >>>> '--download-p4est=1',<br>
> >>>> '--with-debugging=1',<br>
> >>>> '--with-batch=0',<br>
> >>>> 'PETSC_ARCH=arch-linux2-c-<wbr>dbg32',<br>
> >>>> '--with-openmp=0',<br>
> >>>> '--download-p4est=0'<br>
> >>>> ]<br>
> >>>> configure.petsc_configure(<wbr>configure_options)<br>
> >>>><br>
> >>>> ~<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> On Fri, Nov 10, 2017 at 10:56 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> >>>><br>
> >>>>> This is comming from blas. How did you configure blas?<br>
> >>>>><br>
> >>>>> On Fri, Nov 10, 2017 at 10:38 AM, Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br>
> >>>>><br>
> >>>>>> Using petsc machine, I get<br>
> >>>>>> hzhang@petsc /sandbox/hzhang/petsc/src/<wbr>snes/examples/tutorials<br>
> >>>>>> (hzhang/fix-submat_<wbr>samerowdist)<br>
> >>>>>> $ mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3<br>
> >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol<br>
> >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg<br>
> >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10<br>
> >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1<br>
> >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason<br>
> >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type<br>
> >>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10<br>
> >>>>>> -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type<br>
> >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br>
> >>>>>><br>
> >>>>>> ==28811== Memcheck, a memory error detector<br>
> >>>>>> ==28811== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et<br>
> >>>>>> al.<br>
> >>>>>> ==28811== Using Valgrind-3.10.1 and LibVEX; rerun with -h for<br>
> >>>>>> copyright info<br>
> >>>>>> ==28811== Command: ./ex56 -cells 2,2,1 -max_conv_its 3<br>
> >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol<br>
> >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg<br>
> >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10<br>
> >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1<br>
> >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason<br>
> >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type<br>
> >>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10<br>
> >>>>>> -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type<br>
> >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br>
> >>>>>> ...<br>
> >>>>>> ==28811== Invalid read of size 16<br>
> >>>>>> ==28811== at 0x8550946: dswap_k_NEHALEM (in<br>
> >>>>>> /usr/lib/openblas-base/<wbr>libblas.so.3)<br>
> >>>>>> ==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas<br>
> >>>>>> .so.3)<br>
> >>>>>> ==28811== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.3<br>
> >>>>>> .0)<br>
> >>>>>> ==28811== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)<br>
> >>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)<br>
> >>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)<br>
> >>>>>> ==28811== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28811== Address 0xdc650d0 is 52,480 bytes inside a block of size<br>
> >>>>>> 52,488 alloc'd<br>
> >>>>>> ==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me<br>
> >>>>>> mcheck-amd64-linux.so)<br>
> >>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)<br>
> >>>>>> ==28811== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)<br>
> >>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)<br>
> >>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)<br>
> >>>>>> ==28811== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28811==<br>
> >>>>>> ==28812== Invalid read of size 16<br>
> >>>>>> ==28812== at 0x8550946: dswap_k_NEHALEM (in<br>
> >>>>>> /usr/lib/openblas-base/<wbr>libblas.so.3)<br>
> >>>>>> ==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas<br>
> >>>>>> .so.3)<br>
> >>>>>> ==28812== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.3<br>
> >>>>>> .0)<br>
> >>>>>> ==28812== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)<br>
> >>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)<br>
> >>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)<br>
> >>>>>> ==28812== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28812== Address 0xd9c7600 is 52,480 bytes inside a block of size<br>
> >>>>>> 52,488 alloc'd<br>
> >>>>>> ==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me<br>
> >>>>>> mcheck-amd64-linux.so)<br>
> >>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)<br>
> >>>>>> ==28812== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)<br>
> >>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)<br>
> >>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)<br>
> >>>>>> ==28812== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28812==<br>
> >>>>>> ==28811== Invalid read of size 16<br>
> >>>>>> ==28811== at 0x8550A55: dswap_k_NEHALEM (in<br>
> >>>>>> /usr/lib/openblas-base/<wbr>libblas.so.3)<br>
> >>>>>> ==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas<br>
> >>>>>> .so.3)<br>
> >>>>>> ==28811== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.3<br>
> >>>>>> .0)<br>
> >>>>>> ==28811== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)<br>
> >>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)<br>
> >>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)<br>
> >>>>>> ==28811== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28811== Address 0xd99cbe0 is 64 bytes inside a block of size 72<br>
> >>>>>> alloc'd<br>
> >>>>>> ==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me<br>
> >>>>>> mcheck-amd64-linux.so)<br>
> >>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)<br>
> >>>>>> ==28811== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)<br>
> >>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)<br>
> >>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)<br>
> >>>>>> ==28811== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28811==<br>
> >>>>>> ==28812== Invalid read of size 16<br>
> >>>>>> ==28812== at 0x8550A55: dswap_k_NEHALEM (in<br>
> >>>>>> /usr/lib/openblas-base/<wbr>libblas.so.3)<br>
> >>>>>> ==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas<br>
> >>>>>> .so.3)<br>
> >>>>>> ==28812== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.3<br>
> >>>>>> .0)<br>
> >>>>>> ==28812== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)<br>
> >>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)<br>
> >>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)<br>
> >>>>>> ==28812== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28812== Address 0xdc11f30 is 64 bytes inside a block of size 72<br>
> >>>>>> alloc'd<br>
> >>>>>> ==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me<br>
> >>>>>> mcheck-amd64-linux.so)<br>
> >>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)<br>
> >>>>>> ==28812== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)<br>
> >>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)<br>
> >>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)<br>
> >>>>>> ==28812== by 0x41056E: main (ex56.c:395)<br>
> >>>>>> ==28812==<br>
> >>>>>> [0] 27 global equations, 9 vertices<br>
> >>>>>> [0] 27 equations in vector, 9 vertices<br>
> >>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1<br>
> >>>>>> [0] 441 global equations, 147 vertices<br>
> >>>>>> [0] 441 equations in vector, 147 vertices<br>
> >>>>>><br>
> >>>>>> hangs here ...<br>
> >>>>>><br>
> >>>>>> Hong<br>
> >>>>>><br>
> >>>>>> On Fri, Nov 10, 2017 at 8:57 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> >>>>>><br>
> >>>>>>> This printed a little funny in gmail, snes/ex56 is running clean in<br>
> >>>>>>> the first few loops (appended), but the last one is the one with a reduced<br>
> >>>>>>> processor set. Still waiting. This is with 32 bit integers. I'm running<br>
> >>>>>>> another with 64 bit integers.<br>
> >>>>>>><br>
> >>>>>>> ...<br>
> >>>>>>> [0] 27 global equations, 9 vertices<br>
> >>>>>>> [0] 27 equations in vector, 9 vertices<br>
> >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations<br>
> >>>>>>> 1<br>
> >>>>>>> [0] 441 global equations, 147 vertices<br>
> >>>>>>> [0] 441 equations in vector, 147 vertices<br>
> >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations<br>
> >>>>>>> 1<br>
> >>>>>>> [0] 4725 global equations, 1575 vertices<br>
> >>>>>>> [0] 4725 equations in vector, 1575 vertices<br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>> On Fri, Nov 10, 2017 at 9:06 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
> >>>>>>><br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>> On Thu, Nov 9, 2017 at 1:56 PM, Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br>
> >>>>>>>><br>
> >>>>>>>>> Mark:<br>
> >>>>>>>>><br>
> >>>>>>>>>> OK, well, just go with the Linux machine for the regression test.<br>
> >>>>>>>>>> I will keep trying to reproduce this on my Mac with an O build.<br>
> >>>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>> Valgrind error occurs on linux machines with g-build. I cannot<br>
> >>>>>>>>> merge this branch to maint until the bug is fixed.<br>
> >>>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>> Valgrind is failing on this run on my Mac. Moving to cg, like you I<br>
> >>>>>>>> suppose. This takes forever. This is what I have so far. Did you get this<br>
> >>>>>>>> far?<br>
> >>>>>>>><br>
> >>>>>>>> 07:48 hzhang/fix-submat_samerowdist *=<br>
> >>>>>>>> /sandbox/adams/petsc/src/snes/<wbr>examples/tutorials$ make<br>
> >>>>>>>> PETSC_DIR=/sandbox/adams/petsc PETSC_ARCH=arch-linux2-c-dbg32 val<br>
> >>>>>>>> /sandbox/adams/petsc/arch-<wbr>linux2-c-dbg32/bin/mpiexec -n 2 valgrind<br>
> >>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2<br>
> >>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type<br>
> >>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg<br>
> >>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10<br>
> >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1<br>
> >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason<br>
> >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type<br>
> >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10<br>
> >>>>>>>> -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type<br>
> >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br>
> >>>>>>>> ==12414== Memcheck, a memory error detector<br>
> >>>>>>>> ==12414== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward<br>
> >>>>>>>> et al.<br>
> >>>>>>>> ==12414== Using Valgrind-3.10.1 and LibVEX; rerun with -h for<br>
> >>>>>>>> copyright info<br>
> >>>>>>>> ==12415== Memcheck, a memory error detector<br>
> >>>>>>>> ==12415== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward<br>
> >>>>>>>> et al.<br>
> >>>>>>>> ==12415== Using Valgrind-3.10.1 and LibVEX; rerun with -h for<br>
> >>>>>>>> copyright info<br>
> >>>>>>>> ==12415== Command: ./ex56 -cells 2,2,1 -max_conv_its 3<br>
> >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol<br>
> >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg<br>
> >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10<br>
> >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1<br>
> >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason<br>
> >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type<br>
> >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10<br>
> >>>>>>>> -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type<br>
> >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br>
> >>>>>>>> ==12415==<br>
> >>>>>>>> ==12414== Command: ./ex56 -cells 2,2,1 -max_conv_its 3<br>
> >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol<br>
> >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg<br>
> >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10<br>
> >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1<br>
> >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -snes_converged_reason<br>
> >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type<br>
> >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10<br>
> >>>>>>>> -mg_levels_ksp_chebyshev_<wbr>esteig 0,0.05,0,1.05 -mg_levels_pc_type<br>
> >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 -run_type 1<br>
> >>>>>>>> ==12414==<br>
> >>>>>>>> [0] 27 global equations, 9 vertices<br>
> >>>>>>>> [0] 27 equations in vector, 9 vertices<br>
> >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE<br>
> >>>>>>>> iterations 1<br>
> >>>>>>>> [0] 441 global equations, 147 vertices<br>
> >>>>>>>> [0] 441 equations in vector, 147 vertices<br>
> >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE<br>
> >>>>>>>> iterations 1<br>
> >>>>>>>> [0] 4725 global equations, 1575 vertices<br>
> >>>>>>>> [0] 4725 equations in vector, 1575 vertices<br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>><br>
> >>>>>><br>
> >>>>><br>
> >>>><br>
> >>><br>
> >><br>
> ><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>