From karabelaselias at gmail.com Fri Apr 1 08:16:02 2016 From: karabelaselias at gmail.com (Elias Karabelas) Date: Fri, 1 Apr 2016 15:16:02 +0200 Subject: [petsc-users] Problems with FBCGSR Message-ID: <56FE7492.4010806@gmail.com> Hi, I'm using petsc for solving a non symmetric block system with a field-split preconditioner. At the moment I use the fgmres as outer solver. I recently discovered that there is also a flexible BICG variant in petsc. So I wanted to give it a try. However I get strange behaviour when using it. With FGMRES i got the following residual history 0 KSP unpreconditioned resid norm 8.632122370708e+00 true resid norm 8.632122370708e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.881521061968e-04 true resid norm 5.881521061969e-04 ||r(i)||/||b|| 6.813528364620e-05 2 KSP unpreconditioned resid norm 6.142969823931e-05 true resid norm 6.142969823933e-05 ||r(i)||/||b|| 7.116407252031e-06 3 KSP unpreconditioned resid norm 4.110742198982e-05 true resid norm 4.110742198982e-05 ||r(i)||/||b|| 4.762145417367e-06 4 KSP unpreconditioned resid norm 2.628280876832e-05 true resid norm 2.628280876817e-05 ||r(i)||/||b|| 3.044767861187e-06 5 KSP unpreconditioned resid norm 1.293845872473e-05 true resid norm 1.293845872450e-05 ||r(i)||/||b|| 1.498873413612e-06 6 KSP unpreconditioned resid norm 6.930434272105e-06 true resid norm 6.930434271781e-06 ||r(i)||/||b|| 8.028656191550e-07 7 KSP unpreconditioned resid norm 4.654261998874e-06 true resid norm 4.654261998975e-06 ||r(i)||/||b|| 5.391793349419e-07 8 KSP unpreconditioned resid norm 3.370057395937e-06 true resid norm 3.370057396119e-06 ||r(i)||/||b|| 3.904088996184e-07 9 KSP unpreconditioned resid norm 2.540901076284e-06 true resid norm 2.540901076192e-06 ||r(i)||/||b|| 2.943541538306e-07 10 KSP unpreconditioned resid norm 2.039263657470e-06 true resid norm 2.039263657746e-06 ||r(i)||/||b|| 2.362412822907e-07 11 KSP unpreconditioned resid norm 1.583890355441e-06 true resid norm 1.583890355170e-06 ||r(i)||/||b|| 1.834879404102e-07 12 KSP unpreconditioned resid norm 1.184174171199e-06 true resid norm 1.184174170742e-06 ||r(i)||/||b|| 1.371822733607e-07 13 KSP unpreconditioned resid norm 8.493013125005e-07 true resid norm 8.493013119686e-07 ||r(i)||/||b|| 9.838846989132e-08 14 KSP unpreconditioned resid norm 5.905267399341e-07 true resid norm 5.905267385984e-07 ||r(i)||/||b|| 6.841037617843e-08 15 KSP unpreconditioned resid norm 4.160983692482e-07 true resid norm 4.160983680083e-07 ||r(i)||/||b|| 4.820348347010e-08 16 KSP unpreconditioned resid norm 2.929006184135e-07 true resid norm 2.929006164636e-07 ||r(i)||/||b|| 3.393147176151e-08 17 KSP unpreconditioned resid norm 2.085222908044e-07 true resid norm 2.085222901282e-07 ||r(i)||/||b|| 2.415654935984e-08 Switching to FBCGSR however yielded the following results 0 KSP unpreconditioned resid norm 8.632122370749e+00 true resid norm 8.632122370749e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 6.155300069200e-05 true resid norm 4.783918927770e+05 ||r(i)||/||b|| 5.541996188539e+04 2 KSP unpreconditioned resid norm 5.203978464579e-05 true resid norm 4.069231018620e+05 ||r(i)||/||b|| 4.714056223773e+04 3 KSP unpreconditioned resid norm 4.124211335014e-05 true resid norm 8.257465019947e+05 ||r(i)||/||b|| 9.565973077408e+04 4 KSP unpreconditioned resid norm 2.735866558072e-05 true resid norm 4.745745445186e+05 ||r(i)||/||b|| 5.497773596523e+04 5 KSP unpreconditioned resid norm 1.234049073102e-05 true resid norm 3.694211077995e+05 ||r(i)||/||b|| 4.279609254050e+04 6 KSP unpreconditioned resid norm 7.813287928316e-06 true resid norm 3.595303092238e+05 ||r(i)||/||b|| 4.165027947728e+04 7 KSP unpreconditioned resid norm 3.221541950651e-06 true resid norm 3.179392486707e+05 ||r(i)||/||b|| 3.683210629034e+04 8 KSP unpreconditioned resid norm 1.777674826629e-06 true resid norm 3.155842168872e+05 ||r(i)||/||b|| 3.655928441846e+04 9 KSP unpreconditioned resid norm 1.698816021362e-06 true resid norm 3.145809168718e+05 ||r(i)||/||b|| 3.644305575854e+04 10 KSP unpreconditioned resid norm 1.577351929557e-06 true resid norm 3.259758397382e+05 ||r(i)||/||b|| 3.776311615354e+04 11 KSP unpreconditioned resid norm 1.613962504892e-06 true resid norm 3.186021802487e+05 ||r(i)||/||b|| 3.690890450398e+04 12 KSP unpreconditioned resid norm 4.564619465810e-07 true resid norm 3.142082512956e+05 ||r(i)||/||b|| 3.639988380613e+04 13 KSP unpreconditioned resid norm 3.413352474598e-07 true resid norm 3.136609638739e+05 ||r(i)||/||b|| 3.633648254765e+04 14 KSP unpreconditioned resid norm 2.914787162382e-07 true resid norm 3.133747605150e+05 ||r(i)||/||b|| 3.630332692884e+04 15 KSP unpreconditioned resid norm 3.607032335094e-07 true resid norm 3.108398102641e+05 ||r(i)||/||b|| 3.600966215648e+04 16 KSP unpreconditioned resid norm 7.653443566257e-08 true resid norm 3.124857549918e+05 ||r(i)||/||b|| 3.620033887039e+04 So it seems to me something weird is going on here. Could you help me? Kind regards Elias From karabelaselias at gmail.com Fri Apr 1 08:19:30 2016 From: karabelaselias at gmail.com (Elias Karabelas) Date: Fri, 1 Apr 2016 15:19:30 +0200 Subject: [petsc-users] Problems with FBCGSR In-Reply-To: <56FE7492.4010806@gmail.com> References: <56FE7492.4010806@gmail.com> Message-ID: <56FE7562.6050705@gmail.com> Hi, I'm using petsc for solving a non symmetric block system with a field-split preconditioner. At the moment I use the fgmres as outer solver. I recently discovered that there is also a flexible BICG variant in petsc. So I wanted to give it a try. However I get strange behaviour when using it. With FGMRES i got the following residual history 0 KSP unpreconditioned resid norm 8.632122370708e+00 true resid norm 8.632122370708e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.881521061968e-04 true resid norm 5.881521061969e-04 ||r(i)||/||b|| 6.813528364620e-05 2 KSP unpreconditioned resid norm 6.142969823931e-05 true resid norm 6.142969823933e-05 ||r(i)||/||b|| 7.116407252031e-06 3 KSP unpreconditioned resid norm 4.110742198982e-05 true resid norm 4.110742198982e-05 ||r(i)||/||b|| 4.762145417367e-06 4 KSP unpreconditioned resid norm 2.628280876832e-05 true resid norm 2.628280876817e-05 ||r(i)||/||b|| 3.044767861187e-06 5 KSP unpreconditioned resid norm 1.293845872473e-05 true resid norm 1.293845872450e-05 ||r(i)||/||b|| 1.498873413612e-06 6 KSP unpreconditioned resid norm 6.930434272105e-06 true resid norm 6.930434271781e-06 ||r(i)||/||b|| 8.028656191550e-07 7 KSP unpreconditioned resid norm 4.654261998874e-06 true resid norm 4.654261998975e-06 ||r(i)||/||b|| 5.391793349419e-07 8 KSP unpreconditioned resid norm 3.370057395937e-06 true resid norm 3.370057396119e-06 ||r(i)||/||b|| 3.904088996184e-07 9 KSP unpreconditioned resid norm 2.540901076284e-06 true resid norm 2.540901076192e-06 ||r(i)||/||b|| 2.943541538306e-07 10 KSP unpreconditioned resid norm 2.039263657470e-06 true resid norm 2.039263657746e-06 ||r(i)||/||b|| 2.362412822907e-07 11 KSP unpreconditioned resid norm 1.583890355441e-06 true resid norm 1.583890355170e-06 ||r(i)||/||b|| 1.834879404102e-07 12 KSP unpreconditioned resid norm 1.184174171199e-06 true resid norm 1.184174170742e-06 ||r(i)||/||b|| 1.371822733607e-07 13 KSP unpreconditioned resid norm 8.493013125005e-07 true resid norm 8.493013119686e-07 ||r(i)||/||b|| 9.838846989132e-08 14 KSP unpreconditioned resid norm 5.905267399341e-07 true resid norm 5.905267385984e-07 ||r(i)||/||b|| 6.841037617843e-08 15 KSP unpreconditioned resid norm 4.160983692482e-07 true resid norm 4.160983680083e-07 ||r(i)||/||b|| 4.820348347010e-08 16 KSP unpreconditioned resid norm 2.929006184135e-07 true resid norm 2.929006164636e-07 ||r(i)||/||b|| 3.393147176151e-08 17 KSP unpreconditioned resid norm 2.085222908044e-07 true resid norm 2.085222901282e-07 ||r(i)||/||b|| 2.415654935984e-08 Switching to FBCGSR however yielded the following results 0 KSP unpreconditioned resid norm 8.632122370749e+00 true resid norm 8.632122370749e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 6.155300069200e-05 true resid norm 4.783918927770e+05 ||r(i)||/||b|| 5.541996188539e+04 2 KSP unpreconditioned resid norm 5.203978464579e-05 true resid norm 4.069231018620e+05 ||r(i)||/||b|| 4.714056223773e+04 3 KSP unpreconditioned resid norm 4.124211335014e-05 true resid norm 8.257465019947e+05 ||r(i)||/||b|| 9.565973077408e+04 4 KSP unpreconditioned resid norm 2.735866558072e-05 true resid norm 4.745745445186e+05 ||r(i)||/||b|| 5.497773596523e+04 5 KSP unpreconditioned resid norm 1.234049073102e-05 true resid norm 3.694211077995e+05 ||r(i)||/||b|| 4.279609254050e+04 6 KSP unpreconditioned resid norm 7.813287928316e-06 true resid norm 3.595303092238e+05 ||r(i)||/||b|| 4.165027947728e+04 7 KSP unpreconditioned resid norm 3.221541950651e-06 true resid norm 3.179392486707e+05 ||r(i)||/||b|| 3.683210629034e+04 8 KSP unpreconditioned resid norm 1.777674826629e-06 true resid norm 3.155842168872e+05 ||r(i)||/||b|| 3.655928441846e+04 9 KSP unpreconditioned resid norm 1.698816021362e-06 true resid norm 3.145809168718e+05 ||r(i)||/||b|| 3.644305575854e+04 10 KSP unpreconditioned resid norm 1.577351929557e-06 true resid norm 3.259758397382e+05 ||r(i)||/||b|| 3.776311615354e+04 11 KSP unpreconditioned resid norm 1.613962504892e-06 true resid norm 3.186021802487e+05 ||r(i)||/||b|| 3.690890450398e+04 12 KSP unpreconditioned resid norm 4.564619465810e-07 true resid norm 3.142082512956e+05 ||r(i)||/||b|| 3.639988380613e+04 13 KSP unpreconditioned resid norm 3.413352474598e-07 true resid norm 3.136609638739e+05 ||r(i)||/||b|| 3.633648254765e+04 14 KSP unpreconditioned resid norm 2.914787162382e-07 true resid norm 3.133747605150e+05 ||r(i)||/||b|| 3.630332692884e+04 15 KSP unpreconditioned resid norm 3.607032335094e-07 true resid norm 3.108398102641e+05 ||r(i)||/||b|| 3.600966215648e+04 16 KSP unpreconditioned resid norm 7.653443566257e-08 true resid norm 3.124857549918e+05 ||r(i)||/||b|| 3.620033887039e+04 So it seems to me something weird is going on here. Could you help me? Kind regards Elias From xzhao99 at gmail.com Fri Apr 1 16:53:20 2016 From: xzhao99 at gmail.com (Xujun Zhao) Date: Fri, 1 Apr 2016 16:53:20 -0500 Subject: [petsc-users] Changes on direct solver interface??? Message-ID: Hi all, Are there any changes on the PETSc direct solver interface, for example, with superLU_dist and MUMPS. I found that my libMesh code failed with both of them, but still works with iterative solver (GMRES by default). Thanks a lot. Best, Xujun -------------------------------------------------------------------------------------------------------------- [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown [0]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown [1]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 [cli_1]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 ------------------------------------------------------------------------------------------------------------------- | Processor id: 0 | | Num Processors: 4 | | Time: Fri Apr 1 15:57:40 2016 | | OS: Darwin | | HostName: mcswl105.mcs.anl.gov | | OS Release: 15.2.0 | | OS Version: Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64 | | Machine: x86_64 | | Username: xzhao | | Configuration: ./configure '--prefix=/Users/xzhao/software/libmesh/libmesh-dev' | | '--with-methods=opt' | | '--enable-everything' | | '--enable-parmesh' | | '--disable-strict-lgpl' | | '--with-vtk-include=/usr/local/include/vtk-6.2' | | '--with-vtk-lib=/usr/local/lib' | | 'PETSC_DIR=/Users/xzhao/software/petsc/petsc-dev' | | 'PETSC_ARCH=arch-darwin-c-opt' | | 'SLEPC_DIR=/Users/xzhao/software/slepc/slepc-3.6' | ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Apr 1 17:13:38 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 1 Apr 2016 17:13:38 -0500 Subject: [petsc-users] Changes on direct solver interface??? In-Reply-To: References: Message-ID: please run the code in debugger and obtain a stack trace. Satish On Fri, 1 Apr 2016, Xujun Zhao wrote: > Hi all, > > Are there any changes on the PETSc direct solver interface, for example, > with superLU_dist and MUMPS. I found that my libMesh code failed with both > of them, but still works with iterative solver (GMRES by default). Thanks a > lot. > > Best, > Xujun > > -------------------------------------------------------------------------------------------------------------- > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X > to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > [0]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named > mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich > --download-fblaslapack --download-scalapack --download-mumps > --download-superlu_dist --download-hypre --download-ml --download-metis > --download-parmetis --download-triangle --download-chaco > --download-elemental --with-debugging=0 > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [cli_0]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X > to find memory corruption errors > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > > [1]PETSC ERROR: to get more information on the crash. > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: Signal received > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > [1]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named > mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich > --download-fblaslapack --download-scalapack --download-mumps > --download-superlu_dist --download-hypre --download-ml --download-metis > --download-parmetis --download-triangle --download-chaco > --download-elemental --with-debugging=0 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > > [cli_1]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > > > ------------------------------------------------------------------------------------------------------------------- > > | Processor id: 0 > | > > | Num Processors: 4 > | > > | Time: Fri Apr 1 15:57:40 2016 > | > > | OS: Darwin > | > > | HostName: mcswl105.mcs.anl.gov > | > > | OS Release: 15.2.0 > | > > | OS Version: Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST > 2015; root:xnu-3248.20.55~2/RELEASE_X86_64 | > > | Machine: x86_64 > | > > | Username: xzhao > | > > | Configuration: ./configure > '--prefix=/Users/xzhao/software/libmesh/libmesh-dev' > | > > | '--with-methods=opt' > | > > | '--enable-everything' > | > > | '--enable-parmesh' > | > > | '--disable-strict-lgpl' > | > > | '--with-vtk-include=/usr/local/include/vtk-6.2' > | > > | '--with-vtk-lib=/usr/local/lib' > | > > | 'PETSC_DIR=/Users/xzhao/software/petsc/petsc-dev' > | > > | 'PETSC_ARCH=arch-darwin-c-opt' > | > > | 'SLEPC_DIR=/Users/xzhao/software/slepc/slepc-3.6' > | > > ------------------------------------------------------------------------------------------------------------------- > From xzhao99 at gmail.com Fri Apr 1 17:30:18 2016 From: xzhao99 at gmail.com (Xujun Zhao) Date: Fri, 1 Apr 2016 17:30:18 -0500 Subject: [petsc-users] Changes on direct solver interface??? In-Reply-To: References: Message-ID: No dbg version complied now, but will do now. A first look with -info shows something is wrong when SetUpPC for superLU_dist: [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] MatStashScatterBegin_Private(): No of messages: 0 [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatStashScatterBegin_Private(): No of messages: 1 [1] MatStashScatterBegin_Private(): Mesg_to: 0: size: 28160 bytes [1] MatAssemblyBegin_MPIAIJ(): Stash has 1760 entries, uses 0 mallocs. [3] MatStashScatterBegin_Private(): No of messages: 2 [3] MatStashScatterBegin_Private(): Mesg_to: 0: size: 22528 bytes [2] MatStashScatterBegin_Private(): No of messages: 2 [2] MatStashScatterBegin_Private(): Mesg_to: 0: size: 1056 bytes [3] MatStashScatterBegin_Private(): Mesg_to: 2: size: 18656 bytes [3] MatAssemblyBegin_MPIAIJ(): Stash has 2574 entries, uses 0 mallocs. [2] MatStashScatterBegin_Private(): Mesg_to: 1: size: 36960 bytes [2] MatAssemblyBegin_MPIAIJ(): Stash has 2376 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 504 X 504; storage space: 0 unneeded,17634 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 504) < 0.6. Do not use CompressedRow routines. [3] MatSeqAIJCheckInode(): Found 278 nodes of 504. Limit used: 5. Using Inode routines [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 495 X 495; storage space: 355 unneeded,17115 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 605 X 605; storage space: 1017 unneeded,22159 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 495) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 605) < 0.6. Do not use CompressedRow routines. [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 574 X 574; storage space: 795 unneeded,20096 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 574) < 0.6. Do not use CompressedRow routines. [2] MatSeqAIJCheckInode(): Found 261 nodes of 495. Limit used: 5. Using Inode routines [0] MatSeqAIJCheckInode(): Found 327 nodes of 605. Limit used: 5. Using Inode routines [1] MatSeqAIJCheckInode(): Found 298 nodes of 574. Limit used: 5. Using Inode routines [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter [0] VecScatterCreate(): General case: MPI to Seq [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 495 X 146; storage space: 15 unneeded,1754 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 40 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 605 X 181; storage space: 30 unneeded,1703 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 37 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 512)/(num_localrows 605) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 574 X 140; storage space: 15 unneeded,1945 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 31 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 409)/(num_localrows 574) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 378)/(num_localrows 495) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 504 X 78; storage space: 0 unneeded,1378 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 23 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 378)/(num_localrows 504) > 0.6. Use CompressedRow routines. [0] MatStashScatterBegin_Private(): No of messages: 0 [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 605 X 605; storage space: 0 unneeded,22159 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [1] MatStashScatterBegin_Private(): No of messages: 0 [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 605) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 605 X 181; storage space: 0 unneeded,1703 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 37 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 574 X 574; storage space: 0 unneeded,20096 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 574) < 0.6. Do not use CompressedRow routines. [2] MatStashScatterBegin_Private(): No of messages: 0 [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 495 X 495; storage space: 0 unneeded,17115 used [3] MatStashScatterBegin_Private(): No of messages: 0 [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 504 X 504; storage space: 0 unneeded,17634 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 512)/(num_localrows 605) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 574 X 140; storage space: 0 unneeded,1945 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 31 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 409)/(num_localrows 574) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 495) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 495 X 146; storage space: 0 unneeded,1754 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 40 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 504) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 504 X 78; storage space: 0 unneeded,1378 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 23 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 378)/(num_localrows 504) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 378)/(num_localrows 495) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] PCSetUp(): Setting up PC for first time [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown [1]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 17:26:55 2016 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 [cli_1]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.6.3, unknown [3]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 17:26:55 2016 [3]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [3]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [cli_3]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown [0]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 17:26:55 2016 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.6.3, unknown [2]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named mcswl105.mcs.anl.gov by xzhao Fri Apr 1 17:26:55 2016 [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-fblaslapack --download-scalapack --download-mumps --download-superlu_dist --download-hypre --download-ml --download-metis --download-parmetis --download-triangle --download-chaco --download-elemental --with-debugging=0 [2]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 [cli_2]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 On Fri, Apr 1, 2016 at 5:13 PM, Satish Balay wrote: > please run the code in debugger and obtain a stack trace. > > Satish > > On Fri, 1 Apr 2016, Xujun Zhao wrote: > > > Hi all, > > > > Are there any changes on the PETSc direct solver interface, for example, > > with superLU_dist and MUMPS. I found that my libMesh code failed with > both > > of them, but still works with iterative solver (GMRES by default). > Thanks a > > lot. > > > > Best, > > Xujun > > > > > -------------------------------------------------------------------------------------------------------------- > > > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > > probably memory access out of range > > > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [0]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X > > to find memory corruption errors > > > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and > > run > > > > [0]PETSC ERROR: to get more information on the crash. > > > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > > > [0]PETSC ERROR: Signal received > > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for > > trouble shooting. > > > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > > > [0]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named > > mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > > --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich > > --download-fblaslapack --download-scalapack --download-mumps > > --download-superlu_dist --download-hypre --download-ml --download-metis > > --download-parmetis --download-triangle --download-chaco > > --download-elemental --with-debugging=0 > > > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > [cli_0]: aborting job: > > > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > [1]PETSC ERROR: > > ------------------------------------------------------------------------ > > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > > probably memory access out of range > > > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [1]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X > > to find memory corruption errors > > > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and > > run > > > > [1]PETSC ERROR: to get more information on the crash. > > > > [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > > > [1]PETSC ERROR: Signal received > > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for > > trouble shooting. > > > > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > > > [1]PETSC ERROR: ./example-opt on a arch-darwin-c-opt named > > mcswl105.mcs.anl.gov by xzhao Fri Apr 1 15:57:40 2016 > > > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > > --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich > > --download-fblaslapack --download-scalapack --download-mumps > > --download-superlu_dist --download-hypre --download-ml --download-metis > > --download-parmetis --download-triangle --download-chaco > > --download-elemental --with-debugging=0 > > > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > > > > [cli_1]: aborting job: > > > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > > > > > > > ------------------------------------------------------------------------------------------------------------------- > > > > | Processor id: 0 > > | > > > > | Num Processors: 4 > > | > > > > | Time: Fri Apr 1 15:57:40 2016 > > | > > > > | OS: Darwin > > | > > > > | HostName: mcswl105.mcs.anl.gov > > | > > > > | OS Release: 15.2.0 > > | > > > > | OS Version: Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST > > 2015; root:xnu-3248.20.55~2/RELEASE_X86_64 | > > > > | Machine: x86_64 > > | > > > > | Username: xzhao > > | > > > > | Configuration: ./configure > > '--prefix=/Users/xzhao/software/libmesh/libmesh-dev' > > | > > > > | '--with-methods=opt' > > | > > > > | '--enable-everything' > > | > > > > | '--enable-parmesh' > > | > > > > | '--disable-strict-lgpl' > > | > > > > | '--with-vtk-include=/usr/local/include/vtk-6.2' > > | > > > > | '--with-vtk-lib=/usr/local/lib' > > | > > > > | 'PETSC_DIR=/Users/xzhao/software/petsc/petsc-dev' > > | > > > > | 'PETSC_ARCH=arch-darwin-c-opt' > > | > > > > | 'SLEPC_DIR=/Users/xzhao/software/slepc/slepc-3.6' > > | > > > > > ------------------------------------------------------------------------------------------------------------------- > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 03:51:40 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 16:51:40 +0800 Subject: [petsc-users] Vec is locked read only Message-ID: <56FF881C.5050307@siat.ac.cn> Dear All, My code got the following error messages, but the code works well for the petsc optimized version (--with-debugging=0). Anyone can tell me how to fix this problem? Best regards, Rongliang [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Vec is locked read only, argument # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr 2 15:40:32 2016 [0]PETSC ERROR: Configure options --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared --with-debugging=1 --download-fblaslapack --download-chaco [0]PETSC ERROR: #1 VecGetArray() line 1646 in /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupFunctions.c [0]PETSC ERROR: #4 FormFunction() line 46 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupFunctions.c [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c [0]PETSC ERROR: #7 SNESSolve() line 3906 in /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun.c [0]PETSC ERROR: #9 main() line 452 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun.c From abhyshr at anl.gov Sat Apr 2 04:03:57 2016 From: abhyshr at anl.gov (Abhyankar, Shrirang G.) Date: Sat, 2 Apr 2016 09:03:57 +0000 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FF881C.5050307@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> Message-ID: Use VecGetArrayRead instead of VecGetArray http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray Read.html Shri -----Original Message----- From: Rongliang Chen Date: Saturday, April 2, 2016 at 3:51 AM To: PETSc users list , "rongliang.chan at gmail.com" Subject: [petsc-users] Vec is locked read only >Dear All, > >My code got the following error messages, but the code works well for >the petsc optimized version (--with-debugging=0). Anyone can tell me how >to fix this problem? > >Best regards, >Rongliang > > >[0]PETSC ERROR: --------------------- Error Message >-------------------------------------------------------------- >[0]PETSC ERROR: Object is in wrong state >[0]PETSC ERROR: Vec is locked read only, argument # 1 >[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >for trouble shooting. >[0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >[0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr >2 15:40:32 2016 >[0]PETSC ERROR: Configure options --download-blacs --download-scalapack >--download-metis --download-parmetis --download-exodusii >--download-netcdf --download-hdf5 >--with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >--with-debugging=1 --download-fblaslapack --download-chaco >[0]PETSC ERROR: #1 VecGetArray() line 1646 in >/home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >[0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >/home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >[0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >unctions.c >[0]PETSC ERROR: #4 FormFunction() line 46 in >/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >unctions.c >[0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >/home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >[0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >/home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >[0]PETSC ERROR: #7 SNESSolve() line 3906 in >/home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >[0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >c >[0]PETSC ERROR: #9 main() line 452 in >/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >c > > From rl.chen at siat.ac.cn Sat Apr 2 04:18:45 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 17:18:45 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> Message-ID: <56FF8E75.9040106@siat.ac.cn> Hi Shri, Thanks for your reply. Do you mean that I need to change the VecGetArrary() in /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? I tried it and I got a warning when I make the petsc: /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function ?DMLocalToGlobalBegin?: /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: passing argument 2 of ?VecGetArrayRead? from incompatible pointer type [enabled by default] In file included from /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: expected ?const PetscScalar **? but argument is of type ?PetscScalar **? Best regards, Rongliang --------------------- Rongliang Chen, PhD Associate Professor Laboratory for Engineering and Scientific Computing Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, Guangdong (518055), P. R. China E-mail: rl.chen at siat.ac.cn Phone: +86-755-86392312 On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: > Use VecGetArrayRead instead of VecGetArray > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray > Read.html > > > Shri > > -----Original Message----- > From: Rongliang Chen > Date: Saturday, April 2, 2016 at 3:51 AM > To: PETSc users list , "rongliang.chan at gmail.com" > > Subject: [petsc-users] Vec is locked read only > >> Dear All, >> >> My code got the following error messages, but the code works well for >> the petsc optimized version (--with-debugging=0). Anyone can tell me how >> to fix this problem? >> >> Best regards, >> Rongliang >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Vec is locked read only, argument # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr >> 2 15:40:32 2016 >> [0]PETSC ERROR: Configure options --download-blacs --download-scalapack >> --download-metis --download-parmetis --download-exodusii >> --download-netcdf --download-hdf5 >> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >> --with-debugging=1 --download-fblaslapack --download-chaco >> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >> unctions.c >> [0]PETSC ERROR: #4 FormFunction() line 46 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >> unctions.c >> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >> c >> [0]PETSC ERROR: #9 main() line 452 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >> c >> >> > From dave.mayhem23 at gmail.com Sat Apr 2 05:20:37 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Sat, 2 Apr 2016 12:20:37 +0200 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FF8E75.9040106@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> Message-ID: On 2 April 2016 at 11:18, Rongliang Chen wrote: > Hi Shri, > > Thanks for your reply. > > Do you mean that I need to change the VecGetArrary() in > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? > No - you should change it in your function FormMassTimeStepFunction(). The input vector x passed into SNESComputeFunction() is read only. > I tried it and I got a warning when I make the petsc: > > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function > ?DMLocalToGlobalBegin?: > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: > passing argument 2 of ?VecGetArrayRead? from incompatible pointer type > [enabled by default] > In file included from /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, > from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, > from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, > from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: > /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: expected > ?const PetscScalar **? but argument is of type ?PetscScalar **? > > Best regards, > Rongliang > > --------------------- > Rongliang Chen, PhD > Associate Professor > > Laboratory for Engineering and Scientific Computing > Shenzhen Institutes of Advanced Technology > Chinese Academy of Sciences > Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, > Guangdong (518055), P. R. China > E-mail: rl.chen at siat.ac.cn > Phone: +86-755-86392312 > > > On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: > >> Use VecGetArrayRead instead of VecGetArray >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >> Read.html >> >> >> Shri >> >> -----Original Message----- >> From: Rongliang Chen >> Date: Saturday, April 2, 2016 at 3:51 AM >> To: PETSc users list , "rongliang.chan at gmail.com >> " >> >> Subject: [petsc-users] Vec is locked read only >> >> Dear All, >>> >>> My code got the following error messages, but the code works well for >>> the petsc optimized version (--with-debugging=0). Anyone can tell me how >>> to fix this problem? >>> >>> Best regards, >>> Rongliang >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr >>> 2 15:40:32 2016 >>> [0]PETSC ERROR: Configure options --download-blacs --download-scalapack >>> --download-metis --download-parmetis --download-exodusii >>> --download-netcdf --download-hdf5 >>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>> --with-debugging=1 --download-fblaslapack --download-chaco >>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>> >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>> unctions.c >>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>> >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>> unctions.c >>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>> >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>> c >>> [0]PETSC ERROR: #9 main() line 452 in >>> >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>> c >>> >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 06:49:54 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 19:49:54 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> Message-ID: <56FFB1E2.3020801@siat.ac.cn> Hi Dave, I did not call any VecGetArrary and VecGetArraryRead in function FormMassTimeStepFunction(). Best regards, Rongliang On 04/02/2016 06:20 PM, Dave May wrote: > > > On 2 April 2016 at 11:18, Rongliang Chen > wrote: > > Hi Shri, > > Thanks for your reply. > > Do you mean that I need to change the VecGetArrary() in > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to > VecGetArrayRead()? > > > No - you should change it in your function FormMassTimeStepFunction(). > The input vector x passed into SNESComputeFunction() is read only. > > > > I tried it and I got a warning when I make the petsc: > > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function > ?DMLocalToGlobalBegin?: > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: > warning: passing argument 2 of ?VecGetArrayRead? from incompatible > pointer type [enabled by default] > In file included from > /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, > from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, > from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, > from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: > /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: > expected ?const PetscScalar **? but argument is of type > ?PetscScalar **? > > Best regards, > Rongliang > > --------------------- > Rongliang Chen, PhD > Associate Professor > > Laboratory for Engineering and Scientific Computing > Shenzhen Institutes of Advanced Technology > Chinese Academy of Sciences > Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, > Guangdong (518055), P. R. China > E-mail: rl.chen at siat.ac.cn > Phone: +86-755-86392312 > > > On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: > > Use VecGetArrayRead instead of VecGetArray > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray > Read.html > > > Shri > > -----Original Message----- > From: Rongliang Chen > > Date: Saturday, April 2, 2016 at 3:51 AM > To: PETSc users list >, "rongliang.chan at gmail.com > " > > > Subject: [petsc-users] Vec is locked read only > > Dear All, > > My code got the following error messages, but the code > works well for > the petsc optimized version (--with-debugging=0). Anyone > can tell me how > to fix this problem? > > Best regards, > Rongliang > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Vec is locked read only, argument # 1 > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by > rlchen Sat Apr > 2 15:40:32 2016 > [0]PETSC ERROR: Configure options --download-blacs > --download-scalapack > --download-metis --download-parmetis --download-exodusii > --download-netcdf --download-hdf5 > --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared > --with-debugging=1 --download-fblaslapack --download-chaco > [0]PETSC ERROR: #1 VecGetArray() line 1646 in > /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in > /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c > [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF > unctions.c > [0]PETSC ERROR: #4 FormFunction() line 46 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF > unctions.c > [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in > /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c > [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in > /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #7 SNESSolve() line 3906 in > /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c > [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. > c > [0]PETSC ERROR: #9 main() line 452 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. > c > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 06:59:04 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 06:59:04 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFB1E2.3020801@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen wrote: > Hi Dave, > > I did not call any VecGetArrary and VecGetArraryRead in function > FormMassTimeStepFunction(). > It looks like you have a function FormMassTimeStep() inside your FormFunction() method. This has an argument X, the solution guess, which is read-only. However, you are calling DMLocalToGlobal() using that X, which is illegal, since its read-only. Is this right? Matt > Best regards, > Rongliang > > On 04/02/2016 06:20 PM, Dave May wrote: > > > > On 2 April 2016 at 11:18, Rongliang Chen wrote: > >> Hi Shri, >> >> Thanks for your reply. >> >> Do you mean that I need to change the VecGetArrary() in >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >> > > No - you should change it in your function FormMassTimeStepFunction(). > The input vector x passed into SNESComputeFunction() is read only. > > > >> I tried it and I got a warning when I make the petsc: >> >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >> ?DMLocalToGlobalBegin?: >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >> [enabled by default] >> In file included from >> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: expected >> ?const PetscScalar **? but argument is of type ?PetscScalar **? >> >> Best regards, >> Rongliang >> >> --------------------- >> Rongliang Chen, PhD >> Associate Professor >> >> Laboratory for Engineering and Scientific Computing >> Shenzhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >> Guangdong (518055), P. R. China >> E-mail: rl.chen at siat.ac.cn >> Phone: +86-755-86392312 >> >> >> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >> >>> Use VecGetArrayRead instead of VecGetArray >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>> Read.html >>> >>> >>> Shri >>> >>> -----Original Message----- >>> From: Rongliang Chen >>> Date: Saturday, April 2, 2016 at 3:51 AM >>> To: PETSc users list , " >>> rongliang.chan at gmail.com" >>> >>> Subject: [petsc-users] Vec is locked read only >>> >>> Dear All, >>>> >>>> My code got the following error messages, but the code works well for >>>> the petsc optimized version (--with-debugging=0). Anyone can tell me how >>>> to fix this problem? >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr >>>> 2 15:40:32 2016 >>>> [0]PETSC ERROR: Configure options --download-blacs --download-scalapack >>>> --download-metis --download-parmetis --download-exodusii >>>> --download-netcdf --download-hdf5 >>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>> >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>> unctions.c >>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>> >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>> unctions.c >>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>> >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>> c >>>> [0]PETSC ERROR: #9 main() line 452 in >>>> >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>> c >>>> >>>> >>>> >>> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 07:15:39 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 20:15:39 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> Message-ID: <56FFB7EB.1060700@siat.ac.cn> Hi Matt, That's right. The FormMassTimeStep() is inside the FormFunction() and I call the FormFunction() in this way: SNESSetFunction(user->snes, algebra->f, FormFunction, (void*)user). How can I know the X is read-only or not? If I do not use the SNESSolve, that is I use the explicit method with the same function FormMassTimeStep(), this error disappeared. So it seems that the SNESSolve locked the vector X. Best regards, Rongliang On 04/02/2016 07:59 PM, Matthew Knepley wrote: > On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen > wrote: > > Hi Dave, > > I did not call any VecGetArrary and VecGetArraryRead in function > FormMassTimeStepFunction(). > > > It looks like you have a function FormMassTimeStep() inside your > FormFunction() method. This > has an argument X, the solution guess, which is read-only. However, > you are calling DMLocalToGlobal() > using that X, which is illegal, since its read-only. Is this right? > > Matt > > Best regards, > Rongliang > > On 04/02/2016 06:20 PM, Dave May wrote: >> >> >> On 2 April 2016 at 11:18, Rongliang Chen > > wrote: >> >> Hi Shri, >> >> Thanks for your reply. >> >> Do you mean that I need to change the VecGetArrary() in >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to >> VecGetArrayRead()? >> >> >> No - you should change it in your function >> FormMassTimeStepFunction(). >> The input vector x passed into SNESComputeFunction() is read only. >> >> >> >> I tried it and I got a warning when I make the petsc: >> >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In >> function ?DMLocalToGlobalBegin?: >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >> warning: passing argument 2 of ?VecGetArrayRead? from >> incompatible pointer type [enabled by default] >> In file included from >> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >> from >> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >> note: expected ?const PetscScalar **? but argument is of type >> ?PetscScalar **? >> >> Best regards, >> Rongliang >> >> --------------------- >> Rongliang Chen, PhD >> Associate Professor >> >> Laboratory for Engineering and Scientific Computing >> Shenzhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> Address: 1068 Xueyuan Avenue, Shenzhen University Town, >> Shenzhen, Guangdong (518055), P. R. China >> E-mail: rl.chen at siat.ac.cn >> Phone: +86-755-86392312 >> >> >> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >> >> Use VecGetArrayRead instead of VecGetArray >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >> Read.html >> >> >> Shri >> >> -----Original Message----- >> From: Rongliang Chen > > >> Date: Saturday, April 2, 2016 at 3:51 AM >> To: PETSc users list > >, >> "rongliang.chan at gmail.com " >> > >> Subject: [petsc-users] Vec is locked read only >> >> Dear All, >> >> My code got the following error messages, but the >> code works well for >> the petsc optimized version (--with-debugging=0). >> Anyone can tell me how >> to fix this problem? >> >> Best regards, >> Rongliang >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Vec is locked read only, argument # 1 >> [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >> 2015 >> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen >> by rlchen Sat Apr >> 2 15:40:32 2016 >> [0]PETSC ERROR: Configure options --download-blacs >> --download-scalapack >> --download-metis --download-parmetis --download-exodusii >> --download-netcdf --download-hdf5 >> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >> --with-debugging=1 --download-fblaslapack >> --download-chaco >> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >> unctions.c >> [0]PETSC ERROR: #4 FormFunction() line 46 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >> unctions.c >> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >> c >> [0]PETSC ERROR: #9 main() line 452 in >> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >> c >> >> >> >> >> >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 07:19:35 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 07:19:35 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFB7EB.1060700@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen wrote: > Hi Matt, > > That's right. The FormMassTimeStep() is inside the FormFunction() and I > call the FormFunction() in this way: SNESSetFunction(user->snes, > algebra->f, FormFunction, (void*)user). How can I know the X is read-only > or not? > Exactly that you get this error message, and it should be in the documentation. > If I do not use the SNESSolve, that is I use the explicit method with the > same function FormMassTimeStep(), this error disappeared. So it seems that > the SNESSolve locked the vector X. > Then we need to put that check in the explicit method as well. You should never be changing the input guess. Thanks, Matt > Best regards, > Rongliang > > On 04/02/2016 07:59 PM, Matthew Knepley wrote: > > On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen wrote: > >> Hi Dave, >> >> I did not call any VecGetArrary and VecGetArraryRead in function >> FormMassTimeStepFunction(). >> > > It looks like you have a function FormMassTimeStep() inside your > FormFunction() method. This > has an argument X, the solution guess, which is read-only. However, you > are calling DMLocalToGlobal() > using that X, which is illegal, since its read-only. Is this right? > > Matt > > >> Best regards, >> Rongliang >> >> On 04/02/2016 06:20 PM, Dave May wrote: >> >> >> >> On 2 April 2016 at 11:18, Rongliang Chen wrote: >> >>> Hi Shri, >>> >>> Thanks for your reply. >>> >>> Do you mean that I need to change the VecGetArrary() in >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >>> >> >> No - you should change it in your function FormMassTimeStepFunction(). >> The input vector x passed into SNESComputeFunction() is read only. >> >> >> >>> I tried it and I got a warning when I make the petsc: >>> >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >>> ?DMLocalToGlobalBegin?: >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >>> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >>> [enabled by default] >>> In file included from >>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: expected >>> ?const PetscScalar **? but argument is of type ?PetscScalar **? >>> >>> Best regards, >>> Rongliang >>> >>> --------------------- >>> Rongliang Chen, PhD >>> Associate Professor >>> >>> Laboratory for Engineering and Scientific Computing >>> Shenzhen Institutes of Advanced Technology >>> Chinese Academy of Sciences >>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >>> Guangdong (518055), P. R. China >>> E-mail: rl.chen at siat.ac.cn >>> Phone: +86-755-86392312 >>> >>> >>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>> >>>> Use VecGetArrayRead instead of VecGetArray >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>> Read.html >>>> >>>> >>>> Shri >>>> >>>> -----Original Message----- >>>> From: Rongliang Chen >>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>> To: PETSc users list , " >>>> rongliang.chan at gmail.com" >>>> >>>> Subject: [petsc-users] Vec is locked read only >>>> >>>> Dear All, >>>>> >>>>> My code got the following error messages, but the code works well for >>>>> the petsc optimized version (--with-debugging=0). Anyone can tell me >>>>> how >>>>> to fix this problem? >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat Apr >>>>> 2 15:40:32 2016 >>>>> [0]PETSC ERROR: Configure options --download-blacs --download-scalapack >>>>> --download-metis --download-parmetis --download-exodusii >>>>> --download-netcdf --download-hdf5 >>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>>> >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>> unctions.c >>>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>>> >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>> unctions.c >>>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>>> >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>> c >>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>> >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>> c >>>>> >>>>> >>>>> >>>> >>> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 07:37:14 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 20:37:14 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> Message-ID: <56FFBCFA.7030401@siat.ac.cn> Hi Matt, How to check in the explicit method? This kind of errors does come out in the explicit method. What do you mean "You should never be changing the input guess"? Do you mean that the value of the solution vector X in FormFunction are not allowed to be changed by the user defined function? Best regards, Rongliang On 04/02/2016 08:19 PM, Matthew Knepley wrote: > On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen > wrote: > > Hi Matt, > > That's right. The FormMassTimeStep() is inside the FormFunction() > and I call the FormFunction() in this way: > SNESSetFunction(user->snes, algebra->f, FormFunction, > (void*)user). How can I know the X is read-only or not? > > > Exactly that you get this error message, and it should be in the > documentation. > > If I do not use the SNESSolve, that is I use the explicit method > with the same function FormMassTimeStep(), this error disappeared. > So it seems that the SNESSolve locked the vector X. > > > Then we need to put that check in the explicit method as well. You > should never be changing the input guess. > > Thanks, > > Matt > > Best regards, > Rongliang > > On 04/02/2016 07:59 PM, Matthew Knepley wrote: >> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >> > wrote: >> >> Hi Dave, >> >> I did not call any VecGetArrary and VecGetArraryRead in >> function FormMassTimeStepFunction(). >> >> >> It looks like you have a function FormMassTimeStep() inside your >> FormFunction() method. This >> has an argument X, the solution guess, which is read-only. >> However, you are calling DMLocalToGlobal() >> using that X, which is illegal, since its read-only. Is this right? >> >> Matt >> >> Best regards, >> Rongliang >> >> On 04/02/2016 06:20 PM, Dave May wrote: >>> >>> >>> On 2 April 2016 at 11:18, Rongliang Chen >> > wrote: >>> >>> Hi Shri, >>> >>> Thanks for your reply. >>> >>> Do you mean that I need to change the VecGetArrary() in >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to >>> VecGetArrayRead()? >>> >>> >>> No - you should change it in your function >>> FormMassTimeStepFunction(). >>> The input vector x passed into SNESComputeFunction() is read >>> only. >>> >>> >>> >>> I tried it and I got a warning when I make the petsc: >>> >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In >>> function ?DMLocalToGlobalBegin?: >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >>> warning: passing argument 2 of ?VecGetArrayRead? from >>> incompatible pointer type [enabled by default] >>> In file included from >>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>> from >>> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >>> note: expected ?const PetscScalar **? but argument is of >>> type ?PetscScalar **? >>> >>> Best regards, >>> Rongliang >>> >>> --------------------- >>> Rongliang Chen, PhD >>> Associate Professor >>> >>> Laboratory for Engineering and Scientific Computing >>> Shenzhen Institutes of Advanced Technology >>> Chinese Academy of Sciences >>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, >>> Shenzhen, Guangdong (518055), P. R. China >>> E-mail: rl.chen at siat.ac.cn >>> Phone: +86-755-86392312 >>> >>> >>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>> >>> Use VecGetArrayRead instead of VecGetArray >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>> Read.html >>> >>> >>> Shri >>> >>> -----Original Message----- >>> From: Rongliang Chen >> > >>> Date: Saturday, April 2, 2016 at 3:51 AM >>> To: PETSc users list >> >, >>> "rongliang.chan at gmail.com >>> " >>> >> > >>> Subject: [petsc-users] Vec is locked read only >>> >>> Dear All, >>> >>> My code got the following error messages, but >>> the code works well for >>> the petsc optimized version >>> (--with-debugging=0). Anyone can tell me how >>> to fix this problem? >>> >>> Best regards, >>> Rongliang >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Vec is locked read only, >>> argument # 1 >>> [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, >>> Dec, 03, 2015 >>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named >>> rlchen by rlchen Sat Apr >>> 2 15:40:32 2016 >>> [0]PETSC ERROR: Configure options >>> --download-blacs --download-scalapack >>> --download-metis --download-parmetis >>> --download-exodusii >>> --download-netcdf --download-hdf5 >>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>> --with-debugging=1 --download-fblaslapack >>> --download-chaco >>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line >>> 1913 in >>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() >>> line 191 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>> unctions.c >>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>> unctions.c >>> [0]PETSC ERROR: #5 SNESComputeFunction() line >>> 2067 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>> c >>> [0]PETSC ERROR: #9 main() line 452 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>> c >>> >>> >>> >>> >>> >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 07:37:24 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 07:37:24 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFBCFA.7030401@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen wrote: > Hi Matt, > > How to check in the explicit method? This kind of errors does come out in > the explicit method. > You said "that is I use the explicit method with the same function FormMassTimeStep(), this error disappeared" > What do you mean "You should never be changing the input guess"? Do you > mean that the value of the solution vector X in FormFunction are not > allowed to be changed by the user defined function? > Yes. Matt > Best regards, > Rongliang > > On 04/02/2016 08:19 PM, Matthew Knepley wrote: > > On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen wrote: > >> Hi Matt, >> >> That's right. The FormMassTimeStep() is inside the FormFunction() and I >> call the FormFunction() in this way: SNESSetFunction(user->snes, >> algebra->f, FormFunction, (void*)user). How can I know the X is read-only >> or not? >> > > Exactly that you get this error message, and it should be in the > documentation. > > >> If I do not use the SNESSolve, that is I use the explicit method with the >> same function FormMassTimeStep(), this error disappeared. So it seems that >> the SNESSolve locked the vector X. >> > > Then we need to put that check in the explicit method as well. You should > never be changing the input guess. > > Thanks, > > Matt > > >> Best regards, >> Rongliang >> >> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >> >> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >> wrote: >> >>> Hi Dave, >>> >>> I did not call any VecGetArrary and VecGetArraryRead in function >>> FormMassTimeStepFunction(). >>> >> >> It looks like you have a function FormMassTimeStep() inside your >> FormFunction() method. This >> has an argument X, the solution guess, which is read-only. However, you >> are calling DMLocalToGlobal() >> using that X, which is illegal, since its read-only. Is this right? >> >> Matt >> >> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 06:20 PM, Dave May wrote: >>> >>> >>> >>> On 2 April 2016 at 11:18, Rongliang Chen wrote: >>> >>>> Hi Shri, >>>> >>>> Thanks for your reply. >>>> >>>> Do you mean that I need to change the VecGetArrary() in >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >>>> >>> >>> No - you should change it in your function FormMassTimeStepFunction(). >>> The input vector x passed into SNESComputeFunction() is read only. >>> >>> >>> >>>> I tried it and I got a warning when I make the petsc: >>>> >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >>>> ?DMLocalToGlobalBegin?: >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >>>> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >>>> [enabled by default] >>>> In file included from >>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: expected >>>> ?const PetscScalar **? but argument is of type ?PetscScalar **? >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> --------------------- >>>> Rongliang Chen, PhD >>>> Associate Professor >>>> >>>> Laboratory for Engineering and Scientific Computing >>>> Shenzhen Institutes of Advanced Technology >>>> Chinese Academy of Sciences >>>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >>>> Guangdong (518055), P. R. China >>>> E-mail: rl.chen at siat.ac.cn >>>> Phone: +86-755-86392312 >>>> >>>> >>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>>> >>>>> Use VecGetArrayRead instead of VecGetArray >>>>> >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>> Read.html >>>>> >>>>> >>>>> Shri >>>>> >>>>> -----Original Message----- >>>>> From: Rongliang Chen >>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>> To: PETSc users list , " >>>>> rongliang.chan at gmail.com" >>>>> >>>>> Subject: [petsc-users] Vec is locked read only >>>>> >>>>> Dear All, >>>>>> >>>>>> My code got the following error messages, but the code works well for >>>>>> the petsc optimized version (--with-debugging=0). Anyone can tell me >>>>>> how >>>>>> to fix this problem? >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>> [0]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat >>>>>> Apr >>>>>> 2 15:40:32 2016 >>>>>> [0]PETSC ERROR: Configure options --download-blacs >>>>>> --download-scalapack >>>>>> --download-metis --download-parmetis --download-exodusii >>>>>> --download-netcdf --download-hdf5 >>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>>>> >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>> unctions.c >>>>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>>>> >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>> unctions.c >>>>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>>>> >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>> c >>>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>>> >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>> c >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 07:46:26 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 20:46:26 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> Message-ID: <56FFBF22.60407@siat.ac.cn> But I need to setup the boundary conditions for the solution vector X, which is in FormMassTimeStep(). Best regards, Rongliang On 04/02/2016 08:37 PM, Matthew Knepley wrote: > On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen > wrote: > > Hi Matt, > > How to check in the explicit method? This kind of errors does come > out in the explicit method. > > > You said "that is I use the explicit method with the same function > FormMassTimeStep(), this error disappeared" > > What do you mean "You should never be changing the input guess"? > Do you mean that the value of the solution vector X in > FormFunction are not allowed to be changed by the user defined > function? > > > Yes. > > Matt > > Best regards, > Rongliang > > On 04/02/2016 08:19 PM, Matthew Knepley wrote: >> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >> > wrote: >> >> Hi Matt, >> >> That's right. The FormMassTimeStep() is inside the >> FormFunction() and I call the FormFunction() in this way: >> SNESSetFunction(user->snes, algebra->f, FormFunction, >> (void*)user). How can I know the X is read-only or not? >> >> >> Exactly that you get this error message, and it should be in the >> documentation. >> >> If I do not use the SNESSolve, that is I use the explicit >> method with the same function FormMassTimeStep(), this error >> disappeared. So it seems that the SNESSolve locked the vector X. >> >> >> Then we need to put that check in the explicit method as well. >> You should never be changing the input guess. >> >> Thanks, >> >> Matt >> >> Best regards, >> Rongliang >> >> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>> > wrote: >>> >>> Hi Dave, >>> >>> I did not call any VecGetArrary and VecGetArraryRead in >>> function FormMassTimeStepFunction(). >>> >>> >>> It looks like you have a function FormMassTimeStep() inside >>> your FormFunction() method. This >>> has an argument X, the solution guess, which is read-only. >>> However, you are calling DMLocalToGlobal() >>> using that X, which is illegal, since its read-only. Is this >>> right? >>> >>> Matt >>> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 06:20 PM, Dave May wrote: >>>> >>>> >>>> On 2 April 2016 at 11:18, Rongliang Chen >>>> > wrote: >>>> >>>> Hi Shri, >>>> >>>> Thanks for your reply. >>>> >>>> Do you mean that I need to change the >>>> VecGetArrary() in >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>> to VecGetArrayRead()? >>>> >>>> >>>> No - you should change it in your function >>>> FormMassTimeStepFunction(). >>>> The input vector x passed into SNESComputeFunction() is >>>> read only. >>>> >>>> >>>> >>>> I tried it and I got a warning when I make the petsc: >>>> >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In >>>> function ?DMLocalToGlobalBegin?: >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >>>> warning: passing argument 2 of ?VecGetArrayRead? >>>> from incompatible pointer type [enabled by default] >>>> In file included from >>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>> from >>>> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>> from >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >>>> note: expected ?const PetscScalar **? but argument >>>> is of type ?PetscScalar **? >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> --------------------- >>>> Rongliang Chen, PhD >>>> Associate Professor >>>> >>>> Laboratory for Engineering and Scientific Computing >>>> Shenzhen Institutes of Advanced Technology >>>> Chinese Academy of Sciences >>>> Address: 1068 Xueyuan Avenue, Shenzhen University >>>> Town, Shenzhen, Guangdong (518055), P. R. China >>>> E-mail: rl.chen at siat.ac.cn >>>> Phone: +86-755-86392312 >>>> >>>> >>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>>> >>>> Use VecGetArrayRead instead of VecGetArray >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>> Read.html >>>> >>>> >>>> Shri >>>> >>>> -----Original Message----- >>>> From: Rongliang Chen >>> > >>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>> To: PETSc users list >>> >, >>>> "rongliang.chan at gmail.com >>>> " >>>> >>> > >>>> Subject: [petsc-users] Vec is locked read only >>>> >>>> Dear All, >>>> >>>> My code got the following error messages, >>>> but the code works well for >>>> the petsc optimized version >>>> (--with-debugging=0). Anyone can tell me how >>>> to fix this problem? >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error >>>> Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec is locked read only, >>>> argument # 1 >>>> [0]PETSC ERROR: See >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version >>>> 3.6.3, Dec, 03, 2015 >>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug >>>> named rlchen by rlchen Sat Apr >>>> 2 15:40:32 2016 >>>> [0]PETSC ERROR: Configure options >>>> --download-blacs --download-scalapack >>>> --download-metis --download-parmetis >>>> --download-exodusii >>>> --download-netcdf --download-hdf5 >>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>> --with-debugging=1 --download-fblaslapack >>>> --download-chaco >>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() >>>> line 1913 in >>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>> [0]PETSC ERROR: #3 >>>> FormMassTimeStepFunction() line 191 in >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>> unctions.c >>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>> unctions.c >>>> [0]PETSC ERROR: #5 SNESComputeFunction() >>>> line 2067 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() >>>> line 184 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #8 SolveTimeDependent() >>>> line 843 in >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>> c >>>> [0]PETSC ERROR: #9 main() line 452 in >>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>> c >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 07:49:43 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 07:49:43 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFBF22.60407@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen wrote: > But I need to setup the boundary conditions for the solution vector X, > which is in FormMassTimeStep(). > Make a copy. The solution vector is used in the solver, and you have no idea exactly what it is doing with it. Or put your boundary conditions in the initial guess to SNESSolve(). Matt > Best regards, > Rongliang > > > On 04/02/2016 08:37 PM, Matthew Knepley wrote: > > On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen wrote: > >> Hi Matt, >> >> How to check in the explicit method? This kind of errors does come out in >> the explicit method. >> > > You said "that is I use the explicit method with the same function > FormMassTimeStep(), this error disappeared" > > >> What do you mean "You should never be changing the input guess"? Do you >> mean that the value of the solution vector X in FormFunction are not >> allowed to be changed by the user defined function? >> > > Yes. > > Matt > > >> Best regards, >> Rongliang >> >> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >> >> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >> wrote: >> >>> Hi Matt, >>> >>> That's right. The FormMassTimeStep() is inside the FormFunction() and I >>> call the FormFunction() in this way: SNESSetFunction(user->snes, >>> algebra->f, FormFunction, (void*)user). How can I know the X is read-only >>> or not? >>> >> >> Exactly that you get this error message, and it should be in the >> documentation. >> >> >>> If I do not use the SNESSolve, that is I use the explicit method with >>> the same function FormMassTimeStep(), this error disappeared. So it seems >>> that the SNESSolve locked the vector X. >>> >> >> Then we need to put that check in the explicit method as well. You should >> never be changing the input guess. >> >> Thanks, >> >> Matt >> >> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>> >>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>> wrote: >>> >>>> Hi Dave, >>>> >>>> I did not call any VecGetArrary and VecGetArraryRead in function >>>> FormMassTimeStepFunction(). >>>> >>> >>> It looks like you have a function FormMassTimeStep() inside your >>> FormFunction() method. This >>> has an argument X, the solution guess, which is read-only. However, you >>> are calling DMLocalToGlobal() >>> using that X, which is illegal, since its read-only. Is this right? >>> >>> Matt >>> >>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>> >>>> >>>> >>>> On 2 April 2016 at 11:18, Rongliang Chen wrote: >>>> >>>>> Hi Shri, >>>>> >>>>> Thanks for your reply. >>>>> >>>>> Do you mean that I need to change the VecGetArrary() in >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >>>>> >>>> >>>> No - you should change it in your function FormMassTimeStepFunction(). >>>> The input vector x passed into SNESComputeFunction() is read only. >>>> >>>> >>>> >>>>> I tried it and I got a warning when I make the petsc: >>>>> >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >>>>> ?DMLocalToGlobalBegin?: >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >>>>> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >>>>> [enabled by default] >>>>> In file included from >>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: >>>>> expected ?const PetscScalar **? but argument is of type ?PetscScalar **? >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> --------------------- >>>>> Rongliang Chen, PhD >>>>> Associate Professor >>>>> >>>>> Laboratory for Engineering and Scientific Computing >>>>> Shenzhen Institutes of Advanced Technology >>>>> Chinese Academy of Sciences >>>>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >>>>> Guangdong (518055), P. R. China >>>>> E-mail: rl.chen at siat.ac.cn >>>>> Phone: +86-755-86392312 >>>>> >>>>> >>>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>>>> >>>>>> Use VecGetArrayRead instead of VecGetArray >>>>>> >>>>>> >>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>>> Read.html >>>>>> >>>>>> >>>>>> Shri >>>>>> >>>>>> -----Original Message----- >>>>>> From: Rongliang Chen >>>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>>> To: PETSc users list , " >>>>>> rongliang.chan at gmail.com" >>>>>> >>>>>> Subject: [petsc-users] Vec is locked read only >>>>>> >>>>>> Dear All, >>>>>>> >>>>>>> My code got the following error messages, but the code works well for >>>>>>> the petsc optimized version (--with-debugging=0). Anyone can tell me >>>>>>> how >>>>>>> to fix this problem? >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat >>>>>>> Apr >>>>>>> 2 15:40:32 2016 >>>>>>> [0]PETSC ERROR: Configure options --download-blacs >>>>>>> --download-scalapack >>>>>>> --download-metis --download-parmetis --download-exodusii >>>>>>> --download-netcdf --download-hdf5 >>>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>>>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>>>>> >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>> unctions.c >>>>>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>>>>> >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>> unctions.c >>>>>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>>>>> >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>> c >>>>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>>>> >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>> c >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 08:03:22 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 21:03:22 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> Message-ID: <56FFC31A.5060808@siat.ac.cn> Hi Matt, Many thanks for your helps. I will change the way that I used for the boundary condition setup. Best regards, Rongliang On 04/02/2016 08:49 PM, Matthew Knepley wrote: > On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen > wrote: > > But I need to setup the boundary conditions for the solution > vector X, which is in FormMassTimeStep(). > > > Make a copy. The solution vector is used in the solver, and you have > no idea exactly what it is doing with it. > Or put your boundary conditions in the initial guess to SNESSolve(). > > Matt > > Best regards, > Rongliang > > > On 04/02/2016 08:37 PM, Matthew Knepley wrote: >> On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen >> > wrote: >> >> Hi Matt, >> >> How to check in the explicit method? This kind of errors does >> come out in the explicit method. >> >> >> You said "that is I use the explicit method with the same >> function FormMassTimeStep(), this error disappeared" >> >> What do you mean "You should never be changing the input >> guess"? Do you mean that the value of the solution vector X >> in FormFunction are not allowed to be changed by the user >> defined function? >> >> >> Yes. >> >> Matt >> >> Best regards, >> Rongliang >> >> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >>> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >>> > wrote: >>> >>> Hi Matt, >>> >>> That's right. The FormMassTimeStep() is inside the >>> FormFunction() and I call the FormFunction() in this >>> way: SNESSetFunction(user->snes, algebra->f, >>> FormFunction, (void*)user). How can I know the X is >>> read-only or not? >>> >>> >>> Exactly that you get this error message, and it should be in >>> the documentation. >>> >>> If I do not use the SNESSolve, that is I use the >>> explicit method with the same function >>> FormMassTimeStep(), this error disappeared. So it seems >>> that the SNESSolve locked the vector X. >>> >>> >>> Then we need to put that check in the explicit method as >>> well. You should never be changing the input guess. >>> >>> Thanks, >>> >>> Matt >>> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>>> > wrote: >>>> >>>> Hi Dave, >>>> >>>> I did not call any VecGetArrary and >>>> VecGetArraryRead in function >>>> FormMassTimeStepFunction(). >>>> >>>> >>>> It looks like you have a function FormMassTimeStep() >>>> inside your FormFunction() method. This >>>> has an argument X, the solution guess, which is >>>> read-only. However, you are calling DMLocalToGlobal() >>>> using that X, which is illegal, since its read-only. Is >>>> this right? >>>> >>>> Matt >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>>> >>>>> >>>>> On 2 April 2016 at 11:18, Rongliang Chen >>>>> > >>>>> wrote: >>>>> >>>>> Hi Shri, >>>>> >>>>> Thanks for your reply. >>>>> >>>>> Do you mean that I need to change the >>>>> VecGetArrary() in >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>> to VecGetArrayRead()? >>>>> >>>>> >>>>> No - you should change it in your function >>>>> FormMassTimeStepFunction(). >>>>> The input vector x passed into >>>>> SNESComputeFunction() is read only. >>>>> >>>>> >>>>> >>>>> I tried it and I got a warning when I make the >>>>> petsc: >>>>> >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: >>>>> In function ?DMLocalToGlobalBegin?: >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >>>>> warning: passing argument 2 of >>>>> ?VecGetArrayRead? from incompatible pointer >>>>> type [enabled by default] >>>>> In file included from >>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>> from >>>>> /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>> from >>>>> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>> from >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >>>>> note: expected ?const PetscScalar **? but >>>>> argument is of type ?PetscScalar **? >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> --------------------- >>>>> Rongliang Chen, PhD >>>>> Associate Professor >>>>> >>>>> Laboratory for Engineering and Scientific >>>>> Computing >>>>> Shenzhen Institutes of Advanced Technology >>>>> Chinese Academy of Sciences >>>>> Address: 1068 Xueyuan Avenue, Shenzhen >>>>> University Town, Shenzhen, Guangdong (518055), >>>>> P. R. China >>>>> E-mail: rl.chen at siat.ac.cn >>>>> >>>>> Phone: +86-755-86392312 >>>>> >>>>> >>>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. >>>>> wrote: >>>>> >>>>> Use VecGetArrayRead instead of VecGetArray >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>> Read.html >>>>> >>>>> >>>>> Shri >>>>> >>>>> -----Original Message----- >>>>> From: Rongliang Chen >>>> > >>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>> To: PETSc users list >>>>> >>>> >, >>>>> "rongliang.chan at gmail.com >>>>> " >>>>> >>>> > >>>>> Subject: [petsc-users] Vec is locked read only >>>>> >>>>> Dear All, >>>>> >>>>> My code got the following error >>>>> messages, but the code works well for >>>>> the petsc optimized version >>>>> (--with-debugging=0). Anyone can tell >>>>> me how >>>>> to fix this problem? >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> >>>>> [0]PETSC ERROR: --------------------- >>>>> Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Vec is locked read >>>>> only, argument # 1 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version >>>>> 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./Nwtun on a >>>>> 64bit-debug named rlchen by rlchen Sat Apr >>>>> 2 15:40:32 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --download-blacs --download-scalapack >>>>> --download-metis --download-parmetis >>>>> --download-exodusii >>>>> --download-netcdf --download-hdf5 >>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>> --with-debugging=1 >>>>> --download-fblaslapack --download-chaco >>>>> [0]PETSC ERROR: #1 VecGetArray() line >>>>> 1646 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>> [0]PETSC ERROR: #2 >>>>> DMLocalToGlobalBegin() line 1913 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>> [0]PETSC ERROR: #3 >>>>> FormMassTimeStepFunction() line 191 in >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>> unctions.c >>>>> [0]PETSC ERROR: #4 FormFunction() line >>>>> 46 in >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>> unctions.c >>>>> [0]PETSC ERROR: #5 >>>>> SNESComputeFunction() line 2067 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #6 >>>>> SNESSolve_NEWTONLS() line 184 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>> [0]PETSC ERROR: #7 SNESSolve() line >>>>> 3906 in >>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #8 >>>>> SolveTimeDependent() line 843 in >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>> c >>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>> c >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 08:08:11 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 08:08:11 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFC31A.5060808@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> <56FFC31A.5060808@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 8:03 AM, Rongliang Chen wrote: > Hi Matt, > > Many thanks for your helps. I will change the way that I used for the > boundary condition setup. > I will note that we (Barry, Jed, Mark, and I at least) have adopted the view that by and large a) global vectors are for solvers b) local vectors are for assembly For me, that means local vectors have ghost values and boundary conditions, but global vectors have them removed. This is how SNES ex12 and ex62 are organized. Thanks, Matt > Best regards, > Rongliang > > > > On 04/02/2016 08:49 PM, Matthew Knepley wrote: > > On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen wrote: > >> But I need to setup the boundary conditions for the solution vector X, >> which is in FormMassTimeStep(). >> > > Make a copy. The solution vector is used in the solver, and you have no > idea exactly what it is doing with it. > Or put your boundary conditions in the initial guess to SNESSolve(). > > Matt > > >> Best regards, >> Rongliang >> >> >> On 04/02/2016 08:37 PM, Matthew Knepley wrote: >> >> On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen >> wrote: >> >>> Hi Matt, >>> >>> How to check in the explicit method? This kind of errors does come out >>> in the explicit method. >>> >> >> You said "that is I use the explicit method with the same function >> FormMassTimeStep(), this error disappeared" >> >> >>> What do you mean "You should never be changing the input guess"? Do you >>> mean that the value of the solution vector X in FormFunction are not >>> allowed to be changed by the user defined function? >>> >> >> Yes. >> >> Matt >> >> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >>> >>> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >>> wrote: >>> >>>> Hi Matt, >>>> >>>> That's right. The FormMassTimeStep() is inside the FormFunction() and I >>>> call the FormFunction() in this way: SNESSetFunction(user->snes, >>>> algebra->f, FormFunction, (void*)user). How can I know the X is read-only >>>> or not? >>>> >>> >>> Exactly that you get this error message, and it should be in the >>> documentation. >>> >>> >>>> If I do not use the SNESSolve, that is I use the explicit method with >>>> the same function FormMassTimeStep(), this error disappeared. So it seems >>>> that the SNESSolve locked the vector X. >>>> >>> >>> Then we need to put that check in the explicit method as well. You >>> should never be changing the input guess. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>>> >>>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>>> wrote: >>>> >>>>> Hi Dave, >>>>> >>>>> I did not call any VecGetArrary and VecGetArraryRead in function >>>>> FormMassTimeStepFunction(). >>>>> >>>> >>>> It looks like you have a function FormMassTimeStep() inside your >>>> FormFunction() method. This >>>> has an argument X, the solution guess, which is read-only. However, you >>>> are calling DMLocalToGlobal() >>>> using that X, which is illegal, since its read-only. Is this right? >>>> >>>> Matt >>>> >>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>>> >>>>> >>>>> >>>>> On 2 April 2016 at 11:18, Rongliang Chen wrote: >>>>> >>>>>> Hi Shri, >>>>>> >>>>>> Thanks for your reply. >>>>>> >>>>>> Do you mean that I need to change the VecGetArrary() in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >>>>>> >>>>> >>>>> No - you should change it in your function FormMassTimeStepFunction(). >>>>> The input vector x passed into SNESComputeFunction() is read only. >>>>> >>>>> >>>>> >>>>>> I tried it and I got a warning when I make the petsc: >>>>>> >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >>>>>> ?DMLocalToGlobalBegin?: >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >>>>>> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >>>>>> [enabled by default] >>>>>> In file included from >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>>> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: >>>>>> expected ?const PetscScalar **? but argument is of type ?PetscScalar **? >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> --------------------- >>>>>> Rongliang Chen, PhD >>>>>> Associate Professor >>>>>> >>>>>> Laboratory for Engineering and Scientific Computing >>>>>> Shenzhen Institutes of Advanced Technology >>>>>> Chinese Academy of Sciences >>>>>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >>>>>> Guangdong (518055), P. R. China >>>>>> E-mail: rl.chen at siat.ac.cn >>>>>> Phone: +86-755-86392312 >>>>>> >>>>>> >>>>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>>>>> >>>>>>> Use VecGetArrayRead instead of VecGetArray >>>>>>> >>>>>>> >>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>>>> Read.html >>>>>>> >>>>>>> >>>>>>> Shri >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Rongliang Chen >>>>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>>>> To: PETSc users list , " >>>>>>> rongliang.chan at gmail.com" >>>>>>> >>>>>>> Subject: [petsc-users] Vec is locked read only >>>>>>> >>>>>>> Dear All, >>>>>>>> >>>>>>>> My code got the following error messages, but the code works well >>>>>>>> for >>>>>>>> the petsc optimized version (--with-debugging=0). Anyone can tell >>>>>>>> me how >>>>>>>> to fix this problem? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Rongliang >>>>>>>> >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>>> [0]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen Sat >>>>>>>> Apr >>>>>>>> 2 15:40:32 2016 >>>>>>>> [0]PETSC ERROR: Configure options --download-blacs >>>>>>>> --download-scalapack >>>>>>>> --download-metis --download-parmetis --download-exodusii >>>>>>>> --download-netcdf --download-hdf5 >>>>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>>>>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>>>>>> >>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>>> unctions.c >>>>>>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>>>>>> >>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>>> unctions.c >>>>>>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>>>>>> >>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>>> c >>>>>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>>>>> >>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>>> c >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 08:18:55 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 21:18:55 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> <56FFC31A.5060808@siat.ac.cn> Message-ID: <56FFC6BF.20200@siat.ac.cn> On 04/02/2016 09:08 PM, Matthew Knepley wrote: > On Sat, Apr 2, 2016 at 8:03 AM, Rongliang Chen > wrote: > > Hi Matt, > > Many thanks for your helps. I will change the way that I used for > the boundary condition setup. > > > I will note that we (Barry, Jed, Mark, and I at least) have adopted > the view that by and large > > a) global vectors are for solvers > > b) local vectors are for assembly > > For me, that means local vectors have ghost values and boundary > conditions, but global vectors > have them removed. This is how SNES ex12 and ex62 are organized. > Does this means that if I want to use the local vector (obtained by DMGlobalToLocalBegin and DMGlobalToLocalEnd), I need to reset the boundary conditions for the local vector before I use it? Best regards, Rongliang > Thanks, > > Matt > > Best regards, > Rongliang > > > > On 04/02/2016 08:49 PM, Matthew Knepley wrote: >> On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen >> > wrote: >> >> But I need to setup the boundary conditions for the solution >> vector X, which is in FormMassTimeStep(). >> >> >> Make a copy. The solution vector is used in the solver, and you >> have no idea exactly what it is doing with it. >> Or put your boundary conditions in the initial guess to SNESSolve(). >> >> Matt >> >> Best regards, >> Rongliang >> >> >> On 04/02/2016 08:37 PM, Matthew Knepley wrote: >>> On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen >>> > wrote: >>> >>> Hi Matt, >>> >>> How to check in the explicit method? This kind of errors >>> does come out in the explicit method. >>> >>> >>> You said "that is I use the explicit method with the same >>> function FormMassTimeStep(), this error disappeared" >>> >>> What do you mean "You should never be changing the input >>> guess"? Do you mean that the value of the solution >>> vector X in FormFunction are not allowed to be changed >>> by the user defined function? >>> >>> >>> Yes. >>> >>> Matt >>> >>> Best regards, >>> Rongliang >>> >>> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >>>> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >>>> > wrote: >>>> >>>> Hi Matt, >>>> >>>> That's right. The FormMassTimeStep() is inside the >>>> FormFunction() and I call the FormFunction() in >>>> this way: SNESSetFunction(user->snes, algebra->f, >>>> FormFunction, (void*)user). How can I know the X is >>>> read-only or not? >>>> >>>> >>>> Exactly that you get this error message, and it should >>>> be in the documentation. >>>> >>>> If I do not use the SNESSolve, that is I use the >>>> explicit method with the same function >>>> FormMassTimeStep(), this error disappeared. So it >>>> seems that the SNESSolve locked the vector X. >>>> >>>> >>>> Then we need to put that check in the explicit method >>>> as well. You should never be changing the input guess. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>>>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>>>> > >>>>> wrote: >>>>> >>>>> Hi Dave, >>>>> >>>>> I did not call any VecGetArrary and >>>>> VecGetArraryRead in function >>>>> FormMassTimeStepFunction(). >>>>> >>>>> >>>>> It looks like you have a function >>>>> FormMassTimeStep() inside your FormFunction() >>>>> method. This >>>>> has an argument X, the solution guess, which is >>>>> read-only. However, you are calling DMLocalToGlobal() >>>>> using that X, which is illegal, since its >>>>> read-only. Is this right? >>>>> >>>>> Matt >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>>>> >>>>>> >>>>>> On 2 April 2016 at 11:18, Rongliang Chen >>>>>> >>>>> > wrote: >>>>>> >>>>>> Hi Shri, >>>>>> >>>>>> Thanks for your reply. >>>>>> >>>>>> Do you mean that I need to change the >>>>>> VecGetArrary() in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>> to VecGetArrayRead()? >>>>>> >>>>>> >>>>>> No - you should change it in your function >>>>>> FormMassTimeStepFunction(). >>>>>> The input vector x passed into >>>>>> SNESComputeFunction() is read only. >>>>>> >>>>>> >>>>>> >>>>>> I tried it and I got a warning when I >>>>>> make the petsc: >>>>>> >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: >>>>>> In function ?DMLocalToGlobalBegin?: >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >>>>>> warning: passing argument 2 of >>>>>> ?VecGetArrayRead? from incompatible >>>>>> pointer type [enabled by default] >>>>>> In file included from >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>>> from >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>>> from >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>>> from >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >>>>>> note: expected ?const PetscScalar **? but >>>>>> argument is of type ?PetscScalar **? >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> --------------------- >>>>>> Rongliang Chen, PhD >>>>>> Associate Professor >>>>>> >>>>>> Laboratory for Engineering and Scientific >>>>>> Computing >>>>>> Shenzhen Institutes of Advanced Technology >>>>>> Chinese Academy of Sciences >>>>>> Address: 1068 Xueyuan Avenue, Shenzhen >>>>>> University Town, Shenzhen, Guangdong >>>>>> (518055), P. R. China >>>>>> E-mail: rl.chen at siat.ac.cn >>>>>> >>>>>> Phone: +86-755-86392312 >>>>>> >>>>>> >>>>>> >>>>>> On 04/02/2016 05:03 PM, Abhyankar, >>>>>> Shrirang G. wrote: >>>>>> >>>>>> Use VecGetArrayRead instead of >>>>>> VecGetArray >>>>>> >>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>>> Read.html >>>>>> >>>>>> >>>>>> Shri >>>>>> >>>>>> -----Original Message----- >>>>>> From: Rongliang Chen >>>>>> >>>>> > >>>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>>> To: PETSc users list >>>>>> >>>>> >, >>>>>> "rongliang.chan at gmail.com >>>>>> " >>>>>> >>>>> > >>>>>> Subject: [petsc-users] Vec is locked >>>>>> read only >>>>>> >>>>>> Dear All, >>>>>> >>>>>> My code got the following error >>>>>> messages, but the code works well for >>>>>> the petsc optimized version >>>>>> (--with-debugging=0). Anyone can >>>>>> tell me how >>>>>> to fix this problem? >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> >>>>>> [0]PETSC ERROR: >>>>>> --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Object is in >>>>>> wrong state >>>>>> [0]PETSC ERROR: Vec is locked >>>>>> read only, argument # 1 >>>>>> [0]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release >>>>>> Version 3.6.3, Dec, 03, 2015 >>>>>> [0]PETSC ERROR: ./Nwtun on a >>>>>> 64bit-debug named rlchen by >>>>>> rlchen Sat Apr >>>>>> 2 15:40:32 2016 >>>>>> [0]PETSC ERROR: Configure options >>>>>> --download-blacs --download-scalapack >>>>>> --download-metis >>>>>> --download-parmetis >>>>>> --download-exodusii >>>>>> --download-netcdf --download-hdf5 >>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>> --with-debugging=1 >>>>>> --download-fblaslapack >>>>>> --download-chaco >>>>>> [0]PETSC ERROR: #1 VecGetArray() >>>>>> line 1646 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>> [0]PETSC ERROR: #2 >>>>>> DMLocalToGlobalBegin() line 1913 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #3 >>>>>> FormMassTimeStepFunction() line >>>>>> 191 in >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>> unctions.c >>>>>> [0]PETSC ERROR: #4 FormFunction() >>>>>> line 46 in >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>> unctions.c >>>>>> [0]PETSC ERROR: #5 >>>>>> SNESComputeFunction() line 2067 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #6 >>>>>> SNESSolve_NEWTONLS() line 184 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>> [0]PETSC ERROR: #7 SNESSolve() >>>>>> line 3906 in >>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #8 >>>>>> SolveTimeDependent() line 843 in >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>> c >>>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>> c >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before >>>>> they begin their experiments is infinitely more >>>>> interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 08:22:37 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 08:22:37 -0500 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFC6BF.20200@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> <56FFC31A.5060808@siat.ac.cn> <56FFC6BF.20200@siat.ac.cn> Message-ID: On Sat, Apr 2, 2016 at 8:18 AM, Rongliang Chen wrote: > On 04/02/2016 09:08 PM, Matthew Knepley wrote: > > On Sat, Apr 2, 2016 at 8:03 AM, Rongliang Chen wrote: > >> Hi Matt, >> >> Many thanks for your helps. I will change the way that I used for the >> boundary condition setup. >> > > I will note that we (Barry, Jed, Mark, and I at least) have adopted the > view that by and large > > a) global vectors are for solvers > > b) local vectors are for assembly > > For me, that means local vectors have ghost values and boundary > conditions, but global vectors > have them removed. This is how SNES ex12 and ex62 are organized. > > Does this means that if I want to use the local vector (obtained by > DMGlobalToLocalBegin and DMGlobalToLocalEnd), I need to reset the boundary > conditions for the local vector before I use it? > Yes, that is how PETSc is setup internally. 1) The solver generates a new initial guess 2) PETSc calls DMGlobalToLocal(). In Plex, this adds in ghosts and BC 3) The local guess is passed to FormFunctionLocal() Thanks, Matt > Best regards, > Rongliang > > Thanks, > > Matt > > >> Best regards, >> Rongliang >> >> >> >> On 04/02/2016 08:49 PM, Matthew Knepley wrote: >> >> On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen >> wrote: >> >>> But I need to setup the boundary conditions for the solution vector X, >>> which is in FormMassTimeStep(). >>> >> >> Make a copy. The solution vector is used in the solver, and you have no >> idea exactly what it is doing with it. >> Or put your boundary conditions in the initial guess to SNESSolve(). >> >> Matt >> >> >>> Best regards, >>> Rongliang >>> >>> >>> On 04/02/2016 08:37 PM, Matthew Knepley wrote: >>> >>> On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen >>> wrote: >>> >>>> Hi Matt, >>>> >>>> How to check in the explicit method? This kind of errors does come out >>>> in the explicit method. >>>> >>> >>> You said "that is I use the explicit method with the same function >>> FormMassTimeStep(), this error disappeared" >>> >>> >>>> What do you mean "You should never be changing the input guess"? Do you >>>> mean that the value of the solution vector X in FormFunction are not >>>> allowed to be changed by the user defined function? >>>> >>> >>> Yes. >>> >>> Matt >>> >>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >>>> >>>> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >>>> wrote: >>>> >>>>> Hi Matt, >>>>> >>>>> That's right. The FormMassTimeStep() is inside the FormFunction() and >>>>> I call the FormFunction() in this way: SNESSetFunction(user->snes, >>>>> algebra->f, FormFunction, (void*)user). How can I know the X is read-only >>>>> or not? >>>>> >>>> >>>> Exactly that you get this error message, and it should be in the >>>> documentation. >>>> >>>> >>>>> If I do not use the SNESSolve, that is I use the explicit method with >>>>> the same function FormMassTimeStep(), this error disappeared. So it seems >>>>> that the SNESSolve locked the vector X. >>>>> >>>> >>>> Then we need to put that check in the explicit method as well. You >>>> should never be changing the input guess. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>>>> >>>>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>>>> wrote: >>>>> >>>>>> Hi Dave, >>>>>> >>>>>> I did not call any VecGetArrary and VecGetArraryRead in function >>>>>> FormMassTimeStepFunction(). >>>>>> >>>>> >>>>> It looks like you have a function FormMassTimeStep() inside your >>>>> FormFunction() method. This >>>>> has an argument X, the solution guess, which is read-only. However, >>>>> you are calling DMLocalToGlobal() >>>>> using that X, which is illegal, since its read-only. Is this right? >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 2 April 2016 at 11:18, Rongliang Chen wrote: >>>>>> >>>>>>> Hi Shri, >>>>>>> >>>>>>> Thanks for your reply. >>>>>>> >>>>>>> Do you mean that I need to change the VecGetArrary() in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c to VecGetArrayRead()? >>>>>>> >>>>>> >>>>>> No - you should change it in your function FormMassTimeStepFunction(). >>>>>> The input vector x passed into SNESComputeFunction() is read only. >>>>>> >>>>>> >>>>>> >>>>>>> I tried it and I got a warning when I make the petsc: >>>>>>> >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: In function >>>>>>> ?DMLocalToGlobalBegin?: >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: warning: >>>>>>> passing argument 2 of ?VecGetArrayRead? from incompatible pointer type >>>>>>> [enabled by default] >>>>>>> In file included from >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>>>> from /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>>>> from /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>>>> from /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: note: >>>>>>> expected ?const PetscScalar **? but argument is of type ?PetscScalar **? >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>>>> --------------------- >>>>>>> Rongliang Chen, PhD >>>>>>> Associate Professor >>>>>>> >>>>>>> Laboratory for Engineering and Scientific Computing >>>>>>> Shenzhen Institutes of Advanced Technology >>>>>>> Chinese Academy of Sciences >>>>>>> Address: 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, >>>>>>> Guangdong (518055), P. R. China >>>>>>> E-mail: rl.chen at siat.ac.cn >>>>>>> Phone: +86-755-86392312 >>>>>>> >>>>>>> >>>>>>> On 04/02/2016 05:03 PM, Abhyankar, Shrirang G. wrote: >>>>>>> >>>>>>>> Use VecGetArrayRead instead of VecGetArray >>>>>>>> >>>>>>>> >>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>>>>> Read.html >>>>>>>> >>>>>>>> >>>>>>>> Shri >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Rongliang Chen >>>>>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>>>>> To: PETSc users list , " >>>>>>>> rongliang.chan at gmail.com" >>>>>>>> >>>>>>>> Subject: [petsc-users] Vec is locked read only >>>>>>>> >>>>>>>> Dear All, >>>>>>>>> >>>>>>>>> My code got the following error messages, but the code works well >>>>>>>>> for >>>>>>>>> the petsc optimized version (--with-debugging=0). Anyone can tell >>>>>>>>> me how >>>>>>>>> to fix this problem? >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Rongliang >>>>>>>>> >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>>>> [0]PETSC ERROR: See >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>>>> [0]PETSC ERROR: ./Nwtun on a 64bit-debug named rlchen by rlchen >>>>>>>>> Sat Apr >>>>>>>>> 2 15:40:32 2016 >>>>>>>>> [0]PETSC ERROR: Configure options --download-blacs >>>>>>>>> --download-scalapack >>>>>>>>> --download-metis --download-parmetis --download-exodusii >>>>>>>>> --download-netcdf --download-hdf5 >>>>>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>>>>> --with-debugging=1 --download-fblaslapack --download-chaco >>>>>>>>> [0]PETSC ERROR: #1 VecGetArray() line 1646 in >>>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>>>>> [0]PETSC ERROR: #2 DMLocalToGlobalBegin() line 1913 in >>>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>>>>> [0]PETSC ERROR: #3 FormMassTimeStepFunction() line 191 in >>>>>>>>> >>>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>>>> unctions.c >>>>>>>>> [0]PETSC ERROR: #4 FormFunction() line 46 in >>>>>>>>> >>>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>>>> unctions.c >>>>>>>>> [0]PETSC ERROR: #5 SNESComputeFunction() line 2067 in >>>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>>>> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 184 in >>>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>>>>> [0]PETSC ERROR: #7 SNESSolve() line 3906 in >>>>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>>>> [0]PETSC ERROR: #8 SolveTimeDependent() line 843 in >>>>>>>>> >>>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>>>> c >>>>>>>>> [0]PETSC ERROR: #9 main() line 452 in >>>>>>>>> >>>>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>>>> c >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rl.chen at siat.ac.cn Sat Apr 2 08:26:34 2016 From: rl.chen at siat.ac.cn (Rongliang Chen) Date: Sat, 02 Apr 2016 21:26:34 +0800 Subject: [petsc-users] Vec is locked read only In-Reply-To: <56FFC6BF.20200@siat.ac.cn> References: <56FF881C.5050307@siat.ac.cn> <56FF8E75.9040106@siat.ac.cn> <56FFB1E2.3020801@siat.ac.cn> <56FFB7EB.1060700@siat.ac.cn> <56FFBCFA.7030401@siat.ac.cn> <56FFBF22.60407@siat.ac.cn> <56FFC31A.5060808@siat.ac.cn> <56FFC6BF.20200@siat.ac.cn> Message-ID: <56FFC88A.3050902@siat.ac.cn> Got it. Thanks Matt. Best regards, Rongliang On 04/02/2016 09:18 PM, Rongliang Chen wrote: > On 04/02/2016 09:08 PM, Matthew Knepley wrote: >> On Sat, Apr 2, 2016 at 8:03 AM, Rongliang Chen > > wrote: >> >> Hi Matt, >> >> Many thanks for your helps. I will change the way that I used for >> the boundary condition setup. >> >> >> I will note that we (Barry, Jed, Mark, and I at least) have adopted >> the view that by and large >> >> a) global vectors are for solvers >> >> b) local vectors are for assembly >> >> For me, that means local vectors have ghost values and boundary >> conditions, but global vectors >> have them removed. This is how SNES ex12 and ex62 are organized. >> > Does this means that if I want to use the local vector (obtained by > DMGlobalToLocalBegin and DMGlobalToLocalEnd), I need to reset the > boundary conditions for the local vector before I use it? > > Best regards, > Rongliang > >> Thanks, >> >> Matt >> >> Best regards, >> Rongliang >> >> >> >> On 04/02/2016 08:49 PM, Matthew Knepley wrote: >>> On Sat, Apr 2, 2016 at 7:46 AM, Rongliang Chen >>> > wrote: >>> >>> But I need to setup the boundary conditions for the solution >>> vector X, which is in FormMassTimeStep(). >>> >>> >>> Make a copy. The solution vector is used in the solver, and you >>> have no idea exactly what it is doing with it. >>> Or put your boundary conditions in the initial guess to SNESSolve(). >>> >>> Matt >>> >>> Best regards, >>> Rongliang >>> >>> >>> On 04/02/2016 08:37 PM, Matthew Knepley wrote: >>>> On Sat, Apr 2, 2016 at 7:37 AM, Rongliang Chen >>>> > wrote: >>>> >>>> Hi Matt, >>>> >>>> How to check in the explicit method? This kind of >>>> errors does come out in the explicit method. >>>> >>>> >>>> You said "that is I use the explicit method with the same >>>> function FormMassTimeStep(), this error disappeared" >>>> >>>> What do you mean "You should never be changing the >>>> input guess"? Do you mean that the value of the >>>> solution vector X in FormFunction are not allowed to be >>>> changed by the user defined function? >>>> >>>> >>>> Yes. >>>> >>>> Matt >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 04/02/2016 08:19 PM, Matthew Knepley wrote: >>>>> On Sat, Apr 2, 2016 at 7:15 AM, Rongliang Chen >>>>> > wrote: >>>>> >>>>> Hi Matt, >>>>> >>>>> That's right. The FormMassTimeStep() is inside the >>>>> FormFunction() and I call the FormFunction() in >>>>> this way: SNESSetFunction(user->snes, algebra->f, >>>>> FormFunction, (void*)user). How can I know the X >>>>> is read-only or not? >>>>> >>>>> >>>>> Exactly that you get this error message, and it should >>>>> be in the documentation. >>>>> >>>>> If I do not use the SNESSolve, that is I use the >>>>> explicit method with the same function >>>>> FormMassTimeStep(), this error disappeared. So it >>>>> seems that the SNESSolve locked the vector X. >>>>> >>>>> >>>>> Then we need to put that check in the explicit method >>>>> as well. You should never be changing the input guess. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> On 04/02/2016 07:59 PM, Matthew Knepley wrote: >>>>>> On Sat, Apr 2, 2016 at 6:49 AM, Rongliang Chen >>>>>> > >>>>>> wrote: >>>>>> >>>>>> Hi Dave, >>>>>> >>>>>> I did not call any VecGetArrary and >>>>>> VecGetArraryRead in function >>>>>> FormMassTimeStepFunction(). >>>>>> >>>>>> >>>>>> It looks like you have a function >>>>>> FormMassTimeStep() inside your FormFunction() >>>>>> method. This >>>>>> has an argument X, the solution guess, which is >>>>>> read-only. However, you are calling DMLocalToGlobal() >>>>>> using that X, which is illegal, since its >>>>>> read-only. Is this right? >>>>>> >>>>>> Matt >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> On 04/02/2016 06:20 PM, Dave May wrote: >>>>>>> >>>>>>> >>>>>>> On 2 April 2016 at 11:18, Rongliang Chen >>>>>>> >>>>>> > wrote: >>>>>>> >>>>>>> Hi Shri, >>>>>>> >>>>>>> Thanks for your reply. >>>>>>> >>>>>>> Do you mean that I need to change the >>>>>>> VecGetArrary() in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>>> to VecGetArrayRead()? >>>>>>> >>>>>>> >>>>>>> No - you should change it in your function >>>>>>> FormMassTimeStepFunction(). >>>>>>> The input vector x passed into >>>>>>> SNESComputeFunction() is read only. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I tried it and I got a warning when I >>>>>>> make the petsc: >>>>>>> >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c: >>>>>>> In function ?DMLocalToGlobalBegin?: >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1913:5: >>>>>>> warning: passing argument 2 of >>>>>>> ?VecGetArrayRead? from incompatible >>>>>>> pointer type [enabled by default] >>>>>>> In file included from >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscmat.h:6:0, >>>>>>> from >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscdm.h:6, >>>>>>> from >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petsc/private/dmimpl.h:6, >>>>>>> from >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c:1: >>>>>>> /home/rlchen/soft/petsc-3.6.3/include/petscvec.h:420:29: >>>>>>> note: expected ?const PetscScalar **? >>>>>>> but argument is of type ?PetscScalar **? >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>>>> --------------------- >>>>>>> Rongliang Chen, PhD >>>>>>> Associate Professor >>>>>>> >>>>>>> Laboratory for Engineering and >>>>>>> Scientific Computing >>>>>>> Shenzhen Institutes of Advanced Technology >>>>>>> Chinese Academy of Sciences >>>>>>> Address: 1068 Xueyuan Avenue, Shenzhen >>>>>>> University Town, Shenzhen, Guangdong >>>>>>> (518055), P. R. China >>>>>>> E-mail: rl.chen at siat.ac.cn >>>>>>> >>>>>>> Phone: +86-755-86392312 >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 04/02/2016 05:03 PM, Abhyankar, >>>>>>> Shrirang G. wrote: >>>>>>> >>>>>>> Use VecGetArrayRead instead of >>>>>>> VecGetArray >>>>>>> >>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray >>>>>>> Read.html >>>>>>> >>>>>>> >>>>>>> Shri >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Rongliang Chen >>>>>>> >>>>>> > >>>>>>> Date: Saturday, April 2, 2016 at 3:51 AM >>>>>>> To: PETSc users list >>>>>>> >>>>>> >, >>>>>>> "rongliang.chan at gmail.com >>>>>>> " >>>>>>> >>>>>> > >>>>>>> Subject: [petsc-users] Vec is locked >>>>>>> read only >>>>>>> >>>>>>> Dear All, >>>>>>> >>>>>>> My code got the following error >>>>>>> messages, but the code works >>>>>>> well for >>>>>>> the petsc optimized version >>>>>>> (--with-debugging=0). Anyone can >>>>>>> tell me how >>>>>>> to fix this problem? >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>>>> >>>>>>> [0]PETSC ERROR: >>>>>>> --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Object is in >>>>>>> wrong state >>>>>>> [0]PETSC ERROR: Vec is locked >>>>>>> read only, argument # 1 >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release >>>>>>> Version 3.6.3, Dec, 03, 2015 >>>>>>> [0]PETSC ERROR: ./Nwtun on a >>>>>>> 64bit-debug named rlchen by >>>>>>> rlchen Sat Apr >>>>>>> 2 15:40:32 2016 >>>>>>> [0]PETSC ERROR: Configure >>>>>>> options --download-blacs >>>>>>> --download-scalapack >>>>>>> --download-metis >>>>>>> --download-parmetis >>>>>>> --download-exodusii >>>>>>> --download-netcdf --download-hdf5 >>>>>>> --with-mpi-dir=/home/rlchen/soft/Program/mpich2-shared >>>>>>> --with-debugging=1 >>>>>>> --download-fblaslapack >>>>>>> --download-chaco >>>>>>> [0]PETSC ERROR: #1 VecGetArray() >>>>>>> line 1646 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/vec/vec/interface/rvector.c >>>>>>> [0]PETSC ERROR: #2 >>>>>>> DMLocalToGlobalBegin() line 1913 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #3 >>>>>>> FormMassTimeStepFunction() line >>>>>>> 191 in >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>> unctions.c >>>>>>> [0]PETSC ERROR: #4 >>>>>>> FormFunction() line 46 in >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/SetupF >>>>>>> unctions.c >>>>>>> [0]PETSC ERROR: #5 >>>>>>> SNESComputeFunction() line 2067 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #6 >>>>>>> SNESSolve_NEWTONLS() line 184 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/impls/ls/ls.c >>>>>>> [0]PETSC ERROR: #7 SNESSolve() >>>>>>> line 3906 in >>>>>>> /home/rlchen/soft/petsc-3.6.3/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #8 >>>>>>> SolveTimeDependent() line 843 in >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>> c >>>>>>> [0]PETSC ERROR: #9 main() line >>>>>>> 452 in >>>>>>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.6/Nwtun. >>>>>>> c >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before >>>>>> they begin their experiments is infinitely more >>>>>> interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they >>>>> begin their experiments is infinitely more interesting >>>>> than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Sat Apr 2 17:20:12 2016 From: ysjosh.lo at gmail.com (Josh Lo) Date: Sat, 2 Apr 2016 17:20:12 -0500 Subject: [petsc-users] compile fortran module with PETSc Message-ID: Hi, I am trying to compile my main source code with module. I need PETSc in both main code and module, so I include the header file in both. it gives me some error when compiling. The error is: src/main.F:2.9: use fmodule 2 Error: Symbol 'mpi_source' at (1) conflicts with symbol from module 'fmodule', use-associated at (2) /Users/XXXXXXX/Desktop/project/petsc/petsc_3.6.3_install/include/mpif.h:10:30: PARAMETER (MPI_SOURCE=3,MPI_TAG=4,MPI_ERROR=5) 1 Error: Cannot change attributes of USE-associated symbol mpi_source at (1) it goes on and on, most relates to MPI. my main source code is like: program main USE fmodule implicit none #include "petsc/finclude/petsc.h" ....... end program main the module: module fmodule #include "petsc/finclude/petsc.h" ...... end module fmodule if the I don't include the header file in module, then it is no error, but I do need petsc in my module. How can I solve this problem? Thanks, Lo -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 2 17:23:24 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 2 Apr 2016 17:23:24 -0500 Subject: [petsc-users] compile fortran module with PETSc In-Reply-To: References: Message-ID: On Sat, Apr 2, 2016 at 5:20 PM, Josh Lo wrote: > Hi, > > I am trying to compile my main source code with module. > I need PETSc in both main code and module, so I include the header file in > both. > it gives me some error when compiling. > The error is: > > src/main.F:2.9: > > > use fmodule > > 2 > > Error: Symbol 'mpi_source' at (1) conflicts with symbol from module > 'fmodule', use-associated at (2) > > > /Users/XXXXXXX/Desktop/project/petsc/petsc_3.6.3_install/include/mpif.h:10:30: > > > PARAMETER (MPI_SOURCE=3,MPI_TAG=4,MPI_ERROR=5) > > 1 > Error: Cannot change attributes of USE-associated symbol mpi_source at (1) > > > > > it goes on and on, most relates to MPI. > > my main source code is like: > > program main > USE fmodule > implicit none > #include "petsc/finclude/petsc.h" > > ....... > > > end program main > > the module: > > module fmodule > #include "petsc/finclude/petsc.h" > > ...... > > end module fmodule > > if the I don't include the header file in module, then it is no error, > but I do need petsc in my module. > > How can I solve this problem? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html Change the include. Thanks Matt > Thanks, > Lo > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Apr 2 17:27:46 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Apr 2016 17:27:46 -0500 Subject: [petsc-users] compile fortran module with PETSc In-Reply-To: References: Message-ID: <07CA18D6-F7BF-4578-8D1A-9BA6F3273B69@mcs.anl.gov> If you include any files such as petscmat.h in the module then you do not need to include them again in the function, but files that end in def.h such as petscmatdef.h may need to be included in both the module and the function. In other words if you include petsc.h in your module then just include in your function files of the form petsc*def.h Barry > On Apr 2, 2016, at 5:20 PM, Josh Lo wrote: > > Hi, > > I am trying to compile my main source code with module. > I need PETSc in both main code and module, so I include the header file in both. > it gives me some error when compiling. > The error is: > > src/main.F:2.9: > > > > use fmodule > > 2 > > Error: Symbol 'mpi_source' at (1) conflicts with symbol from module 'fmodule', use-associated at (2) > > /Users/XXXXXXX/Desktop/project/petsc/petsc_3.6.3_install/include/mpif.h:10:30: > > > > PARAMETER (MPI_SOURCE=3,MPI_TAG=4,MPI_ERROR=5) > > 1 > > Error: Cannot change attributes of USE-associated symbol mpi_source at (1) > > > > > it goes on and on, most relates to MPI. > > my main source code is like: > > program main > USE fmodule > implicit none > #include "petsc/finclude/petsc.h" > > ....... > > > end program main > > the module: > > module fmodule > #include "petsc/finclude/petsc.h" > > ...... > > end module fmodule > > if the I don't include the header file in module, then it is no error, > but I do need petsc in my module. > > How can I solve this problem? > > > Thanks, > Lo > From Sander.Arens at UGent.be Sun Apr 3 06:55:09 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Sun, 3 Apr 2016 13:55:09 +0200 Subject: [petsc-users] PetscDSSetJacobianPreconditioner causing DIVERGED_LINE_SEARCH for multi-field problem In-Reply-To: References: Message-ID: Any thoughts on this, Matt? Thanks, Sander On 21 March 2016 at 09:49, Sander Arens wrote: > Matt, > > I think the problem is in DMCreateSubDM_Section_Private. The block size > can be counted there (very similar as in DMCreateMatrix_Plex) and then > passed down to the IS. I can create a pull request for this if you'd like > (or should I just add this to my other pull request for the Neumann bc's)? > > On 3 March 2016 at 18:02, Sander Arens wrote: > >> Ok, thanks for clearing that up! >> >> On 3 March 2016 at 18:00, Matthew Knepley wrote: >> >>> On Thu, Mar 3, 2016 at 10:48 AM, Sander Arens >>> wrote: >>> >>>> So is the block size here refering to the dimension of the >>>> near-nullspace or has it something to do with the coarsening? >>>> >>>> I would have thought to also see the block size in this part: >>>> >>> >>> Dang, that is a problem. It should have the correct block size when FS >>> pulls it out of the overall matrix. This should be >>> passed down using the block size of the IS. Something has been broken >>> here. I will put it on my list of things to look at. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Mat Object: 1 MPI processes >>>> type: seqaij >>>> rows=1701, cols=1701 >>>> total: nonzeros=109359, allocated nonzeros=109359 >>>> total number of mallocs used during MatSetValues >>>> calls =0 >>>> has attached near null space >>>> using I-node routines: found 567 nodes, limit used >>>> is 5 >>>> >>>> Thanks, >>>> Sander >>>> >>>> >>>> On 3 March 2016 at 17:29, Matthew Knepley wrote: >>>> >>>>> You can see the block size in the output >>>>> >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=12, cols=12, bs=6 >>>>> total: nonzeros=144, allocated nonzeros=144 >>>>> total number of mallocs used during MatSetValues >>>>> calls =0 >>>>> using I-node routines: found 3 nodes, limit used >>>>> is 5 >>>>> linear system matrix = precond matrix: >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=12, cols=12, bs=6 >>>>> total: nonzeros=144, allocated nonzeros=144 >>>>> total number of mallocs used during MatSetValues calls >>>>> =0 >>>>> using I-node routines: found 3 nodes, limit used is 5 >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> On Thu, Mar 3, 2016 at 10:28 AM, Sander Arens >>>>> wrote: >>>>> >>>>>> Sure, here it is. >>>>>> >>>>>> Thanks, >>>>>> Sander >>>>>> >>>>>> On 3 March 2016 at 15:33, Matthew Knepley wrote: >>>>>> >>>>>>> On Thu, Mar 3, 2016 at 8:05 AM, Sander Arens >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, the matrix is created by one of the assembly functions in >>>>>>>> Plex, so that answers my question. I was confused by this because I >>>>>>>> couldn't see this information when using -snes_view. >>>>>>>> >>>>>>> >>>>>>> Hmm, then something is wrong because the block size should be >>>>>>> printed for the matrix at the end of the solver in -snes_view, Can you >>>>>>> send that to me? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks for the helpful replies, >>>>>>>> Sander >>>>>>>> >>>>>>>> On 3 March 2016 at 14:52, Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Mar 3, 2016 at 7:49 AM, Sander Arens < >>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>> >>>>>>>>>> And how can I do this? Because when I look at all the options >>>>>>>>>> with -help I can strangely enough only find >>>>>>>>>> -fieldsplit_pressure_mat_block_size and not >>>>>>>>>> -fieldsplit_displacement_mat_block_size. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Maybe I am missing something here. The matrix from which you >>>>>>>>> calculate the preconditioner using GAMG must be created somewhere. Why >>>>>>>>> is the block size not specified at creation time? If the matrix is >>>>>>>>> created by one of the assembly functions in Plex, and the PetscSection has >>>>>>>>> a number of field components, then the block size will already be >>>>>>>>> set. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sander >>>>>>>>>> >>>>>>>>>> On 3 March 2016 at 14:21, Matthew Knepley >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 3, 2016 at 6:20 AM, Sander Arens < >>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok, I forgot to call SNESSetJacobian(snes, J, P, NULL, NULL) >>>>>>>>>>>> with J != P, which caused to write the mass matrix into the (otherwise >>>>>>>>>>>> zero) (1,1) block of the Jacobian and which was the reason for the >>>>>>>>>>>> linesearch to fail. >>>>>>>>>>>> However, after fixing that and trying to solve it with >>>>>>>>>>>> FieldSplit with LU factorization for the (0,0) block it failed because >>>>>>>>>>>> there were zero pivots for all rows. >>>>>>>>>>>> >>>>>>>>>>>> Anyway, I found out that attaching the mass matrix to the >>>>>>>>>>>> Lagrange multiplier field also worked. >>>>>>>>>>>> >>>>>>>>>>>> Another related question for my elasticity problem: after >>>>>>>>>>>> creating the rigid body modes with DMPlexCreateRigidBody and attaching it >>>>>>>>>>>> to the displacement field, does the matrix block size of the (0,0) block >>>>>>>>>>>> still have to be set for good performance with gamg? If so, how can I do >>>>>>>>>>>> this? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes, it should be enough to set the block size of the >>>>>>>>>>> preconditioner matrix. >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Sander >>>>>>>>>>>> >>>>>>>>>>>> On 2 March 2016 at 12:25, Matthew Knepley >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 2, 2016 at 5:13 AM, Sander Arens < >>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm trying to set a mass matrix preconditioner for the Schur >>>>>>>>>>>>>> complement of an incompressible finite elasticity problem. I tried using >>>>>>>>>>>>>> the command PetscDSSetJacobianPreconditioner(prob, 1, 1, g0_pre_mass_pp, >>>>>>>>>>>>>> NULL, NULL, NULL) (field 1 is the Lagrange multiplier field). >>>>>>>>>>>>>> However, this causes a DIVERGED_LINE_SEARCH due to to Nan or >>>>>>>>>>>>>> Inf in the function evaluation after Newton iteration 1. (Btw, I'm using >>>>>>>>>>>>>> the next branch). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is this because I didn't use PetscDSSetJacobianPreconditioner >>>>>>>>>>>>>> for the other blocks (which uses the Jacobian itself for preconditioning)? >>>>>>>>>>>>>> If so, how can I tell Petsc to use the Jacobian for those blocks? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 1) I put that code in very recently, and do not even have >>>>>>>>>>>>> sufficient test, so it may be buggy >>>>>>>>>>>>> >>>>>>>>>>>>> 2) If you are using FieldSplit, you can control which blocks >>>>>>>>>>>>> come from A and which come from the preconditioner P >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetDiagUseAmat.html#PCFieldSplitSetDiagUseAmat >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetOffDiagUseAmat.html#PCFieldSplitSetOffDiagUseAmat >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> I guess when using PetscDSSetJacobianPreconditioner the >>>>>>>>>>>>>> preconditioner is recomputed at every Newton step, so for a constant mass >>>>>>>>>>>>>> matrix this might not be ideal. How can I avoid recomputing this at every >>>>>>>>>>>>>> Newton iteration? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Maybe we need another flag like >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html >>>>>>>>>>>>> >>>>>>>>>>>>> or we need to expand >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagJacobian.html >>>>>>>>>>>>> >>>>>>>>>>>>> to separately cover the preconditioner matrix. However, both >>>>>>>>>>>>> matrices are computed by one call so this would >>>>>>>>>>>>> involve interface changes to user code, which we do not like >>>>>>>>>>>>> to do. Right now it seems like a small optimization. >>>>>>>>>>>>> I would want to wait and see whether it would really be >>>>>>>>>>>>> maningful. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Sander >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 3 08:17:37 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 3 Apr 2016 08:17:37 -0500 Subject: [petsc-users] PetscDSSetJacobianPreconditioner causing DIVERGED_LINE_SEARCH for multi-field problem In-Reply-To: References: Message-ID: On Sun, Apr 3, 2016 at 6:55 AM, Sander Arens wrote: > Any thoughts on this, Matt? > Shoot, this fell out of my working set, and I am behind on incorporating your pull request. I will look at both this week. Thanks, Matt > Thanks, > Sander > > On 21 March 2016 at 09:49, Sander Arens wrote: > >> Matt, >> >> I think the problem is in DMCreateSubDM_Section_Private. The block size >> can be counted there (very similar as in DMCreateMatrix_Plex) and then >> passed down to the IS. I can create a pull request for this if you'd like >> (or should I just add this to my other pull request for the Neumann bc's)? >> >> On 3 March 2016 at 18:02, Sander Arens wrote: >> >>> Ok, thanks for clearing that up! >>> >>> On 3 March 2016 at 18:00, Matthew Knepley wrote: >>> >>>> On Thu, Mar 3, 2016 at 10:48 AM, Sander Arens >>>> wrote: >>>> >>>>> So is the block size here refering to the dimension of the >>>>> near-nullspace or has it something to do with the coarsening? >>>>> >>>>> I would have thought to also see the block size in this part: >>>>> >>>> >>>> Dang, that is a problem. It should have the correct block size when FS >>>> pulls it out of the overall matrix. This should be >>>> passed down using the block size of the IS. Something has been broken >>>> here. I will put it on my list of things to look at. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=1701, cols=1701 >>>>> total: nonzeros=109359, allocated nonzeros=109359 >>>>> total number of mallocs used during MatSetValues >>>>> calls =0 >>>>> has attached near null space >>>>> using I-node routines: found 567 nodes, limit >>>>> used is 5 >>>>> >>>>> Thanks, >>>>> Sander >>>>> >>>>> >>>>> On 3 March 2016 at 17:29, Matthew Knepley wrote: >>>>> >>>>>> You can see the block size in the output >>>>>> >>>>>> Mat Object: 1 MPI processes >>>>>> type: seqaij >>>>>> rows=12, cols=12, bs=6 >>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>> total number of mallocs used during MatSetValues >>>>>> calls =0 >>>>>> using I-node routines: found 3 nodes, limit >>>>>> used is 5 >>>>>> linear system matrix = precond matrix: >>>>>> Mat Object: 1 MPI processes >>>>>> type: seqaij >>>>>> rows=12, cols=12, bs=6 >>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>> total number of mallocs used during MatSetValues >>>>>> calls =0 >>>>>> using I-node routines: found 3 nodes, limit used is >>>>>> 5 >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> On Thu, Mar 3, 2016 at 10:28 AM, Sander Arens >>>>>> wrote: >>>>>> >>>>>>> Sure, here it is. >>>>>>> >>>>>>> Thanks, >>>>>>> Sander >>>>>>> >>>>>>> On 3 March 2016 at 15:33, Matthew Knepley wrote: >>>>>>> >>>>>>>> On Thu, Mar 3, 2016 at 8:05 AM, Sander Arens >>>>>>> > wrote: >>>>>>>> >>>>>>>>> Yes, the matrix is created by one of the assembly functions in >>>>>>>>> Plex, so that answers my question. I was confused by this because I >>>>>>>>> couldn't see this information when using -snes_view. >>>>>>>>> >>>>>>>> >>>>>>>> Hmm, then something is wrong because the block size should be >>>>>>>> printed for the matrix at the end of the solver in -snes_view, Can you >>>>>>>> send that to me? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks for the helpful replies, >>>>>>>>> Sander >>>>>>>>> >>>>>>>>> On 3 March 2016 at 14:52, Matthew Knepley >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 3, 2016 at 7:49 AM, Sander Arens < >>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>> >>>>>>>>>>> And how can I do this? Because when I look at all the options >>>>>>>>>>> with -help I can strangely enough only find >>>>>>>>>>> -fieldsplit_pressure_mat_block_size and not >>>>>>>>>>> -fieldsplit_displacement_mat_block_size. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Maybe I am missing something here. The matrix from which you >>>>>>>>>> calculate the preconditioner using GAMG must be created somewhere. Why >>>>>>>>>> is the block size not specified at creation time? If the matrix >>>>>>>>>> is created by one of the assembly functions in Plex, and the PetscSection >>>>>>>>>> has >>>>>>>>>> a number of field components, then the block size will already be >>>>>>>>>> set. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Sander >>>>>>>>>>> >>>>>>>>>>> On 3 March 2016 at 14:21, Matthew Knepley >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 3, 2016 at 6:20 AM, Sander Arens < >>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, I forgot to call SNESSetJacobian(snes, J, P, NULL, NULL) >>>>>>>>>>>>> with J != P, which caused to write the mass matrix into the (otherwise >>>>>>>>>>>>> zero) (1,1) block of the Jacobian and which was the reason for the >>>>>>>>>>>>> linesearch to fail. >>>>>>>>>>>>> However, after fixing that and trying to solve it with >>>>>>>>>>>>> FieldSplit with LU factorization for the (0,0) block it failed because >>>>>>>>>>>>> there were zero pivots for all rows. >>>>>>>>>>>>> >>>>>>>>>>>>> Anyway, I found out that attaching the mass matrix to the >>>>>>>>>>>>> Lagrange multiplier field also worked. >>>>>>>>>>>>> >>>>>>>>>>>>> Another related question for my elasticity problem: after >>>>>>>>>>>>> creating the rigid body modes with DMPlexCreateRigidBody and attaching it >>>>>>>>>>>>> to the displacement field, does the matrix block size of the (0,0) block >>>>>>>>>>>>> still have to be set for good performance with gamg? If so, how can I do >>>>>>>>>>>>> this? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Yes, it should be enough to set the block size of the >>>>>>>>>>>> preconditioner matrix. >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Sander >>>>>>>>>>>>> >>>>>>>>>>>>> On 2 March 2016 at 12:25, Matthew Knepley >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Mar 2, 2016 at 5:13 AM, Sander Arens < >>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm trying to set a mass matrix preconditioner for the Schur >>>>>>>>>>>>>>> complement of an incompressible finite elasticity problem. I tried using >>>>>>>>>>>>>>> the command PetscDSSetJacobianPreconditioner(prob, 1, 1, g0_pre_mass_pp, >>>>>>>>>>>>>>> NULL, NULL, NULL) (field 1 is the Lagrange multiplier field). >>>>>>>>>>>>>>> However, this causes a DIVERGED_LINE_SEARCH due to to Nan or >>>>>>>>>>>>>>> Inf in the function evaluation after Newton iteration 1. (Btw, I'm using >>>>>>>>>>>>>>> the next branch). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is this because I didn't use >>>>>>>>>>>>>>> PetscDSSetJacobianPreconditioner for the other blocks (which uses the >>>>>>>>>>>>>>> Jacobian itself for preconditioning)? If so, how can I tell Petsc to use >>>>>>>>>>>>>>> the Jacobian for those blocks? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) I put that code in very recently, and do not even have >>>>>>>>>>>>>> sufficient test, so it may be buggy >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) If you are using FieldSplit, you can control which blocks >>>>>>>>>>>>>> come from A and which come from the preconditioner P >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetDiagUseAmat.html#PCFieldSplitSetDiagUseAmat >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetOffDiagUseAmat.html#PCFieldSplitSetOffDiagUseAmat >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I guess when using PetscDSSetJacobianPreconditioner the >>>>>>>>>>>>>>> preconditioner is recomputed at every Newton step, so for a constant mass >>>>>>>>>>>>>>> matrix this might not be ideal. How can I avoid recomputing this at every >>>>>>>>>>>>>>> Newton iteration? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maybe we need another flag like >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> or we need to expand >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagJacobian.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> to separately cover the preconditioner matrix. However, both >>>>>>>>>>>>>> matrices are computed by one call so this would >>>>>>>>>>>>>> involve interface changes to user code, which we do not like >>>>>>>>>>>>>> to do. Right now it seems like a small optimization. >>>>>>>>>>>>>> I would want to wait and see whether it would really be >>>>>>>>>>>>>> maningful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>> their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sander.Arens at UGent.be Sun Apr 3 09:26:40 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Sun, 3 Apr 2016 16:26:40 +0200 Subject: [petsc-users] PetscDSSetJacobianPreconditioner causing DIVERGED_LINE_SEARCH for multi-field problem In-Reply-To: References: Message-ID: No problem. I think I solved the problem with the block size, so I wanted to know if I should also add this to my pull request or if I should make a separate one? Thanks, Sander On 3 April 2016 at 15:17, Matthew Knepley wrote: > On Sun, Apr 3, 2016 at 6:55 AM, Sander Arens > wrote: > >> Any thoughts on this, Matt? >> > > Shoot, this fell out of my working set, and I am behind on incorporating > your pull request. I will > look at both this week. > > Thanks, > > Matt > > >> Thanks, >> Sander >> >> On 21 March 2016 at 09:49, Sander Arens wrote: >> >>> Matt, >>> >>> I think the problem is in DMCreateSubDM_Section_Private. The block size >>> can be counted there (very similar as in DMCreateMatrix_Plex) and then >>> passed down to the IS. I can create a pull request for this if you'd like >>> (or should I just add this to my other pull request for the Neumann bc's)? >>> >>> On 3 March 2016 at 18:02, Sander Arens wrote: >>> >>>> Ok, thanks for clearing that up! >>>> >>>> On 3 March 2016 at 18:00, Matthew Knepley wrote: >>>> >>>>> On Thu, Mar 3, 2016 at 10:48 AM, Sander Arens >>>>> wrote: >>>>> >>>>>> So is the block size here refering to the dimension of the >>>>>> near-nullspace or has it something to do with the coarsening? >>>>>> >>>>>> I would have thought to also see the block size in this part: >>>>>> >>>>> >>>>> Dang, that is a problem. It should have the correct block size when FS >>>>> pulls it out of the overall matrix. This should be >>>>> passed down using the block size of the IS. Something has been broken >>>>> here. I will put it on my list of things to look at. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Mat Object: 1 MPI >>>>>> processes >>>>>> type: seqaij >>>>>> rows=1701, cols=1701 >>>>>> total: nonzeros=109359, allocated nonzeros=109359 >>>>>> total number of mallocs used during MatSetValues >>>>>> calls =0 >>>>>> has attached near null space >>>>>> using I-node routines: found 567 nodes, limit >>>>>> used is 5 >>>>>> >>>>>> Thanks, >>>>>> Sander >>>>>> >>>>>> >>>>>> On 3 March 2016 at 17:29, Matthew Knepley wrote: >>>>>> >>>>>>> You can see the block size in the output >>>>>>> >>>>>>> Mat Object: 1 MPI processes >>>>>>> type: seqaij >>>>>>> rows=12, cols=12, bs=6 >>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>> total number of mallocs used during MatSetValues >>>>>>> calls =0 >>>>>>> using I-node routines: found 3 nodes, limit >>>>>>> used is 5 >>>>>>> linear system matrix = precond matrix: >>>>>>> Mat Object: 1 MPI processes >>>>>>> type: seqaij >>>>>>> rows=12, cols=12, bs=6 >>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>> total number of mallocs used during MatSetValues >>>>>>> calls =0 >>>>>>> using I-node routines: found 3 nodes, limit used >>>>>>> is 5 >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> On Thu, Mar 3, 2016 at 10:28 AM, Sander Arens >>>>>> > wrote: >>>>>>> >>>>>>>> Sure, here it is. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sander >>>>>>>> >>>>>>>> On 3 March 2016 at 15:33, Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Mar 3, 2016 at 8:05 AM, Sander Arens < >>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>> >>>>>>>>>> Yes, the matrix is created by one of the assembly functions in >>>>>>>>>> Plex, so that answers my question. I was confused by this because I >>>>>>>>>> couldn't see this information when using -snes_view. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hmm, then something is wrong because the block size should be >>>>>>>>> printed for the matrix at the end of the solver in -snes_view, Can you >>>>>>>>> send that to me? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks for the helpful replies, >>>>>>>>>> Sander >>>>>>>>>> >>>>>>>>>> On 3 March 2016 at 14:52, Matthew Knepley >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 3, 2016 at 7:49 AM, Sander Arens < >>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> And how can I do this? Because when I look at all the options >>>>>>>>>>>> with -help I can strangely enough only find >>>>>>>>>>>> -fieldsplit_pressure_mat_block_size and not >>>>>>>>>>>> -fieldsplit_displacement_mat_block_size. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Maybe I am missing something here. The matrix from which you >>>>>>>>>>> calculate the preconditioner using GAMG must be created somewhere. Why >>>>>>>>>>> is the block size not specified at creation time? If the matrix >>>>>>>>>>> is created by one of the assembly functions in Plex, and the PetscSection >>>>>>>>>>> has >>>>>>>>>>> a number of field components, then the block size will already >>>>>>>>>>> be set. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Sander >>>>>>>>>>>> >>>>>>>>>>>> On 3 March 2016 at 14:21, Matthew Knepley >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 3, 2016 at 6:20 AM, Sander Arens < >>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, I forgot to call SNESSetJacobian(snes, J, P, NULL, NULL) >>>>>>>>>>>>>> with J != P, which caused to write the mass matrix into the (otherwise >>>>>>>>>>>>>> zero) (1,1) block of the Jacobian and which was the reason for the >>>>>>>>>>>>>> linesearch to fail. >>>>>>>>>>>>>> However, after fixing that and trying to solve it with >>>>>>>>>>>>>> FieldSplit with LU factorization for the (0,0) block it failed because >>>>>>>>>>>>>> there were zero pivots for all rows. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyway, I found out that attaching the mass matrix to the >>>>>>>>>>>>>> Lagrange multiplier field also worked. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Another related question for my elasticity problem: after >>>>>>>>>>>>>> creating the rigid body modes with DMPlexCreateRigidBody and attaching it >>>>>>>>>>>>>> to the displacement field, does the matrix block size of the (0,0) block >>>>>>>>>>>>>> still have to be set for good performance with gamg? If so, how can I do >>>>>>>>>>>>>> this? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, it should be enough to set the block size of the >>>>>>>>>>>>> preconditioner matrix. >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Sander >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2 March 2016 at 12:25, Matthew Knepley >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Mar 2, 2016 at 5:13 AM, Sander Arens < >>>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm trying to set a mass matrix preconditioner for the >>>>>>>>>>>>>>>> Schur complement of an incompressible finite elasticity problem. I tried >>>>>>>>>>>>>>>> using the command PetscDSSetJacobianPreconditioner(prob, 1, 1, >>>>>>>>>>>>>>>> g0_pre_mass_pp, NULL, NULL, NULL) (field 1 is the Lagrange multiplier >>>>>>>>>>>>>>>> field). >>>>>>>>>>>>>>>> However, this causes a DIVERGED_LINE_SEARCH due to to Nan >>>>>>>>>>>>>>>> or Inf in the function evaluation after Newton iteration 1. (Btw, I'm using >>>>>>>>>>>>>>>> the next branch). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is this because I didn't use >>>>>>>>>>>>>>>> PetscDSSetJacobianPreconditioner for the other blocks (which uses the >>>>>>>>>>>>>>>> Jacobian itself for preconditioning)? If so, how can I tell Petsc to use >>>>>>>>>>>>>>>> the Jacobian for those blocks? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) I put that code in very recently, and do not even have >>>>>>>>>>>>>>> sufficient test, so it may be buggy >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) If you are using FieldSplit, you can control which blocks >>>>>>>>>>>>>>> come from A and which come from the preconditioner P >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetDiagUseAmat.html#PCFieldSplitSetDiagUseAmat >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetOffDiagUseAmat.html#PCFieldSplitSetOffDiagUseAmat >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I guess when using PetscDSSetJacobianPreconditioner the >>>>>>>>>>>>>>>> preconditioner is recomputed at every Newton step, so for a constant mass >>>>>>>>>>>>>>>> matrix this might not be ideal. How can I avoid recomputing this at every >>>>>>>>>>>>>>>> Newton iteration? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Maybe we need another flag like >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> or we need to expand >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagJacobian.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> to separately cover the preconditioner matrix. However, both >>>>>>>>>>>>>>> matrices are computed by one call so this would >>>>>>>>>>>>>>> involve interface changes to user code, which we do not like >>>>>>>>>>>>>>> to do. Right now it seems like a small optimization. >>>>>>>>>>>>>>> I would want to wait and see whether it would really be >>>>>>>>>>>>>>> maningful. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 3 09:53:56 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 3 Apr 2016 09:53:56 -0500 Subject: [petsc-users] PetscDSSetJacobianPreconditioner causing DIVERGED_LINE_SEARCH for multi-field problem In-Reply-To: References: Message-ID: Go ahead and add it. I will review everything. Thanks, Matt On Sun, Apr 3, 2016 at 9:26 AM, Sander Arens wrote: > No problem. I think I solved the problem with the block size, so I wanted > to know if I should also add this to my pull request or if I should make a > separate one? > > Thanks, > Sander > > On 3 April 2016 at 15:17, Matthew Knepley wrote: > >> On Sun, Apr 3, 2016 at 6:55 AM, Sander Arens >> wrote: >> >>> Any thoughts on this, Matt? >>> >> >> Shoot, this fell out of my working set, and I am behind on incorporating >> your pull request. I will >> look at both this week. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Sander >>> >>> On 21 March 2016 at 09:49, Sander Arens wrote: >>> >>>> Matt, >>>> >>>> I think the problem is in DMCreateSubDM_Section_Private. The block size >>>> can be counted there (very similar as in DMCreateMatrix_Plex) and then >>>> passed down to the IS. I can create a pull request for this if you'd like >>>> (or should I just add this to my other pull request for the Neumann bc's)? >>>> >>>> On 3 March 2016 at 18:02, Sander Arens wrote: >>>> >>>>> Ok, thanks for clearing that up! >>>>> >>>>> On 3 March 2016 at 18:00, Matthew Knepley wrote: >>>>> >>>>>> On Thu, Mar 3, 2016 at 10:48 AM, Sander Arens >>>>>> wrote: >>>>>> >>>>>>> So is the block size here refering to the dimension of the >>>>>>> near-nullspace or has it something to do with the coarsening? >>>>>>> >>>>>>> I would have thought to also see the block size in this part: >>>>>>> >>>>>> >>>>>> Dang, that is a problem. It should have the correct block size when >>>>>> FS pulls it out of the overall matrix. This should be >>>>>> passed down using the block size of the IS. Something has been broken >>>>>> here. I will put it on my list of things to look at. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Mat Object: 1 MPI >>>>>>> processes >>>>>>> type: seqaij >>>>>>> rows=1701, cols=1701 >>>>>>> total: nonzeros=109359, allocated nonzeros=109359 >>>>>>> total number of mallocs used during MatSetValues >>>>>>> calls =0 >>>>>>> has attached near null space >>>>>>> using I-node routines: found 567 nodes, limit >>>>>>> used is 5 >>>>>>> >>>>>>> Thanks, >>>>>>> Sander >>>>>>> >>>>>>> >>>>>>> On 3 March 2016 at 17:29, Matthew Knepley wrote: >>>>>>> >>>>>>>> You can see the block size in the output >>>>>>>> >>>>>>>> Mat Object: 1 MPI processes >>>>>>>> type: seqaij >>>>>>>> rows=12, cols=12, bs=6 >>>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>>> total number of mallocs used during >>>>>>>> MatSetValues calls =0 >>>>>>>> using I-node routines: found 3 nodes, limit >>>>>>>> used is 5 >>>>>>>> linear system matrix = precond matrix: >>>>>>>> Mat Object: 1 MPI processes >>>>>>>> type: seqaij >>>>>>>> rows=12, cols=12, bs=6 >>>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>>> total number of mallocs used during MatSetValues >>>>>>>> calls =0 >>>>>>>> using I-node routines: found 3 nodes, limit used >>>>>>>> is 5 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> On Thu, Mar 3, 2016 at 10:28 AM, Sander Arens < >>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>> >>>>>>>>> Sure, here it is. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Sander >>>>>>>>> >>>>>>>>> On 3 March 2016 at 15:33, Matthew Knepley >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 3, 2016 at 8:05 AM, Sander Arens < >>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>> >>>>>>>>>>> Yes, the matrix is created by one of the assembly functions in >>>>>>>>>>> Plex, so that answers my question. I was confused by this because I >>>>>>>>>>> couldn't see this information when using -snes_view. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm, then something is wrong because the block size should be >>>>>>>>>> printed for the matrix at the end of the solver in -snes_view, Can you >>>>>>>>>> send that to me? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks for the helpful replies, >>>>>>>>>>> Sander >>>>>>>>>>> >>>>>>>>>>> On 3 March 2016 at 14:52, Matthew Knepley >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 3, 2016 at 7:49 AM, Sander Arens < >>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> And how can I do this? Because when I look at all the options >>>>>>>>>>>>> with -help I can strangely enough only find >>>>>>>>>>>>> -fieldsplit_pressure_mat_block_size and not >>>>>>>>>>>>> -fieldsplit_displacement_mat_block_size. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Maybe I am missing something here. The matrix from which you >>>>>>>>>>>> calculate the preconditioner using GAMG must be created somewhere. Why >>>>>>>>>>>> is the block size not specified at creation time? If the matrix >>>>>>>>>>>> is created by one of the assembly functions in Plex, and the PetscSection >>>>>>>>>>>> has >>>>>>>>>>>> a number of field components, then the block size will already >>>>>>>>>>>> be set. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Sander >>>>>>>>>>>>> >>>>>>>>>>>>> On 3 March 2016 at 14:21, Matthew Knepley >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 3, 2016 at 6:20 AM, Sander Arens < >>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, I forgot to call SNESSetJacobian(snes, J, P, NULL, NULL) >>>>>>>>>>>>>>> with J != P, which caused to write the mass matrix into the (otherwise >>>>>>>>>>>>>>> zero) (1,1) block of the Jacobian and which was the reason for the >>>>>>>>>>>>>>> linesearch to fail. >>>>>>>>>>>>>>> However, after fixing that and trying to solve it with >>>>>>>>>>>>>>> FieldSplit with LU factorization for the (0,0) block it failed because >>>>>>>>>>>>>>> there were zero pivots for all rows. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyway, I found out that attaching the mass matrix to the >>>>>>>>>>>>>>> Lagrange multiplier field also worked. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another related question for my elasticity problem: after >>>>>>>>>>>>>>> creating the rigid body modes with DMPlexCreateRigidBody and attaching it >>>>>>>>>>>>>>> to the displacement field, does the matrix block size of the (0,0) block >>>>>>>>>>>>>>> still have to be set for good performance with gamg? If so, how can I do >>>>>>>>>>>>>>> this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, it should be enough to set the block size of the >>>>>>>>>>>>>> preconditioner matrix. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2 March 2016 at 12:25, Matthew Knepley >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Mar 2, 2016 at 5:13 AM, Sander Arens < >>>>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm trying to set a mass matrix preconditioner for the >>>>>>>>>>>>>>>>> Schur complement of an incompressible finite elasticity problem. I tried >>>>>>>>>>>>>>>>> using the command PetscDSSetJacobianPreconditioner(prob, 1, 1, >>>>>>>>>>>>>>>>> g0_pre_mass_pp, NULL, NULL, NULL) (field 1 is the Lagrange multiplier >>>>>>>>>>>>>>>>> field). >>>>>>>>>>>>>>>>> However, this causes a DIVERGED_LINE_SEARCH due to to Nan >>>>>>>>>>>>>>>>> or Inf in the function evaluation after Newton iteration 1. (Btw, I'm using >>>>>>>>>>>>>>>>> the next branch). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is this because I didn't use >>>>>>>>>>>>>>>>> PetscDSSetJacobianPreconditioner for the other blocks (which uses the >>>>>>>>>>>>>>>>> Jacobian itself for preconditioning)? If so, how can I tell Petsc to use >>>>>>>>>>>>>>>>> the Jacobian for those blocks? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) I put that code in very recently, and do not even have >>>>>>>>>>>>>>>> sufficient test, so it may be buggy >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) If you are using FieldSplit, you can control which >>>>>>>>>>>>>>>> blocks come from A and which come from the preconditioner P >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetDiagUseAmat.html#PCFieldSplitSetDiagUseAmat >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetOffDiagUseAmat.html#PCFieldSplitSetOffDiagUseAmat >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I guess when using PetscDSSetJacobianPreconditioner the >>>>>>>>>>>>>>>>> preconditioner is recomputed at every Newton step, so for a constant mass >>>>>>>>>>>>>>>>> matrix this might not be ideal. How can I avoid recomputing this at every >>>>>>>>>>>>>>>>> Newton iteration? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Maybe we need another flag like >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> or we need to expand >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagJacobian.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> to separately cover the preconditioner matrix. However, >>>>>>>>>>>>>>>> both matrices are computed by one call so this would >>>>>>>>>>>>>>>> involve interface changes to user code, which we do not >>>>>>>>>>>>>>>> like to do. Right now it seems like a small optimization. >>>>>>>>>>>>>>>> I would want to wait and see whether it would really be >>>>>>>>>>>>>>>> maningful. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>> their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sander.Arens at UGent.be Sun Apr 3 10:25:36 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Sun, 3 Apr 2016 17:25:36 +0200 Subject: [petsc-users] PetscDSSetJacobianPreconditioner causing DIVERGED_LINE_SEARCH for multi-field problem In-Reply-To: References: Message-ID: Great! It's added. Thanks, Sander On 3 April 2016 at 16:53, Matthew Knepley wrote: > Go ahead and add it. I will review everything. > > Thanks, > > Matt > > On Sun, Apr 3, 2016 at 9:26 AM, Sander Arens > wrote: > >> No problem. I think I solved the problem with the block size, so I wanted >> to know if I should also add this to my pull request or if I should make a >> separate one? >> >> Thanks, >> Sander >> >> On 3 April 2016 at 15:17, Matthew Knepley wrote: >> >>> On Sun, Apr 3, 2016 at 6:55 AM, Sander Arens >>> wrote: >>> >>>> Any thoughts on this, Matt? >>>> >>> >>> Shoot, this fell out of my working set, and I am behind on incorporating >>> your pull request. I will >>> look at both this week. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Sander >>>> >>>> On 21 March 2016 at 09:49, Sander Arens wrote: >>>> >>>>> Matt, >>>>> >>>>> I think the problem is in DMCreateSubDM_Section_Private. The block >>>>> size can be counted there (very similar as in DMCreateMatrix_Plex) and then >>>>> passed down to the IS. I can create a pull request for this if you'd like >>>>> (or should I just add this to my other pull request for the Neumann bc's)? >>>>> >>>>> On 3 March 2016 at 18:02, Sander Arens wrote: >>>>> >>>>>> Ok, thanks for clearing that up! >>>>>> >>>>>> On 3 March 2016 at 18:00, Matthew Knepley wrote: >>>>>> >>>>>>> On Thu, Mar 3, 2016 at 10:48 AM, Sander Arens >>>>>> > wrote: >>>>>>> >>>>>>>> So is the block size here refering to the dimension of the >>>>>>>> near-nullspace or has it something to do with the coarsening? >>>>>>>> >>>>>>>> I would have thought to also see the block size in this part: >>>>>>>> >>>>>>> >>>>>>> Dang, that is a problem. It should have the correct block size when >>>>>>> FS pulls it out of the overall matrix. This should be >>>>>>> passed down using the block size of the IS. Something has been >>>>>>> broken here. I will put it on my list of things to look at. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Mat Object: 1 MPI >>>>>>>> processes >>>>>>>> type: seqaij >>>>>>>> rows=1701, cols=1701 >>>>>>>> total: nonzeros=109359, allocated nonzeros=109359 >>>>>>>> total number of mallocs used during MatSetValues >>>>>>>> calls =0 >>>>>>>> has attached near null space >>>>>>>> using I-node routines: found 567 nodes, limit >>>>>>>> used is 5 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sander >>>>>>>> >>>>>>>> >>>>>>>> On 3 March 2016 at 17:29, Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> You can see the block size in the output >>>>>>>>> >>>>>>>>> Mat Object: 1 MPI processes >>>>>>>>> type: seqaij >>>>>>>>> rows=12, cols=12, bs=6 >>>>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>>>> total number of mallocs used during >>>>>>>>> MatSetValues calls =0 >>>>>>>>> using I-node routines: found 3 nodes, limit >>>>>>>>> used is 5 >>>>>>>>> linear system matrix = precond matrix: >>>>>>>>> Mat Object: 1 MPI processes >>>>>>>>> type: seqaij >>>>>>>>> rows=12, cols=12, bs=6 >>>>>>>>> total: nonzeros=144, allocated nonzeros=144 >>>>>>>>> total number of mallocs used during MatSetValues >>>>>>>>> calls =0 >>>>>>>>> using I-node routines: found 3 nodes, limit used >>>>>>>>> is 5 >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> On Thu, Mar 3, 2016 at 10:28 AM, Sander Arens < >>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>> >>>>>>>>>> Sure, here it is. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sander >>>>>>>>>> >>>>>>>>>> On 3 March 2016 at 15:33, Matthew Knepley >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 3, 2016 at 8:05 AM, Sander Arens < >>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yes, the matrix is created by one of the assembly functions in >>>>>>>>>>>> Plex, so that answers my question. I was confused by this because I >>>>>>>>>>>> couldn't see this information when using -snes_view. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hmm, then something is wrong because the block size should be >>>>>>>>>>> printed for the matrix at the end of the solver in -snes_view, Can you >>>>>>>>>>> send that to me? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks for the helpful replies, >>>>>>>>>>>> Sander >>>>>>>>>>>> >>>>>>>>>>>> On 3 March 2016 at 14:52, Matthew Knepley >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 3, 2016 at 7:49 AM, Sander Arens < >>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> And how can I do this? Because when I look at all the options >>>>>>>>>>>>>> with -help I can strangely enough only find >>>>>>>>>>>>>> -fieldsplit_pressure_mat_block_size and not >>>>>>>>>>>>>> -fieldsplit_displacement_mat_block_size. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Maybe I am missing something here. The matrix from which you >>>>>>>>>>>>> calculate the preconditioner using GAMG must be created somewhere. Why >>>>>>>>>>>>> is the block size not specified at creation time? If the >>>>>>>>>>>>> matrix is created by one of the assembly functions in Plex, and the >>>>>>>>>>>>> PetscSection has >>>>>>>>>>>>> a number of field components, then the block size will already >>>>>>>>>>>>> be set. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Sander >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 3 March 2016 at 14:21, Matthew Knepley >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Mar 3, 2016 at 6:20 AM, Sander Arens < >>>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok, I forgot to call SNESSetJacobian(snes, J, P, NULL, >>>>>>>>>>>>>>>> NULL) with J != P, which caused to write the mass matrix into the >>>>>>>>>>>>>>>> (otherwise zero) (1,1) block of the Jacobian and which was the reason for >>>>>>>>>>>>>>>> the linesearch to fail. >>>>>>>>>>>>>>>> However, after fixing that and trying to solve it with >>>>>>>>>>>>>>>> FieldSplit with LU factorization for the (0,0) block it failed because >>>>>>>>>>>>>>>> there were zero pivots for all rows. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyway, I found out that attaching the mass matrix to the >>>>>>>>>>>>>>>> Lagrange multiplier field also worked. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Another related question for my elasticity problem: after >>>>>>>>>>>>>>>> creating the rigid body modes with DMPlexCreateRigidBody and attaching it >>>>>>>>>>>>>>>> to the displacement field, does the matrix block size of the (0,0) block >>>>>>>>>>>>>>>> still have to be set for good performance with gamg? If so, how can I do >>>>>>>>>>>>>>>> this? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, it should be enough to set the block size of the >>>>>>>>>>>>>>> preconditioner matrix. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2 March 2016 at 12:25, Matthew Knepley < >>>>>>>>>>>>>>>> knepley at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Mar 2, 2016 at 5:13 AM, Sander Arens < >>>>>>>>>>>>>>>>> Sander.Arens at ugent.be> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm trying to set a mass matrix preconditioner for the >>>>>>>>>>>>>>>>>> Schur complement of an incompressible finite elasticity problem. I tried >>>>>>>>>>>>>>>>>> using the command PetscDSSetJacobianPreconditioner(prob, 1, 1, >>>>>>>>>>>>>>>>>> g0_pre_mass_pp, NULL, NULL, NULL) (field 1 is the Lagrange multiplier >>>>>>>>>>>>>>>>>> field). >>>>>>>>>>>>>>>>>> However, this causes a DIVERGED_LINE_SEARCH due to to Nan >>>>>>>>>>>>>>>>>> or Inf in the function evaluation after Newton iteration 1. (Btw, I'm using >>>>>>>>>>>>>>>>>> the next branch). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is this because I didn't use >>>>>>>>>>>>>>>>>> PetscDSSetJacobianPreconditioner for the other blocks (which uses the >>>>>>>>>>>>>>>>>> Jacobian itself for preconditioning)? If so, how can I tell Petsc to use >>>>>>>>>>>>>>>>>> the Jacobian for those blocks? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) I put that code in very recently, and do not even have >>>>>>>>>>>>>>>>> sufficient test, so it may be buggy >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2) If you are using FieldSplit, you can control which >>>>>>>>>>>>>>>>> blocks come from A and which come from the preconditioner P >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetDiagUseAmat.html#PCFieldSplitSetDiagUseAmat >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetOffDiagUseAmat.html#PCFieldSplitSetOffDiagUseAmat >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I guess when using PetscDSSetJacobianPreconditioner the >>>>>>>>>>>>>>>>>> preconditioner is recomputed at every Newton step, so for a constant mass >>>>>>>>>>>>>>>>>> matrix this might not be ideal. How can I avoid recomputing this at every >>>>>>>>>>>>>>>>>> Newton iteration? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Maybe we need another flag like >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> or we need to expand >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagJacobian.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> to separately cover the preconditioner matrix. However, >>>>>>>>>>>>>>>>> both matrices are computed by one call so this would >>>>>>>>>>>>>>>>> involve interface changes to user code, which we do not >>>>>>>>>>>>>>>>> like to do. Right now it seems like a small optimization. >>>>>>>>>>>>>>>>> I would want to wait and see whether it would really be >>>>>>>>>>>>>>>>> maningful. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Sander >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Sun Apr 3 13:47:16 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Sun, 3 Apr 2016 13:47:16 -0500 Subject: [petsc-users] eigensolution error with slepc Message-ID: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> Hi, I am using slepc to solve for the natural frequencies of a small-disturbance modal eigensolution of a plate with compressive stresses. M x = (1/omega^2) K(lambda) x lambda is the loading parameter. I first solve for the nonlinear static solution for a given load, which gives K (after linearization), which is used here. Note that I am solving for 1/omega^2, instead of omega^2, since the mass matrix may have zero entries related to some dofs, while the stiffness matrix of the constrained structure has no zero eigenvalues (except, perhaps in the nonlinear region at the bifurcation point). As lambda increases, the structure crosses into the post-buckled region, so a bifurcation point is crossed at some lambda. The fundamental natural frequency with this tangent stiffness matrix is expected to reduce to zero, and then increase after the bifurcation point. I am calling slepc without any command line options, except that I specify LU-decomposition as the linear solver. The challenge that I am facing is that I am able to get eigensolutions till the lowest natural frequency drops to 0, and then the solver quits with the following error message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in Lapack xHSEQR 100 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /Users/manav/Documents/codes/MAST/build/opt/example_driver on a arch-darwin-cxx-opt named Dhcp-90-243.HPC.MsState.Edu by manav Sun Apr 3 13:31:08 2016 [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/ local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes - -download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 -- download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 [0]PETSC ERROR: #1 DSSolve_NHEP() line 622 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/nhep/dsnhep.c [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c [0]PETSC ERROR: #3 EPSSolve_Arnoldi() line 124 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/arnoldi/arnoldi.c [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c Is this related to the loss of positive-definite nature of the stiffness matrix? Is there a way I can still get solution form slepc? I would greatly appreciate any advice. Regards, Manav -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Sun Apr 3 14:39:33 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sun, 3 Apr 2016 21:39:33 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> Message-ID: <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> > El 3 abr 2016, a las 20:47, Manav Bhatia escribi?: > > Hi, > > I am using slepc to solve for the natural frequencies of a small-disturbance modal eigensolution of a plate with compressive stresses. > > M x = (1/omega^2) K(lambda) x > > lambda is the loading parameter. I first solve for the nonlinear static solution for a given load, which gives K (after linearization), which is used here. > > Note that I am solving for 1/omega^2, instead of omega^2, since the mass matrix may have zero entries related to some dofs, while the stiffness matrix of the constrained structure has no zero eigenvalues (except, perhaps in the nonlinear region at the bifurcation point). > > As lambda increases, the structure crosses into the post-buckled region, so a bifurcation point is crossed at some lambda. The fundamental natural frequency with this tangent stiffness matrix is expected to reduce to zero, and then increase after the bifurcation point. > > I am calling slepc without any command line options, except that I specify LU-decomposition as the linear solver. > > The challenge that I am facing is that I am able to get eigensolutions till the lowest natural frequency drops to 0, and then the solver quits with the following error message: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in Lapack xHSEQR 100 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /Users/manav/Documents/codes/MAST/build/opt/example_driver on a arch-darwin-cxx-opt named Dhcp-90-243.HPC.MsState.Edu by manav Sun Apr 3 13:31:08 2016 > [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/ > local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes - > -download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 -- > download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 > [0]PETSC ERROR: #1 DSSolve_NHEP() line 622 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/nhep/dsnhep.c > [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c > [0]PETSC ERROR: #3 EPSSolve_Arnoldi() line 124 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/arnoldi/arnoldi.c > [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c > > > Is this related to the loss of positive-definite nature of the stiffness matrix? Is there a way I can still get solution form slepc? > > I would greatly appreciate any advice. > > Regards, > Manav > You are using the Arnoldi solver, I would strongly recommend the default solver (Krylov-Schur) - although it may not be relevant for this problem. I assume you are using MUMPS for the linear solves. For a better diagnostic I would need to have access to your matrices. Is it possible? Send an email to slepc-maint. Jose From bhatiamanav at gmail.com Sun Apr 3 14:45:07 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Sun, 3 Apr 2016 14:45:07 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> Message-ID: <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> Hi Jose, I did not specify Arnoldi. I am running on a single CPU, so maybe Arnoldi is the default for 1 cpu runs? How would you like me to send you the matrices? Is there a specific format you would like? Thanks, Manav > On Apr 3, 2016, at 2:39 PM, Jose E. Roman wrote: > > >> El 3 abr 2016, a las 20:47, Manav Bhatia escribi?: >> >> Hi, >> >> I am using slepc to solve for the natural frequencies of a small-disturbance modal eigensolution of a plate with compressive stresses. >> >> M x = (1/omega^2) K(lambda) x >> >> lambda is the loading parameter. I first solve for the nonlinear static solution for a given load, which gives K (after linearization), which is used here. >> >> Note that I am solving for 1/omega^2, instead of omega^2, since the mass matrix may have zero entries related to some dofs, while the stiffness matrix of the constrained structure has no zero eigenvalues (except, perhaps in the nonlinear region at the bifurcation point). >> >> As lambda increases, the structure crosses into the post-buckled region, so a bifurcation point is crossed at some lambda. The fundamental natural frequency with this tangent stiffness matrix is expected to reduce to zero, and then increase after the bifurcation point. >> >> I am calling slepc without any command line options, except that I specify LU-decomposition as the linear solver. >> >> The challenge that I am facing is that I am able to get eigensolutions till the lowest natural frequency drops to 0, and then the solver quits with the following error message: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Error in external library >> [0]PETSC ERROR: Error in Lapack xHSEQR 100 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: /Users/manav/Documents/codes/MAST/build/opt/example_driver on a arch-darwin-cxx-opt named Dhcp-90-243.HPC.MsState.Edu by manav Sun Apr 3 13:31:08 2016 >> [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/ >> local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes - >> -download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 -- >> download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 >> [0]PETSC ERROR: #1 DSSolve_NHEP() line 622 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/nhep/dsnhep.c >> [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c >> [0]PETSC ERROR: #3 EPSSolve_Arnoldi() line 124 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/arnoldi/arnoldi.c >> [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c >> >> >> Is this related to the loss of positive-definite nature of the stiffness matrix? Is there a way I can still get solution form slepc? >> >> I would greatly appreciate any advice. >> >> Regards, >> Manav >> > > You are using the Arnoldi solver, I would strongly recommend the default solver (Krylov-Schur) - although it may not be relevant for this problem. > I assume you are using MUMPS for the linear solves. > > For a better diagnostic I would need to have access to your matrices. Is it possible? Send an email to slepc-maint. > > Jose > From jroman at dsic.upv.es Sun Apr 3 15:03:29 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sun, 3 Apr 2016 22:03:29 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> Message-ID: <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> > El 3 abr 2016, a las 21:45, Manav Bhatia escribi?: > > Hi Jose, > > I did not specify Arnoldi. I am running on a single CPU, so maybe Arnoldi is the default for 1 cpu runs? No. Maybe you set it in code or maybe the option is being picked from PETSC_OPTIONS or petscrc file. > > How would you like me to send you the matrices? Is there a specific format you would like? Better in PETSc binary format. If calling EPSSolve() only once, the easiest way is running with -eps_view_mat0 binary:A.petsc -eps_view_mat1 binary:B.petsc > > Thanks, > Manav From bhatiamanav at gmail.com Sun Apr 3 15:17:42 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Sun, 3 Apr 2016 15:17:42 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> Message-ID: <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> I just send you the matrices. Thanks, Manav > On Apr 3, 2016, at 3:03 PM, Jose E. Roman wrote: > > >> El 3 abr 2016, a las 21:45, Manav Bhatia escribi?: >> >> Hi Jose, >> >> I did not specify Arnoldi. I am running on a single CPU, so maybe Arnoldi is the default for 1 cpu runs? > > No. Maybe you set it in code or maybe the option is being picked from PETSC_OPTIONS or petscrc file. > >> >> How would you like me to send you the matrices? Is there a specific format you would like? > > Better in PETSc binary format. If calling EPSSolve() only once, the easiest way is running with > -eps_view_mat0 binary:A.petsc -eps_view_mat1 binary:B.petsc > > >> >> Thanks, >> Manav > From zonexo at gmail.com Mon Apr 4 02:13:45 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 4 Apr 2016 15:13:45 +0800 Subject: [petsc-users] Partition in x direction is too fine error Message-ID: <57021429.6020703@gmail.com> Hi, I tried to run a case with dm and I got the error msg: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Partition in x direction is too fine! 3 4 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./a.out on a petsc-3.6.3_intel_rel named std0210 by tsltaywb Mon Apr 4 15:10:06 2016 [0]PETSC ERROR: Configure options --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 --download-hypre=/home/nus/tsltaywb/source/hypre-2.10.0b-p2.tar.gz --with-debugging=0 --prefix=//home/nus/tsltaywb/lib/petsc-3.6.3_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 My grid size is size_x,size_y,size_z 128 217 210 and I'm using 96 procs. May I know what's wrong? -- Thank you Yours sincerely, TAY wee-beng From jroman at dsic.upv.es Mon Apr 4 03:46:35 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 10:46:35 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> Message-ID: <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> > El 3 abr 2016, a las 22:17, Manav Bhatia escribi?: > > I just send you the matrices. > > Thanks, > Manav I cannot reproduce the problem. I was able to compute the eigenvalues without problems with Krylov-Schur (note that I scaled matrix A by 1e7 because the eigenvalues are tiny): $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian Generalized eigenproblem stored in file. Reading REAL matrices from binary files... Number of iterations of the method: 1 Number of linear iterations of the method: 17 Solution method: krylovschur Number of requested eigenvalues: 1 Stopping condition: tol=1e-08, maxit=768 Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 ---------------------- -------------------- k ||Ax-kBx||/||kx|| ---------------------- -------------------- 11.410228 3.18851e-09 1.323631 2.33005e-09 1.323631 3.98884e-09 0.224497 1.24334e-08 0.168344 9.81685e-08 0.129229 4.85372e-08 ---------------------- -------------------- Note that I am solving the problem as a GHEP, and this gives far better residuals than solving as a GNHEP. The Arnoldi solver did not fail either. I don't know what caused your error. You can try configuring with the source Lapack: --download-f2cblaslapack Jose From knepley at gmail.com Mon Apr 4 07:28:17 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 4 Apr 2016 07:28:17 -0500 Subject: [petsc-users] Partition in x direction is too fine error In-Reply-To: <57021429.6020703@gmail.com> References: <57021429.6020703@gmail.com> Message-ID: On Mon, Apr 4, 2016 at 2:13 AM, TAY wee-beng wrote: > Hi, > > I tried to run a case with dm and I got the error msg: > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Partition in x direction is too fine! 3 4 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./a.out on a petsc-3.6.3_intel_rel named std0210 by > tsltaywb Mon Apr 4 15:10:06 2016 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 > --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 > --download-hypre=/home/nus/tsltaywb/source/hypre-2.10.0b-p2.tar.gz > --with-debugging=0 --prefix=//home/nus/tsltaywb/lib/petsc-3.6.3_intel_rel > --with-shared-libraries=0 --known-mpi-shared-libraries=0 > --with-fortran-interfaces=1 > > My grid size is size_x,size_y,size_z 128 217 210 > > and I'm using 96 procs. > > May I know what's wrong? You must have some other grids being partitioned as well. We need to see the entire stack trace. Here it is saying that the size in x (3) is less than the number of processors in x (4), so maybe you are using multigrid or grid sequencing. Matt > -- > Thank you > > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Mon Apr 4 08:50:07 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 08:50:07 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> Message-ID: Thanks for looking into this, Jose. I will try with the source LAPACK. I am currently running on MAC OS X with the system provided lapack and bias. I have heard of some issues with these libraries, but have not had any issues until now. Hopefully that will sort the issue. I will let you know what I find. Thanks again! Manav > On Apr 4, 2016, at 3:46 AM, Jose E. Roman wrote: > > >> El 3 abr 2016, a las 22:17, Manav Bhatia escribi?: >> >> I just send you the matrices. >> >> Thanks, >> Manav > > I cannot reproduce the problem. I was able to compute the eigenvalues without problems with Krylov-Schur (note that I scaled matrix A by 1e7 because the eigenvalues are tiny): > > $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 17 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > 11.410228 3.18851e-09 > 1.323631 2.33005e-09 > 1.323631 3.98884e-09 > 0.224497 1.24334e-08 > 0.168344 9.81685e-08 > 0.129229 4.85372e-08 > ---------------------- -------------------- > > Note that I am solving the problem as a GHEP, and this gives far better residuals than solving as a GNHEP. The Arnoldi solver did not fail either. > > I don't know what caused your error. You can try configuring with the source Lapack: --download-f2cblaslapack > > Jose > From bhatiamanav at gmail.com Mon Apr 4 09:42:42 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 09:42:42 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> Message-ID: Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. Not sure how to go about debugging this. -Manav > On Apr 4, 2016, at 3:46 AM, Jose E. Roman wrote: > > >> El 3 abr 2016, a las 22:17, Manav Bhatia escribi?: >> >> I just send you the matrices. >> >> Thanks, >> Manav > > I cannot reproduce the problem. I was able to compute the eigenvalues without problems with Krylov-Schur (note that I scaled matrix A by 1e7 because the eigenvalues are tiny): > > $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 17 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > 11.410228 3.18851e-09 > 1.323631 2.33005e-09 > 1.323631 3.98884e-09 > 0.224497 1.24334e-08 > 0.168344 9.81685e-08 > 0.129229 4.85372e-08 > ---------------------- -------------------- > > Note that I am solving the problem as a GHEP, and this gives far better residuals than solving as a GNHEP. The Arnoldi solver did not fail either. > > I don't know what caused your error. You can try configuring with the source Lapack: --download-f2cblaslapack > > Jose > From jroman at dsic.upv.es Mon Apr 4 09:47:06 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 16:47:06 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> Message-ID: <1FA8C476-F0C6-4B57-A1E9-01D470054214@dsic.upv.es> > El 4 abr 2016, a las 16:42, Manav Bhatia escribi?: > > Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. > > I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. > > Not sure how to go about debugging this. > > -Manav What is different in your code with respect to ex7? Run with -eps_view and compare both. From bhatiamanav at gmail.com Mon Apr 4 09:51:48 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 09:51:48 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <1FA8C476-F0C6-4B57-A1E9-01D470054214@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <1FA8C476-F0C6-4B57-A1E9-01D470054214@dsic.upv.es> Message-ID: <1F5B3EA1-9248-4BB0-869A-790344553BE1@gmail.com> Following is the output of -eps_view from ex7 and from my code. I have included only what is different. There are differences in nev, ncv, mpd, etc. -Manav output from ex7: EPS Object: 1 MPI processes type: arnoldi problem type: generalized symmetric eigenvalue problem selected portion of the spectrum: largest eigenvalues in magnitude postprocessing eigenvectors with purification number of eigenvalues (nev): 1 number of column vectors (ncv): 16 maximum dimension of projected problem (mpd): 16 maximum number of iterations: 768 tolerance: 1e-08 convergence test: relative to the eigenvalue BV Object: 1 MPI processes type: svec 17 columns of global length 6150 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) block orthogonalization method: Gram-Schmidt non-standard inner product Mat Object: 1 MPI processes type: seqaij rows=6150, cols=6150 total: nonzeros=314892, allocated nonzeros=314892 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 2050 nodes, limit used is 5 doing matmult as a single matrix-matrix product output from my code: EPS Object: 1 MPI processes type: arnoldi problem type: generalized symmetric eigenvalue problem selected portion of the spectrum: largest eigenvalues in magnitude postprocessing eigenvectors with purification number of eigenvalues (nev): 20 number of column vectors (ncv): 100 maximum dimension of projected problem (mpd): 100 maximum number of iterations: 10000 tolerance: 1e-05 convergence test: relative to the eigenvalue BV Object: 1 MPI processes type: svec 101 columns of global length 6150 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) block orthogonalization method: Gram-Schmidt non-standard inner product Mat Object: 1 MPI processes type: seqaij rows=6150, cols=6150 total: nonzeros=314892, allocated nonzeros=314892 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 2050 nodes, limit used is 5 doing matmult as a single matrix-matrix product > On Apr 4, 2016, at 9:47 AM, Jose E. Roman wrote: > > >> El 4 abr 2016, a las 16:42, Manav Bhatia escribi?: >> >> Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. >> >> I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. >> >> Not sure how to go about debugging this. >> >> -Manav > > What is different in your code with respect to ex7? Run with -eps_view and compare both. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 4 09:52:12 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 4 Apr 2016 09:52:12 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> Message-ID: On Mon, Apr 4, 2016 at 9:42 AM, Manav Bhatia wrote: > Ok. So, I ran ex7 with the same command-line options in your email, and > got a result. This is on my Mac OS X, without any changes to the > lapack/blas/slepc library. > > I also ran my code on a linux machine with lapack/blas build from source > (obtained from netlib), and got the same error as on my mac. > > Not sure how to go about debugging this. > Sounds like a memory error. Run with valgrind. Matt > > -Manav > > > > On Apr 4, 2016, at 3:46 AM, Jose E. Roman wrote: > > > > > >> El 3 abr 2016, a las 22:17, Manav Bhatia > escribi?: > >> > >> I just send you the matrices. > >> > >> Thanks, > >> Manav > > > > I cannot reproduce the problem. I was able to compute the eigenvalues > without problems with Krylov-Schur (note that I scaled matrix A by 1e7 > because the eigenvalues are tiny): > > > > $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian > > > > Generalized eigenproblem stored in file. > > > > Reading REAL matrices from binary files... > > Number of iterations of the method: 1 > > Number of linear iterations of the method: 17 > > Solution method: krylovschur > > > > Number of requested eigenvalues: 1 > > Stopping condition: tol=1e-08, maxit=768 > > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; > iterations 1 > > ---------------------- -------------------- > > k ||Ax-kBx||/||kx|| > > ---------------------- -------------------- > > 11.410228 3.18851e-09 > > 1.323631 2.33005e-09 > > 1.323631 3.98884e-09 > > 0.224497 1.24334e-08 > > 0.168344 9.81685e-08 > > 0.129229 4.85372e-08 > > ---------------------- -------------------- > > > > Note that I am solving the problem as a GHEP, and this gives far better > residuals than solving as a GNHEP. The Arnoldi solver did not fail either. > > > > I don't know what caused your error. You can try configuring with the > source Lapack: --download-f2cblaslapack > > > > Jose > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Mon Apr 4 10:03:31 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 10:03:31 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <1FA8C476-F0C6-4B57-A1E9-01D470054214@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <1FA8C476-F0C6-4B57-A1E9-01D470054214@dsic.upv.es> Message-ID: It turns out that the matrices that I sent you was from a load step before the one that failed. I am calling eps_solve once for each load step, and I get the error at the 14th load step. So, the matrices that I sent were from step 13. The eigenvalues you got match the 1/lambda that I get from my code at load step 13. I will try to output the offending matrices and share those with you. Regards, Manav > On Apr 4, 2016, at 9:47 AM, Jose E. Roman wrote: > > >> El 4 abr 2016, a las 16:42, Manav Bhatia escribi?: >> >> Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. >> >> I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. >> >> Not sure how to go about debugging this. >> >> -Manav > > What is different in your code with respect to ex7? Run with -eps_view and compare both. > From bhatiamanav at gmail.com Mon Apr 4 10:49:07 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 10:49:07 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> Message-ID: <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> I just sent you the updated matrices on slepc-main. Sorry about sending the wrong matrices earlier. Regards, Manav > On Apr 4, 2016, at 9:42 AM, Manav Bhatia wrote: > > Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. > > I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. > > Not sure how to go about debugging this. > > -Manav > > >> On Apr 4, 2016, at 3:46 AM, Jose E. Roman wrote: >> >> >>> El 3 abr 2016, a las 22:17, Manav Bhatia escribi?: >>> >>> I just send you the matrices. >>> >>> Thanks, >>> Manav >> >> I cannot reproduce the problem. I was able to compute the eigenvalues without problems with Krylov-Schur (note that I scaled matrix A by 1e7 because the eigenvalues are tiny): >> >> $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian >> >> Generalized eigenproblem stored in file. >> >> Reading REAL matrices from binary files... >> Number of iterations of the method: 1 >> Number of linear iterations of the method: 17 >> Solution method: krylovschur >> >> Number of requested eigenvalues: 1 >> Stopping condition: tol=1e-08, maxit=768 >> Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 >> ---------------------- -------------------- >> k ||Ax-kBx||/||kx|| >> ---------------------- -------------------- >> 11.410228 3.18851e-09 >> 1.323631 2.33005e-09 >> 1.323631 3.98884e-09 >> 0.224497 1.24334e-08 >> 0.168344 9.81685e-08 >> 0.129229 4.85372e-08 >> ---------------------- -------------------- >> >> Note that I am solving the problem as a GHEP, and this gives far better residuals than solving as a GNHEP. The Arnoldi solver did not fail either. >> >> I don't know what caused your error. You can try configuring with the source Lapack: --download-f2cblaslapack >> >> Jose >> > From bhatiamanav at gmail.com Mon Apr 4 11:06:33 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 11:06:33 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> Message-ID: <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> Hi Jose, I also read these matrices into matlab and found the eigenvalues as >>A = PetscBinaryRead('A.petsc?); >>B = PetscBinaryRead(?B.petsc?); >> [v,d] = eigs(A,B) (*** got a lot of output about poor-conditioning ***) >> diag(d) ans = 1.0e-05 * -0.2219 0.0229 0.0229 0.0025 0.0019 0.0014 >> So, one of these is turning out to be negative, but I am still getting numbers. Using these matrices with ex7 produces an error: Dhcp-90-243:T_400_V_0 manav$ /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian -eps_view -eps_nev 1 Generalized eigenproblem stored in file. Reading REAL matrices from binary files... [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in Lapack xSTEQR 15 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 on a arch-darwin-cxx-opt named ws243-49.walker.dynamic.msstate.edu by manav Mon Apr 4 11:05:30 2016 [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 --download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 [0]PETSC ERROR: #1 DSSolve_HEP_QR() line 495 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/hep/dshep.c [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 68 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c [0]PETSC ERROR: #5 main() line 147 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -eps_gen_hermitian [0]PETSC ERROR: -eps_nev 1 [0]PETSC ERROR: -eps_view [0]PETSC ERROR: -f1 A.petsc [0]PETSC ERROR: -f2 B.petsc [0]PETSC ERROR: -matload_block_size 1 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 76. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- > On Apr 4, 2016, at 10:49 AM, Manav Bhatia wrote: > > I just sent you the updated matrices on slepc-main. > > Sorry about sending the wrong matrices earlier. > > Regards, > Manav > > >> On Apr 4, 2016, at 9:42 AM, Manav Bhatia wrote: >> >> Ok. So, I ran ex7 with the same command-line options in your email, and got a result. This is on my Mac OS X, without any changes to the lapack/blas/slepc library. >> >> I also ran my code on a linux machine with lapack/blas build from source (obtained from netlib), and got the same error as on my mac. >> >> Not sure how to go about debugging this. >> >> -Manav >> >> >>> On Apr 4, 2016, at 3:46 AM, Jose E. Roman wrote: >>> >>> >>>> El 3 abr 2016, a las 22:17, Manav Bhatia escribi?: >>>> >>>> I just send you the matrices. >>>> >>>> Thanks, >>>> Manav >>> >>> I cannot reproduce the problem. I was able to compute the eigenvalues without problems with Krylov-Schur (note that I scaled matrix A by 1e7 because the eigenvalues are tiny): >>> >>> $ ./ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian >>> >>> Generalized eigenproblem stored in file. >>> >>> Reading REAL matrices from binary files... >>> Number of iterations of the method: 1 >>> Number of linear iterations of the method: 17 >>> Solution method: krylovschur >>> >>> Number of requested eigenvalues: 1 >>> Stopping condition: tol=1e-08, maxit=768 >>> Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 >>> ---------------------- -------------------- >>> k ||Ax-kBx||/||kx|| >>> ---------------------- -------------------- >>> 11.410228 3.18851e-09 >>> 1.323631 2.33005e-09 >>> 1.323631 3.98884e-09 >>> 0.224497 1.24334e-08 >>> 0.168344 9.81685e-08 >>> 0.129229 4.85372e-08 >>> ---------------------- -------------------- >>> >>> Note that I am solving the problem as a GHEP, and this gives far better residuals than solving as a GNHEP. The Arnoldi solver did not fail either. >>> >>> I don't know what caused your error. You can try configuring with the source Lapack: --download-f2cblaslapack >>> >>> Jose >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Apr 4 11:18:01 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 18:18:01 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> Message-ID: <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> > El 4 abr 2016, a las 18:06, Manav Bhatia escribi?: > > Hi Jose, > > I also read these matrices into matlab and found the eigenvalues as > > >>A = PetscBinaryRead('A.petsc?); > >>B = PetscBinaryRead(?B.petsc?); > >> [v,d] = eigs(A,B) > (*** got a lot of output about poor-conditioning ***) > >> diag(d) > > ans = > > 1.0e-05 * > > -0.2219 > 0.0229 > 0.0229 > 0.0025 > 0.0019 > 0.0014 > > >> So, one of these is turning out to be negative, but I am still getting numbers. > > > > Using these matrices with ex7 produces an error: > > Dhcp-90-243:T_400_V_0 manav$ /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian -eps_view -eps_nev 1 > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in Lapack xSTEQR 15 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 on a arch-darwin-cxx-opt named ws243-49.walker.dynamic.msstate.edu by manav Mon Apr 4 11:05:30 2016 > [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 --download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 > [0]PETSC ERROR: #1 DSSolve_HEP_QR() line 495 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/hep/dshep.c > [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c > [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 68 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c > [0]PETSC ERROR: #5 main() line 147 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -eps_gen_hermitian > [0]PETSC ERROR: -eps_nev 1 > [0]PETSC ERROR: -eps_view > [0]PETSC ERROR: -f1 A.petsc > [0]PETSC ERROR: -f2 B.petsc > [0]PETSC ERROR: -matload_block_size 1 > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 76. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > The problem is that now K is indefinite. The check for indefinite B-matrix was not done correctly. This is fixed in 3.6.3, where you should get a more informative error: [0]PETSC ERROR: The inner product is not well defined: indefinite matrix When considered as a non-symmetric problem, the solver returns reasonable output: $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc Generalized eigenproblem stored in file. Reading REAL matrices from binary files... Number of iterations of the method: 1 Number of linear iterations of the method: 16 Solution method: krylovschur Number of requested eigenvalues: 1 Stopping condition: tol=1e-08, maxit=768 Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 ---------------------- -------------------- k ||Ax-kBx||/||kx|| ---------------------- -------------------- -22.185956 8.66397e-08 2.291422 1.32466e-07 2.291422 1.55081e-07 0.252494 8.88997e-05 0.192453 0.000780395 0.141618 0.00113141 ---------------------- -------------------- The large residuals are due to bad scaling/conditioning of your matrices. You can also try the symmetric-indefinite solver, but this is not recommended since it is not numerically stable. $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc -eps_gen_indefinite Generalized eigenproblem stored in file. Reading REAL matrices from binary files... Number of iterations of the method: 1 Number of linear iterations of the method: 17 Solution method: krylovschur Number of requested eigenvalues: 1 Stopping condition: tol=1e-08, maxit=768 Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 ---------------------- -------------------- k ||Ax-kBx||/||kx|| ---------------------- -------------------- -22.185956 7.08858e-09 2.291422 5.87577e-09 2.291422 6.39304e-09 0.252494 1.7497e-07 0.192453 1.63916e-06 0.141618 1.75568e-06 ---------------------- -------------------- Jose From bhatiamanav at gmail.com Mon Apr 4 11:23:33 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 11:23:33 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> Message-ID: <59F968CA-031B-474D-AC64-C389E2FF18FC@gmail.com> Thanks, Jose! I am currently running 3.6.2, and will update to 3.6.3. Is there a recommended strategy to automatically switch from GHEP to GNHEP for some subset of problems? Or should I choose to run all my eigenprobelms with GNHEP? Regards, Manav > On Apr 4, 2016, at 11:18 AM, Jose E. Roman wrote: > >> >> El 4 abr 2016, a las 18:06, Manav Bhatia escribi?: >> >> Hi Jose, >> >> I also read these matrices into matlab and found the eigenvalues as >> >>>> A = PetscBinaryRead('A.petsc?); >>>> B = PetscBinaryRead(?B.petsc?); >>>> [v,d] = eigs(A,B) >> (*** got a lot of output about poor-conditioning ***) >>>> diag(d) >> >> ans = >> >> 1.0e-05 * >> >> -0.2219 >> 0.0229 >> 0.0229 >> 0.0025 >> 0.0019 >> 0.0014 >> >>>> So, one of these is turning out to be negative, but I am still getting numbers. >> >> >> >> Using these matrices with ex7 produces an error: >> >> Dhcp-90-243:T_400_V_0 manav$ /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 -f1 A.petsc -f2 B.petsc -eps_gen_hermitian -eps_view -eps_nev 1 >> >> Generalized eigenproblem stored in file. >> >> Reading REAL matrices from binary files... >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Error in external library >> [0]PETSC ERROR: Error in Lapack xSTEQR 15 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 on a arch-darwin-cxx-opt named ws243-49.walker.dynamic.msstate.edu by manav Mon Apr 4 11:05:30 2016 >> [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 --download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 >> [0]PETSC ERROR: #1 DSSolve_HEP_QR() line 495 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/hep/dshep.c >> [0]PETSC ERROR: #2 DSSolve() line 543 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c >> [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 68 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/krylovschur/ks-symm.c >> [0]PETSC ERROR: #4 EPSSolve() line 101 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c >> [0]PETSC ERROR: #5 main() line 147 in /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7.c >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -eps_gen_hermitian >> [0]PETSC ERROR: -eps_nev 1 >> [0]PETSC ERROR: -eps_view >> [0]PETSC ERROR: -f1 A.petsc >> [0]PETSC ERROR: -f2 B.petsc >> [0]PETSC ERROR: -matload_block_size 1 >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 76. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> > > The problem is that now K is indefinite. The check for indefinite B-matrix was not done correctly. This is fixed in 3.6.3, where you should get a more informative error: > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix > > When considered as a non-symmetric problem, the solver returns reasonable output: > > $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 16 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > -22.185956 8.66397e-08 > 2.291422 1.32466e-07 > 2.291422 1.55081e-07 > 0.252494 8.88997e-05 > 0.192453 0.000780395 > 0.141618 0.00113141 > ---------------------- -------------------- > > The large residuals are due to bad scaling/conditioning of your matrices. You can also try the symmetric-indefinite solver, but this is not recommended since it is not numerically stable. > > $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc -eps_gen_indefinite > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 17 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > -22.185956 7.08858e-09 > 2.291422 5.87577e-09 > 2.291422 6.39304e-09 > 0.252494 1.7497e-07 > 0.192453 1.63916e-06 > 0.141618 1.75568e-06 > ---------------------- -------------------- > > > Jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Apr 4 11:30:46 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 18:30:46 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <59F968CA-031B-474D-AC64-C389E2FF18FC@gmail.com> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <59F968CA-031B-474D-AC64-C389E2FF18FC@gmail.com> Message-ID: > El 4 abr 2016, a las 18:23, Manav Bhatia escribi?: > > Thanks, Jose! > > I am currently running 3.6.2, and will update to 3.6.3. > > Is there a recommended strategy to automatically switch from GHEP to GNHEP for some subset of problems? Or should I choose to run all my eigenprobelms with GNHEP? > > Regards, > Manav Well you could consider doing a trick like this: PetscPushErrorHandler(PetscIgnoreErrorHandler,NULL); ierr = EPSSolve(eps); PetscPopErrorHandler(); if (ierr) { ierr = EPSSetProblemType(eps,EPS_GNHEP);CHKERRQ(ierr); ierr = EPSSolve(eps);CHKERRQ(ierr); } But this is very ugly and does not discriminate one particular error from others. A (costly) alternative would be to compute a rough approximation of the leftmost eigenvalue of your K matrix to see if it is negative. Jose From david.knezevic at akselos.com Mon Apr 4 11:32:02 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 4 Apr 2016 12:32:02 -0400 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> Message-ID: Hi Jose, I'm interested to know more about this comment from your email below: "You can also try the symmetric-indefinite solver, but this is not recommended since it is not numerically stable." I frequently use SLEPc's symmetric-indefinite solver for buckling eigenvalue problems (which are naturally symmetric and indefinite), and this works well in my experience. But, based on your comment, I was wondering if you would recommend a different solver for this case? Thanks, David On Mon, Apr 4, 2016 at 12:18 PM, Jose E. Roman wrote: > > > El 4 abr 2016, a las 18:06, Manav Bhatia > escribi?: > > > > Hi Jose, > > > > I also read these matrices into matlab and found the eigenvalues as > > > > >>A = PetscBinaryRead('A.petsc?); > > >>B = PetscBinaryRead(?B.petsc?); > > >> [v,d] = eigs(A,B) > > (*** got a lot of output about poor-conditioning ***) > > >> diag(d) > > > > ans = > > > > 1.0e-05 * > > > > -0.2219 > > 0.0229 > > 0.0229 > > 0.0025 > > 0.0019 > > 0.0014 > > > > >> So, one of these is turning out to be negative, but I am still > getting numbers. > > > > > > > > Using these matrices with ex7 produces an error: > > > > Dhcp-90-243:T_400_V_0 manav$ > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 > -f1 A.petsc -f2 B.petsc -eps_gen_hermitian -eps_view -eps_nev 1 > > > > Generalized eigenproblem stored in file. > > > > Reading REAL matrices from binary files... > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Error in external library > > [0]PETSC ERROR: Error in Lapack xSTEQR 15 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7 > on a arch-darwin-cxx-opt named ws243-49.walker.dynamic.msstate.edu by > manav Mon Apr 4 11:05:30 2016 > > [0]PETSC ERROR: Configure options > --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.6.3/../ > --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 > --with-mpiexec=/opt/local/bin/mpiexec --with-shared-libraries=1 --with-x=1 > --with-x-dir=/opt/X11 --with-debugging=0 > --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes > --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes > --download-scalapack=yes --download-parmetis=yes > --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 > --download-hypre=yes --download-hypre-shared=1 --download-ml=yes > --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 > > [0]PETSC ERROR: #1 DSSolve_HEP_QR() line 495 in > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/impls/hep/dshep.c > > [0]PETSC ERROR: #2 DSSolve() line 543 in > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c > > [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 68 in > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/impls/krylov/krylovschur/ks-symm.c > > [0]PETSC ERROR: #4 EPSSolve() line 101 in > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/interface/epssolve.c > > [0]PETSC ERROR: #5 main() line 147 in > /Users/manav/Documents/codes/numerical_lib/slepc/slepc-3.6.2/src/eps/examples/tutorials/ex7.c > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -eps_gen_hermitian > > [0]PETSC ERROR: -eps_nev 1 > > [0]PETSC ERROR: -eps_view > > [0]PETSC ERROR: -f1 A.petsc > > [0]PETSC ERROR: -f2 B.petsc > > [0]PETSC ERROR: -matload_block_size 1 > > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 76. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > > -------------------------------------------------------------------------- > > > > The problem is that now K is indefinite. The check for indefinite B-matrix > was not done correctly. This is fixed in 3.6.3, where you should get a more > informative error: > > [0]PETSC ERROR: The inner product is not well defined: indefinite matrix > > When considered as a non-symmetric problem, the solver returns reasonable > output: > > $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 16 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; > iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > -22.185956 8.66397e-08 > 2.291422 1.32466e-07 > 2.291422 1.55081e-07 > 0.252494 8.88997e-05 > 0.192453 0.000780395 > 0.141618 0.00113141 > ---------------------- -------------------- > > The large residuals are due to bad scaling/conditioning of your matrices. > You can also try the symmetric-indefinite solver, but this is not > recommended since it is not numerically stable. > > $ ./ex7 -f1 ~/tmp/bhatia/A.petsc -f2 ~/tmp/bhatia/B.petsc > -eps_gen_indefinite > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 1 > Number of linear iterations of the method: 17 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-08, maxit=768 > Linear eigensolve converged (6 eigenpairs) due to CONVERGED_TOL; > iterations 1 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > -22.185956 7.08858e-09 > 2.291422 5.87577e-09 > 2.291422 6.39304e-09 > 0.252494 1.7497e-07 > 0.192453 1.63916e-06 > 0.141618 1.75568e-06 > ---------------------- -------------------- > > > Jose > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Mon Apr 4 11:43:01 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 4 Apr 2016 11:43:01 -0500 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <59F968CA-031B-474D-AC64-C389E2FF18FC@gmail.com> Message-ID: Thanks! I upgraded to 3.6.3 and ran the code, and it went through all the load steps without problem. A followup question: It seems like the eigenvalues are sorted based on abs(lambda). If I am interested in only the lowest N eigenvalues greater than 0, is there a way to tell slepc to ignore all eigenvalues less than 0? Thanks, Manav > On Apr 4, 2016, at 11:30 AM, Jose E. Roman wrote: > > >> El 4 abr 2016, a las 18:23, Manav Bhatia escribi?: >> >> Thanks, Jose! >> >> I am currently running 3.6.2, and will update to 3.6.3. >> >> Is there a recommended strategy to automatically switch from GHEP to GNHEP for some subset of problems? Or should I choose to run all my eigenprobelms with GNHEP? >> >> Regards, >> Manav > > Well you could consider doing a trick like this: > > PetscPushErrorHandler(PetscIgnoreErrorHandler,NULL); > ierr = EPSSolve(eps); > PetscPopErrorHandler(); > if (ierr) { > ierr = EPSSetProblemType(eps,EPS_GNHEP);CHKERRQ(ierr); > ierr = EPSSolve(eps);CHKERRQ(ierr); > } > > But this is very ugly and does not discriminate one particular error from others. A (costly) alternative would be to compute a rough approximation of the leftmost eigenvalue of your K matrix to see if it is negative. > > Jose > From jroman at dsic.upv.es Mon Apr 4 11:47:53 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 18:47:53 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> Message-ID: <77C9311E-F70C-47CD-BC20-2733F08AC4AD@dsic.upv.es> > El 4 abr 2016, a las 18:32, David Knezevic escribi?: > > Hi Jose, > > I'm interested to know more about this comment from your email below: > "You can also try the symmetric-indefinite solver, but this is not recommended since it is not numerically stable." > > I frequently use SLEPc's symmetric-indefinite solver for buckling eigenvalue problems (which are naturally symmetric and indefinite), and this works well in my experience. But, based on your comment, I was wondering if you would recommend a different solver for this case? > > Thanks, > David The symmetric-indefinite solver is based on non-symmetric Lanczos, which is not guaranteed to be numerically stable (it does not rely on orthogonal projections as Arnoldi). In practice, it means that the method may break down or get unstable (in our implementation we detect the latter). For many problems the indefinite solver will provide the solution without problems, but no guarantees it will always work. Details of the indefinite solver can be found in section 3 of this paper: http://dx.doi.org/10.1007/s10543-016-0601-5 Jose From jroman at dsic.upv.es Mon Apr 4 11:54:28 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 18:54:28 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <59F968CA-031B-474D-AC64-C389E2FF18FC@gmail.com> Message-ID: <744EDC7B-60D9-43AF-AA05-23C58E97F171@dsic.upv.es> > El 4 abr 2016, a las 18:43, Manav Bhatia escribi?: > > Thanks! > I upgraded to 3.6.3 and ran the code, and it went through all the load steps without problem. > > A followup question: > > It seems like the eigenvalues are sorted based on abs(lambda). If I am interested in only the lowest N eigenvalues greater than 0, is there a way to tell slepc to ignore all eigenvalues less than 0? > > Thanks, > Manav You can try a user-defined sorting criterion as in this example: http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex18.c.html From david.knezevic at akselos.com Mon Apr 4 12:13:49 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 4 Apr 2016 13:13:49 -0400 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: <77C9311E-F70C-47CD-BC20-2733F08AC4AD@dsic.upv.es> References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <77C9311E-F70C-47CD-BC20-2733F08AC4AD@dsic.upv.es> Message-ID: On Mon, Apr 4, 2016 at 12:47 PM, Jose E. Roman wrote: > > > El 4 abr 2016, a las 18:32, David Knezevic > escribi?: > > > > Hi Jose, > > > > I'm interested to know more about this comment from your email below: > > "You can also try the symmetric-indefinite solver, but this is not > recommended since it is not numerically stable." > > > > I frequently use SLEPc's symmetric-indefinite solver for buckling > eigenvalue problems (which are naturally symmetric and indefinite), and > this works well in my experience. But, based on your comment, I was > wondering if you would recommend a different solver for this case? > > > > Thanks, > > David > > The symmetric-indefinite solver is based on non-symmetric Lanczos, which > is not guaranteed to be numerically stable (it does not rely on orthogonal > projections as Arnoldi). In practice, it means that the method may break > down or get unstable (in our implementation we detect the latter). For many > problems the indefinite solver will provide the solution without problems, > but no guarantees it will always work. > > Details of the indefinite solver can be found in section 3 of this paper: > http://dx.doi.org/10.1007/s10543-016-0601-5 > > Jose > > OK, thanks, I'll have a look at that paper. And just to confirm that I've understood properly: You're saying that the GNHEP solver should converge more robustly than the GHIEP solver for symmetric indefinite problems? Are there any advantages to using GHIEP over GNHEP (e.g. is it faster?), or do you just recommend to always use GNHEP for symmetric indefinite problems? Apologies if the paper already answers those questions, I'll read through it soon. Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Apr 4 12:27:13 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 4 Apr 2016 19:27:13 +0200 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <77C9311E-F70C-47CD-BC20-2733F08AC4AD@dsic.upv.es> Message-ID: > El 4 abr 2016, a las 19:13, David Knezevic escribi?: > > OK, thanks, I'll have a look at that paper. > > And just to confirm that I've understood properly: You're saying that the GNHEP solver should converge more robustly than the GHIEP solver for symmetric indefinite problems? Are there any advantages to using GHIEP over GNHEP (e.g. is it faster?), or do you just recommend to always use GNHEP for symmetric indefinite problems? > > Apologies if the paper already answers those questions, I'll read through it soon. > > Thanks, > David In terms of linear algebra, a symmetric-indefinite matrix pencil is not symmetric (no guarantee it has real eigenvalues and a full set of linearly independent eigenvectors). So in principle a non-symmetric algorithm should be used. The GHIEP solver tries to enforce (pseudo-) symmetry throughout the computation. This may provide more accurate results or maybe converge faster, but the cost per iteration is higher (since it uses B-inner products rather than standard inner products as in Arnoldi). Maybe in problems arising in buckling analysis of structures it is always safe to use GHIEP, I don't know. Jose From david.knezevic at akselos.com Mon Apr 4 12:29:53 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 4 Apr 2016 13:29:53 -0400 Subject: [petsc-users] eigensolution error with slepc In-Reply-To: References: <613377E5-3F74-44E7-9106-7B1645834B17@gmail.com> <948BDA72-C4A0-47A9-BB09-9781C784EA23@dsic.upv.es> <88563426-E52B-44A9-812D-39D022AB3DE2@gmail.com> <5DBB64EC-7FCC-45AD-A03C-84F8AFC28D2C@dsic.upv.es> <3BD1B84B-F341-480A-A4C7-311762E901FA@gmail.com> <5742D94A-5C85-46AA-8A69-B7D677B30AA6@dsic.upv.es> <9924C1E5-05B9-41E9-BEDA-7400325EFBC3@gmail.com> <504FD3D2-5437-42D0-AC60-593833E6FFC6@gmail.com> <76F9E396-9E8D-4A1A-AF3B-8868C12581C7@dsic.upv.es> <77C9311E-F70C-47CD-BC20-2733F08AC4AD@dsic.upv.es> Message-ID: On Mon, Apr 4, 2016 at 1:27 PM, Jose E. Roman wrote: > > > El 4 abr 2016, a las 19:13, David Knezevic > escribi?: > > > > OK, thanks, I'll have a look at that paper. > > > > And just to confirm that I've understood properly: You're saying that > the GNHEP solver should converge more robustly than the GHIEP solver for > symmetric indefinite problems? Are there any advantages to using GHIEP over > GNHEP (e.g. is it faster?), or do you just recommend to always use GNHEP > for symmetric indefinite problems? > > > > Apologies if the paper already answers those questions, I'll read > through it soon. > > > > Thanks, > > David > > In terms of linear algebra, a symmetric-indefinite matrix pencil is not > symmetric (no guarantee it has real eigenvalues and a full set of linearly > independent eigenvectors). So in principle a non-symmetric algorithm should > be used. The GHIEP solver tries to enforce (pseudo-) symmetry throughout > the computation. This may provide more accurate results or maybe converge > faster, but the cost per iteration is higher (since it uses B-inner > products rather than standard inner products as in Arnoldi). Maybe in > problems arising in buckling analysis of structures it is always safe to > use GHIEP, I don't know. OK, got it, thanks! David -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.skjerven at gmail.com Tue Apr 5 03:38:32 2016 From: brian.skjerven at gmail.com (Brian Skjerven) Date: Tue, 5 Apr 2016 16:38:32 +0800 Subject: [petsc-users] Questions about DMComposite & DMRedundant Message-ID: Hi, I'm working on implementing pseudo arc-length continuation using PETSc to do the Newton solves. Right now I'm just working with simple structured meshes, so I'm using 1- & 2-D DMDAs, and I'm trying to make use of DMComposite to handle both the PDE and the pseudo arc-length constraint. I found a similar thread from a while back (https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2011-October/010373.html ) but some of the suggestions appear to be outdated (e.g. DMCompositeAddArray has been removed). So, I guess my questions are: - DMRedundant seems to be the preferred way to do this. Is there any documentation available? I've looked over the DMRedundantCreate man page (http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMRedundantCreate.html ) and the 2 examples, but am still unclear how this works. - Adding arrays to the DMComposite. So my system involves a bordered, banded matrix; the whole matrix is n+1 x n+1 in size, with the inner n x n being the diagonal matrix resulting from finite differences, the n + 1 column coming from the parameter (e.g. lambda in a Bratu problem), and the n + 1 row coming from the pseudo arc-length constraint. Conceptually, I see it as just "sticking" an extra row and column onto my existing DMDA, but I'm not sure how to do that with a combination of DMRedundant and DMComposite. For example, how do I tell PETSc which dimension to use (i.e. adding a row vs adding a column). - I'm providing routines to evaluate the function and the Jacobian. Do I need to do separate functions for each DM in my composite DM? That is, a function and Jacobian for the PDE portion and a separate set of functions for the arc-length constraint? Or do I just need to provide one and SNES is smart to figure things out for the composite DM? I guess I'm unclear how the coupling is handled. Thanks for your help. Brian Skjerven -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Apr 5 03:52:57 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 5 Apr 2016 16:52:57 +0800 Subject: [petsc-users] Partition in x direction is too fine error In-Reply-To: References: <57021429.6020703@gmail.com> Message-ID: <57037CE9.20805@gmail.com> HI Matt, Ya, there's a problem with my CAD body. Thank you Yours sincerely, TAY wee-beng On 4/4/2016 8:28 PM, Matthew Knepley wrote: > On Mon, Apr 4, 2016 at 2:13 AM, TAY wee-beng > wrote: > > Hi, > > I tried to run a case with dm and I got the error msg: > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Partition in x direction is too fine! 3 4 > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./a.out on a petsc-3.6.3_intel_rel named std0210 > by tsltaywb Mon Apr 4 15:10:06 2016 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64 > --with-blas-lapack-dir=/app/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 > --download-hypre=/home/nus/tsltaywb/source/hypre-2.10.0b-p2.tar.gz > --with-debugging=0 > --prefix=//home/nus/tsltaywb/lib/petsc-3.6.3_intel_rel > --with-shared-libraries=0 --known-mpi-shared-libraries=0 > --with-fortran-interfaces=1 > > My grid size is size_x,size_y,size_z 128 217 210 > > and I'm using 96 procs. > > May I know what's wrong? > > > You must have some other grids being partitioned as well. We need to > see the entire stack trace. Here > it is saying that the size in x (3) is less than the number of > processors in x (4), so maybe you are using > multigrid or grid sequencing. > > Matt > > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mono at mek.dtu.dk Tue Apr 5 07:11:27 2016 From: mono at mek.dtu.dk (=?Windows-1252?Q?Morten_Nobel-J=F8rgensen?=) Date: Tue, 5 Apr 2016 12:11:27 +0000 Subject: [petsc-users] DMPlex : Assemble global stiffness matrix problem In-Reply-To: References: <6B03D347796DED499A2696FC095CE81A013A556A@ait-pex02mbx04.win.dtu.dk> Message-ID: Hi Matthew We are now facing a related problem. When running on a single process everything works as intended, where the values of global stiffness matrix are between 0.0 and 2.0 (the number corresponds to the number of elements/cells which affects the relation between two dofs). However if the program is using 2 (or more) processes, then the global stiffness matrix changes, such that the values is now between 0.0 and 1.0 (even though we are using ADD_VALUES in the MatSetValuesLocal). Any idea what could be wrong? I have attached a modified version of the code, where the program outputs the matrix to a file called ?Kmat.m'. Kind regards, Morten Fra: Matthew Knepley > Dato: Tuesday 8 March 2016 at 19:47 Til: Morten Nobel-J?rgensen > Cc: "petsc-users at mcs.anl.gov" > Emne: Re: [petsc-users] DMPlex : Assemble global stiffness matrix problem On Tue, Mar 8, 2016 at 7:10 AM, Matthew Knepley > wrote: On Mon, Mar 7, 2016 at 1:28 PM, Morten Nobel-J?rgensen > wrote: I have some problems using DMPlex on unstructured grids in 3D. After I have created the DMPlex and assigned dofs (3 dofs on each node), I run into some problems when assembling the global stiffness matrix. I have created a small example in the attached cc file. My problems are: * It seems like the matrix (created using DMCreateMatrix) contains no non-zero elements. I was under the impression that the sparsity pattern of the matrix would be created automatically when the dofs has been assigned to the default section. * (Probably as a consequence of this) when assigning values to the matrix I get an: "Argument of of range. New nonzero at (0,0) caused a malloc. Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check" * Finally, I'm reading the nodes of each element using the get-transitive-clojure (where I test if each point is inside the node range), but I have a hard time understanding if the returned values are sorted. And if not, how to sort the values (e.g. using orientation which the get-transitive-clojure function also returns). I hope someone can guide me in the right direction :) I will take a look today or tomorrow. The first thing to do is to look at the nonzero pattern of the Jacobian. I use -mat_view draw -draw_pause -1 I admit that this problem is counter-intuitive, and I will think about a good error check. The problem is that I allow "inconsistent" sections, meaning that the dof for each field do not add up to the total dof. In your code, when you call ierr = PetscSectionSetDof(s, v, 3);CHKERRQ(ierr); you should also call ierr = PetscSectionSetFieldDof(s, v, 0, 3);CHKERRQ(ierr); Then everything works. I am attaching my slight rewrite of your code. Thanks, Matt Thanks, Matt Kind regards, Morten -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex18.cc Type: application/octet-stream Size: 4781 bytes Desc: ex18.cc URL: From bsmith at mcs.anl.gov Tue Apr 5 13:24:00 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 5 Apr 2016 13:24:00 -0500 Subject: [petsc-users] Questions about DMComposite & DMRedundant In-Reply-To: References: Message-ID: <9982B09D-A090-45FB-9D44-C5AD4B1D8729@mcs.anl.gov> Brian, Take a look at src/snes/examples/tutorials/ex21.c and ex22.c I think it should answer your questions. Barry > On Apr 5, 2016, at 3:38 AM, Brian Skjerven wrote: > > Hi, > > I'm working on implementing pseudo arc-length continuation using PETSc to do the Newton solves. Right now I'm just working with simple structured meshes, so I'm using 1- & 2-D DMDAs, and I'm trying to make use of DMComposite to handle both the PDE and the pseudo arc-length constraint. I found a similar thread from a while back (https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2011-October/010373.html) but some of the suggestions appear to be outdated (e.g. DMCompositeAddArray has been removed). So, I guess my questions are: > > - DMRedundant seems to be the preferred way to do this. Is there any documentation available? I've looked over the DMRedundantCreate man page (http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMRedundantCreate.html) and the 2 examples, but am still unclear how this works. > > - Adding arrays to the DMComposite. So my system involves a bordered, banded matrix; the whole matrix is n+1 x n+1 in size, with the inner n x n being the diagonal matrix resulting from finite differences, the n + 1 column coming from the parameter (e.g. lambda in a Bratu problem), and the n + 1 row coming from the pseudo arc-length constraint. Conceptually, I see it as just "sticking" an extra row and column onto my existing DMDA, but I'm not sure how to do that with a combination of DMRedundant and DMComposite. For example, how do I tell PETSc which dimension to use (i.e. adding a row vs adding a column). > > - I'm providing routines to evaluate the function and the Jacobian. Do I need to do separate functions for each DM in my composite DM? That is, a function and Jacobian for the PDE portion and a separate set of functions for the arc-length constraint? Or do I just need to provide one and SNES is smart to figure things out for the composite DM? I guess I'm unclear how the coupling is handled. > > Thanks for your help. > > Brian Skjerven From ppratapa at gatech.edu Tue Apr 5 18:04:20 2016 From: ppratapa at gatech.edu (Phanisri Pradeep Pratapa) Date: Tue, 5 Apr 2016 19:04:20 -0400 Subject: [petsc-users] Complex conjugate of PetscScalar Message-ID: Hi, I am trying to find the complex conjugate of a petsc scalar (petsc configured for complex numbers). Currently I do it by finding real and imaginary parts (using PetscRealPart, PetscImaginaryPart) and then subtracting them by multiplying the imaginary part with PETSC_i. Please let me know if there is a better way to do it for a single PetscScalar variable. Thank you, Regards, Pradeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 5 18:07:38 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 5 Apr 2016 18:07:38 -0500 Subject: [petsc-users] Complex conjugate of PetscScalar In-Reply-To: References: Message-ID: PetscConjComplex() > On Apr 5, 2016, at 6:04 PM, Phanisri Pradeep Pratapa wrote: > > Hi, > > I am trying to find the complex conjugate of a petsc scalar (petsc configured for complex numbers). Currently I do it by finding real and imaginary parts (using PetscRealPart, PetscImaginaryPart) and then subtracting them by multiplying the imaginary part with PETSC_i. Please let me know if there is a better way to do it for a single PetscScalar variable. > > Thank you, > > Regards, > > Pradeep From bsmith at mcs.anl.gov Tue Apr 5 18:45:38 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 5 Apr 2016 18:45:38 -0500 Subject: [petsc-users] Problems with FBCGSR In-Reply-To: <56FE7562.6050705@gmail.com> References: <56FE7492.4010806@gmail.com> <56FE7562.6050705@gmail.com> Message-ID: <7C287CD3-9C30-44AF-87A6-5E8A4BA266A6@mcs.anl.gov> I don't think FBCGSR has any guarantees that it will work so you could just be unlucky. If you really want to explore the issue (probably not worth it) you can run with really tight tolerances on the inner solvers (so they become essentially exact solvers and hence linear operators) and see if the FBCGSR converges. If it does convergence then slowly loosen the tolerances until it messes up the convergences of the FBCGSR Barry > On Apr 1, 2016, at 8:19 AM, Elias Karabelas wrote: > > > Hi, > > I'm using petsc for solving a non symmetric block system with a > field-split preconditioner. At the moment I use the fgmres as outer > solver. I recently discovered that there is also a flexible BICG variant > in petsc. So I wanted to give it a try. However I get strange behaviour > when using it. With FGMRES i got the following residual history > > 0 KSP unpreconditioned resid norm 8.632122370708e+00 true resid norm > 8.632122370708e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 5.881521061968e-04 true resid norm > 5.881521061969e-04 ||r(i)||/||b|| 6.813528364620e-05 > 2 KSP unpreconditioned resid norm 6.142969823931e-05 true resid norm > 6.142969823933e-05 ||r(i)||/||b|| 7.116407252031e-06 > 3 KSP unpreconditioned resid norm 4.110742198982e-05 true resid norm > 4.110742198982e-05 ||r(i)||/||b|| 4.762145417367e-06 > 4 KSP unpreconditioned resid norm 2.628280876832e-05 true resid norm > 2.628280876817e-05 ||r(i)||/||b|| 3.044767861187e-06 > 5 KSP unpreconditioned resid norm 1.293845872473e-05 true resid norm > 1.293845872450e-05 ||r(i)||/||b|| 1.498873413612e-06 > 6 KSP unpreconditioned resid norm 6.930434272105e-06 true resid norm > 6.930434271781e-06 ||r(i)||/||b|| 8.028656191550e-07 > 7 KSP unpreconditioned resid norm 4.654261998874e-06 true resid norm > 4.654261998975e-06 ||r(i)||/||b|| 5.391793349419e-07 > 8 KSP unpreconditioned resid norm 3.370057395937e-06 true resid norm > 3.370057396119e-06 ||r(i)||/||b|| 3.904088996184e-07 > 9 KSP unpreconditioned resid norm 2.540901076284e-06 true resid norm > 2.540901076192e-06 ||r(i)||/||b|| 2.943541538306e-07 > 10 KSP unpreconditioned resid norm 2.039263657470e-06 true resid norm > 2.039263657746e-06 ||r(i)||/||b|| 2.362412822907e-07 > 11 KSP unpreconditioned resid norm 1.583890355441e-06 true resid norm > 1.583890355170e-06 ||r(i)||/||b|| 1.834879404102e-07 > 12 KSP unpreconditioned resid norm 1.184174171199e-06 true resid norm > 1.184174170742e-06 ||r(i)||/||b|| 1.371822733607e-07 > 13 KSP unpreconditioned resid norm 8.493013125005e-07 true resid norm > 8.493013119686e-07 ||r(i)||/||b|| 9.838846989132e-08 > 14 KSP unpreconditioned resid norm 5.905267399341e-07 true resid norm > 5.905267385984e-07 ||r(i)||/||b|| 6.841037617843e-08 > 15 KSP unpreconditioned resid norm 4.160983692482e-07 true resid norm > 4.160983680083e-07 ||r(i)||/||b|| 4.820348347010e-08 > 16 KSP unpreconditioned resid norm 2.929006184135e-07 true resid norm > 2.929006164636e-07 ||r(i)||/||b|| 3.393147176151e-08 > 17 KSP unpreconditioned resid norm 2.085222908044e-07 true resid norm > 2.085222901282e-07 ||r(i)||/||b|| 2.415654935984e-08 > > > Switching to FBCGSR however yielded the following results > 0 KSP unpreconditioned resid norm 8.632122370749e+00 true resid norm > 8.632122370749e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 6.155300069200e-05 true resid norm > 4.783918927770e+05 ||r(i)||/||b|| 5.541996188539e+04 > 2 KSP unpreconditioned resid norm 5.203978464579e-05 true resid norm > 4.069231018620e+05 ||r(i)||/||b|| 4.714056223773e+04 > 3 KSP unpreconditioned resid norm 4.124211335014e-05 true resid norm > 8.257465019947e+05 ||r(i)||/||b|| 9.565973077408e+04 > 4 KSP unpreconditioned resid norm 2.735866558072e-05 true resid norm > 4.745745445186e+05 ||r(i)||/||b|| 5.497773596523e+04 > 5 KSP unpreconditioned resid norm 1.234049073102e-05 true resid norm > 3.694211077995e+05 ||r(i)||/||b|| 4.279609254050e+04 > 6 KSP unpreconditioned resid norm 7.813287928316e-06 true resid norm > 3.595303092238e+05 ||r(i)||/||b|| 4.165027947728e+04 > 7 KSP unpreconditioned resid norm 3.221541950651e-06 true resid norm > 3.179392486707e+05 ||r(i)||/||b|| 3.683210629034e+04 > 8 KSP unpreconditioned resid norm 1.777674826629e-06 true resid norm > 3.155842168872e+05 ||r(i)||/||b|| 3.655928441846e+04 > 9 KSP unpreconditioned resid norm 1.698816021362e-06 true resid norm > 3.145809168718e+05 ||r(i)||/||b|| 3.644305575854e+04 > 10 KSP unpreconditioned resid norm 1.577351929557e-06 true resid norm > 3.259758397382e+05 ||r(i)||/||b|| 3.776311615354e+04 > 11 KSP unpreconditioned resid norm 1.613962504892e-06 true resid norm > 3.186021802487e+05 ||r(i)||/||b|| 3.690890450398e+04 > 12 KSP unpreconditioned resid norm 4.564619465810e-07 true resid norm > 3.142082512956e+05 ||r(i)||/||b|| 3.639988380613e+04 > 13 KSP unpreconditioned resid norm 3.413352474598e-07 true resid norm > 3.136609638739e+05 ||r(i)||/||b|| 3.633648254765e+04 > 14 KSP unpreconditioned resid norm 2.914787162382e-07 true resid norm > 3.133747605150e+05 ||r(i)||/||b|| 3.630332692884e+04 > 15 KSP unpreconditioned resid norm 3.607032335094e-07 true resid norm > 3.108398102641e+05 ||r(i)||/||b|| 3.600966215648e+04 > 16 KSP unpreconditioned resid norm 7.653443566257e-08 true resid norm > 3.124857549918e+05 ||r(i)||/||b|| 3.620033887039e+04 > > > So it seems to me something weird is going on here. Could you help me? > > Kind regards > Elias > > > From d019598 at polito.it Wed Apr 6 04:08:24 2016 From: d019598 at polito.it (FRANCAVILLA MATTEO ALESSANDRO) Date: Wed, 06 Apr 2016 11:08:24 +0200 Subject: [petsc-users] How to extract array portions Message-ID: Hi, I'm trying to set a linear problem with a 2x2 block matrix (say A=[A11 A12; A21 A22] in matlab notation). Obviously I could generate a single matrix, but I would like to keep it as a 2x2 block matrix with the 4 blocks generated and used independently (e.g., because at some point I need to change some of those blocks to matrix-free). My plan was then to generate a MatShell and set a MATOP_MULT shell operation to explicitly compute y=A*x by splitting x=[x1;x2] and y=[y1;y2] and carrying out the single matrix-vector multiplications, and reassembling the final result. My problem is that my efforts in extracting x1 and x2 from the original x vector have failed. The following Fortran code is what seemed to me the most reasonable thing to do to set the matrix-vector multiplication, but it does not work !====================================================== subroutine MyMult(A,x,y,IERR) implicit none #include #include #include #include Mat A Vec x,y, x1, x2, y1, y2 IS :: ix1, ix2, iy1, iy2 PetscErrorCode ierr PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures PetscInt, ALLOCATABLE :: INDICES(:) PetscMPIInt :: RANK CALL VecGetSize(x,N,IERR) NumberOfUnknowns = N / 2 CALL VecGetSize(y,N,IERR) NumberOfMeasures = N - NumberOfUnknowns N = NumberOfUnknowns + MAX(NumberOfUnknowns,NumberOfMeasures) ALLOCATE(INDICES(N)) INDICES = (/ (ii, ii=0, N-1) /) CALL ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) CALL VecGetSubVector(x,ix1,x1,IERR) CALL ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) CALL VecGetSubVector(x,ix2,x2,IERR) CALL ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) CALL VecGetSubVector(y,iy1,y1,IERR) CALL ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) CALL VecGetSubVector(y,iy2,y2,IERR) CALL MatMult(L_discretized,x1,y1,IERR) CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) CALL MatMult(L_MR,x1,y2,IERR) CALL MatMultAdd(K_MR,x2,y2,y2,IERR) CALL VecRestoreSubVector(y,iy1,y1,IERR) CALL VecRestoreSubVector(y,iy2,y2,IERR) return end subroutine MyMult !====================================================== Obviously the sequence of calls to ISCreateGeneral and VecGetSubVector does not do what I expect, as the errors I'm getting are in the following MatMult multiplications (the global sizes of x1 and x2 SHOULD BE both 771, while y1 and y2 should be 771 and 2286): 1) executed with 1 MPI process: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #1 MatMult() line 2215 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #2 MatMultAdd() line 2396 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c 2) executed with 4 MPI processes (I'm copying the output of process 0 only, they're all the same): .... [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #1 MatMult() line 2214 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #2 MatMultAdd() line 2393 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #3 MatMult() line 2214 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [0]PETSC ERROR: #4 MatMultAdd() line 2393 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c .... Any idea? I would like to keep the flexibility of having a sequence of mat-vec multiplications as in CALL MatMult(L_discretized,x1,y1,IERR) CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) CALL MatMult(L_MR,x1,y2,IERR) CALL MatMultAdd(K_MR,x2,y2,y2,IERR) but for doing that I need to be able to extract x1 and x2, and to reassemble the partial results into y. Thanks in advance, Alessandro From dave.mayhem23 at gmail.com Wed Apr 6 04:16:43 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 6 Apr 2016 11:16:43 +0200 Subject: [petsc-users] How to extract array portions In-Reply-To: References: Message-ID: On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO wrote: > Hi, > > I'm trying to set a linear problem with a 2x2 block matrix (say A=[A11 > A12; A21 A22] in matlab notation). Obviously I could generate a single > matrix, but I would like to keep it as a 2x2 block matrix with the 4 blocks > generated and used independently (e.g., because at some point I need to > change some of those blocks to matrix-free). My plan was then to generate a > MatShell and set a MATOP_MULT shell operation to explicitly compute y=A*x > by splitting x=[x1;x2] and y=[y1;y2] and carrying out the single > matrix-vector multiplications, and reassembling the final result. Take a look at MatNest - it does exactly what you want. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST Thanks, Dave > My problem is that my efforts in extracting x1 and x2 from the original x > vector have failed. The following Fortran code is what seemed to me the > most reasonable thing to do to set the matrix-vector multiplication, but it > does not work > > > !====================================================== > subroutine MyMult(A,x,y,IERR) > > implicit none > > #include > #include > #include > #include > > Mat A > Vec x,y, x1, x2, y1, y2 > IS :: ix1, ix2, iy1, iy2 > PetscErrorCode ierr > PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures > PetscInt, ALLOCATABLE :: INDICES(:) > PetscMPIInt :: RANK > > CALL VecGetSize(x,N,IERR) > NumberOfUnknowns = N / 2 > CALL VecGetSize(y,N,IERR) > NumberOfMeasures = N - NumberOfUnknowns > > N = NumberOfUnknowns + MAX(NumberOfUnknowns,NumberOfMeasures) > ALLOCATE(INDICES(N)) > INDICES = (/ (ii, ii=0, N-1) /) > > CALL > ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) > CALL VecGetSubVector(x,ix1,x1,IERR) > CALL > ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) > CALL VecGetSubVector(x,ix2,x2,IERR) > CALL > ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) > CALL VecGetSubVector(y,iy1,y1,IERR) > CALL > ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) > CALL VecGetSubVector(y,iy2,y2,IERR) > > CALL MatMult(L_discretized,x1,y1,IERR) > CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) > CALL MatMult(L_MR,x1,y2,IERR) > CALL MatMultAdd(K_MR,x2,y2,y2,IERR) > > CALL VecRestoreSubVector(y,iy1,y1,IERR) > CALL VecRestoreSubVector(y,iy2,y2,IERR) > > return > end subroutine MyMult > !====================================================== > > > Obviously the sequence of calls to ISCreateGeneral and VecGetSubVector > does not do what I expect, as the errors I'm getting are in the following > MatMult multiplications (the global sizes of x1 and x2 SHOULD BE both 771, > while y1 and y2 should be 771 and 2286): > > 1) executed with 1 MPI process: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #1 MatMult() line 2215 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #2 MatMultAdd() line 2396 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > > > > 2) executed with 4 MPI processes (I'm copying the output of process 0 > only, they're all the same): > > .... > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #1 MatMult() line 2214 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #2 MatMultAdd() line 2393 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #3 MatMult() line 2214 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named > alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [0]PETSC ERROR: #4 MatMultAdd() line 2393 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > .... > > > Any idea? I would like to keep the flexibility of having a sequence of > mat-vec multiplications as in > > CALL MatMult(L_discretized,x1,y1,IERR) > CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) > CALL MatMult(L_MR,x1,y2,IERR) > CALL MatMultAdd(K_MR,x2,y2,y2,IERR) > > but for doing that I need to be able to extract x1 and x2, and to > reassemble the partial results into y. > > Thanks in advance, > Alessandro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.eldred at gmail.com Wed Apr 6 06:01:57 2016 From: chris.eldred at gmail.com (Chris Eldred) Date: Wed, 6 Apr 2016 13:01:57 +0200 Subject: [petsc-users] Correct syntax for DMDA ownership ranges in petsc4py Message-ID: Hey All, Does anyone know the correct syntax for setting ownership ranges for DMDAs in petsc4py? For example, if I have a 1D DMDA with 20 vertices and 3 processes, and I want to distribute the vertices as [8,8,4], should the correct create call be: da = PETSc.DMDA().create(dim=1,dof=1,sizes=[20,],proc_sizes=[nprocs,],boundary_type=[PETSc.DM.BoundaryType.PERIODIC,], stencil_type= PETSc.DMDA.StencilType.BOX, stencil_width=1,ownership_ranges=[8,8,4]) or da = PETSc.DMDA().create(dim=1,dof=1,sizes=[20,],proc_sizes=[nprocs,],boundary_type=[PETSc.DM.BoundaryType.PERIODIC,], stencil_type= PETSc.DMDA.StencilType.BOX, stencil_width=1,ownership_ranges=[[8,8,4],]) For me, the 1st version fails with ValueError: number of dimensions 1 and number ownership ranges 3 (which I expected) but the 2nd version also fails with TypeError: Expected tuple, got list I have tried using tuples instead of lists and nothing changes. Thanks in advance for any assistance! Regards, -Chris -- Chris Eldred Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred at gmail.com From d019598 at polito.it Wed Apr 6 07:12:56 2016 From: d019598 at polito.it (FRANCAVILLA MATTEO ALESSANDRO) Date: Wed, 06 Apr 2016 14:12:56 +0200 Subject: [petsc-users] How to extract array portions In-Reply-To: References: Message-ID: Thanks Dave, that's exactly what I was looking for, and it (partially) solved my problem. The MatMult operation now works fine, unfortunately not for every number of MPI processes (while the MatMult involving each submatrix works fine for any number of processes): .... matArray(1) = L_discretized matArray(2) = K_discretized matArray(3) = L_MR matArray(4) = K_MR CALL MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,TheMatrix,IERR) CALL MatSetFromOptions(TheMatrix,IERR) ! CALL VecCreate(PETSC_COMM_WORLD,x,IERR) CALL VecSetSizes(x,PETSC_DECIDE,2*NumberOfUnknowns,IERR) CALL VecSetFromOptions(x,IERR) CALL VecSet(x,0.+0.*PETSC_i,IERR) CALL VecSetValues(x,1,0,1.+0.*PETSC_i,INSERT_VALUES,IERR) CALL VecAssemblyBegin(x,IERR) CALL VecAssemblyEnd(x,IERR) CALL VecCreate(PETSC_COMM_WORLD,y,IERR) CALL VecSetSizes(y,PETSC_DECIDE,NumberOfUnknowns+NumberOfMeasures,IERR) CALL VecSetFromOptions(y,IERR) CALL MatMult(TheMatrix,x,y,IERR) .... I guess there is something wrong with the distribution of vectors between the MPI processes. With the specific sizes of my problem everything works fine with 1-2-3-6 processes, otherwise I get "Nonconforming object sizes" error, e.g. with 8 processes: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Nonconforming object sizes [2]PETSC ERROR: Mat mat,Vec y: local dim 383 382 [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [2]PETSC ERROR: Nonconforming object sizes ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 13:31:45 2016 [2]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec --with-scalar-type=complex [2]PETSC ERROR: #1 MatMult() line 2216 in /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c ... [6]PETSC ERROR: Mat mat,Vec y: local dim 381 382 ... [7]PETSC ERROR: Mat mat,Vec y: local dim 381 382 ... [1]PETSC ERROR: Mat mat,Vec y: local dim 383 382 ... I noticed that the code works (runs and provides correct results) when the number of processes divides the length of y2 (number of rows of A21 and A22); however when I apply MatMult separately on A21 and A22 everything works fine with any number of processes Any idea where I'm messing up here? Thanks, Alessandro On Wed, 6 Apr 2016 11:16:43 +0200 Dave May wrote: > On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO > > wrote: > >> Hi, >> >> I'm trying to set a linear problem with a 2x2 block >>matrix (say A=[A11 >> A12; A21 A22] in matlab notation). Obviously I could >>generate a single >> matrix, but I would like to keep it as a 2x2 block >>matrix with the 4 blocks >> generated and used independently (e.g., because at some >>point I need to >> change some of those blocks to matrix-free). My plan was >>then to generate a >> MatShell and set a MATOP_MULT shell operation to >>explicitly compute y=A*x >> by splitting x=[x1;x2] and y=[y1;y2] and carrying out >>the single >> matrix-vector multiplications, and reassembling the >>final result. > > > Take a look at MatNest - it does exactly what you want. > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST > > Thanks, > Dave > > >> My problem is that my efforts in extracting x1 and x2 >>from the original x >> vector have failed. The following Fortran code is what >>seemed to me the >> most reasonable thing to do to set the matrix-vector >>multiplication, but it >> does not work >> >> >> !====================================================== >> subroutine MyMult(A,x,y,IERR) >> >> implicit none >> >> #include >> #include >> #include >> #include >> >> Mat A >> Vec x,y, x1, x2, y1, y2 >> IS :: ix1, ix2, iy1, iy2 >> PetscErrorCode ierr >> PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures >> PetscInt, ALLOCATABLE :: INDICES(:) >> PetscMPIInt :: RANK >> >> CALL VecGetSize(x,N,IERR) >> NumberOfUnknowns = N / 2 >> CALL VecGetSize(y,N,IERR) >> NumberOfMeasures = N - NumberOfUnknowns >> >> N = NumberOfUnknowns + >>MAX(NumberOfUnknowns,NumberOfMeasures) >> ALLOCATE(INDICES(N)) >> INDICES = (/ (ii, ii=0, N-1) /) >> >> CALL >> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) >> CALL VecGetSubVector(x,ix1,x1,IERR) >> CALL >> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) >> CALL VecGetSubVector(x,ix2,x2,IERR) >> CALL >> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) >> CALL VecGetSubVector(y,iy1,y1,IERR) >> CALL >> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) >> CALL VecGetSubVector(y,iy2,y2,IERR) >> >> CALL MatMult(L_discretized,x1,y1,IERR) >> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >> CALL MatMult(L_MR,x1,y2,IERR) >> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >> >> CALL VecRestoreSubVector(y,iy1,y1,IERR) >> CALL VecRestoreSubVector(y,iy2,y2,IERR) >> >> return >> end subroutine MyMult >> !====================================================== >> >> >> Obviously the sequence of calls to ISCreateGeneral and >>VecGetSubVector >> does not do what I expect, as the errors I'm getting are >>in the following >> MatMult multiplications (the global sizes of x1 and x2 >>SHOULD BE both 771, >> while y1 and y2 should be 771 and 2286): >> >> 1) executed with 1 MPI process: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:54:03 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #1 MatMult() line 2215 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:54:03 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #2 MatMultAdd() line 2396 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> >> >> >> 2) executed with 4 MPI processes (I'm copying the output >>of process 0 >> only, they're all the same): >> >> .... >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:49:13 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #1 MatMult() line 2214 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:49:13 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #2 MatMultAdd() line 2393 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:49:13 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #3 MatMult() line 2214 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >> [0]PETSC ERROR: See >>http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>10:49:13 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [0]PETSC ERROR: #4 MatMultAdd() line 2393 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> .... >> >> >> Any idea? I would like to keep the flexibility of having >>a sequence of >> mat-vec multiplications as in >> >> CALL MatMult(L_discretized,x1,y1,IERR) >> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >> CALL MatMult(L_MR,x1,y2,IERR) >> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >> >> but for doing that I need to be able to extract x1 and >>x2, and to >> reassemble the partial results into y. >> >> Thanks in advance, >> Alessandro >> ___________________________________ Matteo Alessandro Francavilla, PhD Antenna and EMC Lab (LACE) Istituto Superiore Mario Boella (ISMB) Torino, Italy phone +39 011 2276 711 From dave.mayhem23 at gmail.com Wed Apr 6 07:38:12 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 6 Apr 2016 14:38:12 +0200 Subject: [petsc-users] How to extract array portions In-Reply-To: References: Message-ID: On 6 April 2016 at 14:12, FRANCAVILLA MATTEO ALESSANDRO wrote: > Thanks Dave, that's exactly what I was looking for, and it (partially) > solved my problem. > > The MatMult operation now works fine, unfortunately not for every number > of MPI processes (while the MatMult involving each submatrix works fine for > any number of processes): > > .... > matArray(1) = L_discretized > matArray(2) = K_discretized > matArray(3) = L_MR > matArray(4) = K_MR > CALL > MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,TheMatrix,IERR) > CALL MatSetFromOptions(TheMatrix,IERR) > ! > CALL VecCreate(PETSC_COMM_WORLD,x,IERR) > CALL VecSetSizes(x,PETSC_DECIDE,2*NumberOfUnknowns,IERR) > CALL VecSetFromOptions(x,IERR) > CALL VecSet(x,0.+0.*PETSC_i,IERR) > CALL VecSetValues(x,1,0,1.+0.*PETSC_i,INSERT_VALUES,IERR) > CALL VecAssemblyBegin(x,IERR) > CALL VecAssemblyEnd(x,IERR) > CALL VecCreate(PETSC_COMM_WORLD,y,IERR) > CALL VecSetSizes(y,PETSC_DECIDE,NumberOfUnknowns+NumberOfMeasures,IERR) > CALL VecSetFromOptions(y,IERR) > CALL MatMult(TheMatrix,x,y,IERR) > .... > > > I guess there is something wrong with the distribution of vectors between > the MPI processes. With the specific sizes of my problem everything works > fine with 1-2-3-6 processes, otherwise I get "Nonconforming object sizes" > error, e.g. with 8 processes: > > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Nonconforming object sizes > [2]PETSC ERROR: Mat mat,Vec y: local dim 383 382 > [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [2]PETSC ERROR: Nonconforming object sizes > ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by > alessandro Wed Apr 6 13:31:45 2016 > [2]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl > --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc > --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 > --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec > --with-scalar-type=complex > [2]PETSC ERROR: #1 MatMult() line 2216 in > /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c > ... > [6]PETSC ERROR: Mat mat,Vec y: local dim 381 382 > ... > [7]PETSC ERROR: Mat mat,Vec y: local dim 381 382 > ... > [1]PETSC ERROR: Mat mat,Vec y: local dim 383 382 > ... > > > I noticed that the code works (runs and provides correct results) when the > number of processes divides the length of y2 (number of rows of A21 and > A22); however when I apply MatMult separately on A21 and A22 everything > works fine with any number of processes > > Any idea where I'm messing up here? > Sounds like the large vector you are using to multiple with the matnest object wasn't created via MatCreateVecs(). If you use this method you will obtain a vector with the correct local and global sizes. > > > Thanks, > Alessandro > > > > > On Wed, 6 Apr 2016 11:16:43 +0200 > Dave May wrote: > >> On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO < >> d019598 at polito.it> >> wrote: >> >> Hi, >>> >>> I'm trying to set a linear problem with a 2x2 block matrix (say A=[A11 >>> A12; A21 A22] in matlab notation). Obviously I could generate a single >>> matrix, but I would like to keep it as a 2x2 block matrix with the 4 >>> blocks >>> generated and used independently (e.g., because at some point I need to >>> change some of those blocks to matrix-free). My plan was then to >>> generate a >>> MatShell and set a MATOP_MULT shell operation to explicitly compute y=A*x >>> by splitting x=[x1;x2] and y=[y1;y2] and carrying out the single >>> matrix-vector multiplications, and reassembling the final result. >>> >> >> >> Take a look at MatNest - it does exactly what you want. >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST >> >> Thanks, >> Dave >> >> >> My problem is that my efforts in extracting x1 and x2 from the original x >>> vector have failed. The following Fortran code is what seemed to me the >>> most reasonable thing to do to set the matrix-vector multiplication, but >>> it >>> does not work >>> >>> >>> !====================================================== >>> subroutine MyMult(A,x,y,IERR) >>> >>> implicit none >>> >>> #include >>> #include >>> #include >>> #include >>> >>> Mat A >>> Vec x,y, x1, x2, y1, y2 >>> IS :: ix1, ix2, iy1, iy2 >>> PetscErrorCode ierr >>> PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures >>> PetscInt, ALLOCATABLE :: INDICES(:) >>> PetscMPIInt :: RANK >>> >>> CALL VecGetSize(x,N,IERR) >>> NumberOfUnknowns = N / 2 >>> CALL VecGetSize(y,N,IERR) >>> NumberOfMeasures = N - NumberOfUnknowns >>> >>> N = NumberOfUnknowns + MAX(NumberOfUnknowns,NumberOfMeasures) >>> ALLOCATE(INDICES(N)) >>> INDICES = (/ (ii, ii=0, N-1) /) >>> >>> CALL >>> >>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) >>> CALL VecGetSubVector(x,ix1,x1,IERR) >>> CALL >>> >>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) >>> CALL VecGetSubVector(x,ix2,x2,IERR) >>> CALL >>> >>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) >>> CALL VecGetSubVector(y,iy1,y1,IERR) >>> CALL >>> >>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) >>> CALL VecGetSubVector(y,iy2,y2,IERR) >>> >>> CALL MatMult(L_discretized,x1,y1,IERR) >>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>> CALL MatMult(L_MR,x1,y2,IERR) >>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>> >>> CALL VecRestoreSubVector(y,iy1,y1,IERR) >>> CALL VecRestoreSubVector(y,iy2,y2,IERR) >>> >>> return >>> end subroutine MyMult >>> !====================================================== >>> >>> >>> Obviously the sequence of calls to ISCreateGeneral and VecGetSubVector >>> does not do what I expect, as the errors I'm getting are in the following >>> MatMult multiplications (the global sizes of x1 and x2 SHOULD BE both >>> 771, >>> while y1 and y2 should be 771 and 2286): >>> >>> 1) executed with 1 MPI process: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #1 MatMult() line 2215 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #2 MatMultAdd() line 2396 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> >>> >>> >>> 2) executed with 4 MPI processes (I'm copying the output of process 0 >>> only, they're all the same): >>> >>> .... >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #1 MatMult() line 2214 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #2 MatMultAdd() line 2393 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #3 MatMult() line 2214 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>> [0]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [0]PETSC ERROR: #4 MatMultAdd() line 2393 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> .... >>> >>> >>> Any idea? I would like to keep the flexibility of having a sequence of >>> mat-vec multiplications as in >>> >>> CALL MatMult(L_discretized,x1,y1,IERR) >>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>> CALL MatMult(L_MR,x1,y2,IERR) >>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>> >>> but for doing that I need to be able to extract x1 and x2, and to >>> reassemble the partial results into y. >>> >>> Thanks in advance, >>> Alessandro >>> >>> > ___________________________________ > Matteo Alessandro Francavilla, PhD > Antenna and EMC Lab (LACE) > Istituto Superiore Mario Boella (ISMB) > Torino, Italy > phone +39 011 2276 711 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chevret at lpccaen.in2p3.fr Wed Apr 6 07:43:48 2016 From: chevret at lpccaen.in2p3.fr (Thibault Chevret) Date: Wed, 6 Apr 2016 14:43:48 +0200 (CEST) Subject: [petsc-users] SLEPc eigenpairs relative error Message-ID: <609549062.20066.1459946628298.JavaMail.zimbra@lpccaen.in2p3.fr> Dear all, I am interested in computing the eigenpairs of a sparse real non-symmetric matrix (~8000x8000). I use SLEPc for this purpose, specifying only nev, ncv and tol as options. My problem is, I guess, that I would like to compute the whole spectra, and I've got some issues regarding the errors on some eigenvalues (ev). Real ones are OK, their relative errors being below 1e-10, but for some complex ones I can't get below 1e-1. This is greatly improved if I specify -eps_larger_imaginary, but then the problem is inversed as the errors associated to real ev become worse. My solution, for now, is to merge results obtained for complex ev using a resolution with -eps_larger_imaginary, and those obtained for real ev without this option. This is quite tedious, as I have to resolve my matrix twice, but it works fine. Am I doing something wrong ? I've read on another post that SLEPc should not be used for the computation of the whole spectra, but is there by any chance any way to improve the results using only one calculation ? Thank you for your time. Thibault From jroman at dsic.upv.es Wed Apr 6 08:33:53 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 6 Apr 2016 15:33:53 +0200 Subject: [petsc-users] SLEPc eigenpairs relative error In-Reply-To: <609549062.20066.1459946628298.JavaMail.zimbra@lpccaen.in2p3.fr> References: <609549062.20066.1459946628298.JavaMail.zimbra@lpccaen.in2p3.fr> Message-ID: <3E2D1F57-FBC5-43E6-BD89-0C55F81ABF9C@dsic.upv.es> > El 6 abr 2016, a las 14:43, Thibault Chevret escribi?: > > Dear all, > > I am interested in computing the eigenpairs of a sparse real non-symmetric matrix (~8000x8000). I use SLEPc for this purpose, specifying only nev, ncv and tol as options. > > My problem is, I guess, that I would like to compute the whole spectra, and I've got some issues regarding the errors on some eigenvalues (ev). > Real ones are OK, their relative errors being below 1e-10, but for some complex ones I can't get below 1e-1. > > This is greatly improved if I specify -eps_larger_imaginary, but then the problem is inversed as the errors associated to real ev become worse. > My solution, for now, is to merge results obtained for complex ev using a resolution with -eps_larger_imaginary, and those obtained for real ev without this option. This is quite tedious, as I have to resolve my matrix twice, but it works fine. > > Am I doing something wrong ? I've read on another post that SLEPc should not be used for the computation of the whole spectra, but is there by any chance any way to improve the results using only one calculation ? > > Thank you for your time. > > Thibault SLEPc is not intended for computing the whole spectrum. For this you can use LAPACK (or ScaLAPACK) treating the matrix as dense. LAPACK can also provide information about conditioning, which may be causing large residuals. In SLEPc, you can try setting a value for the mpd parameter (e.g. -eps_mpd 400). See explanation in section 2.6.5 of the users manual. But again SLEPc is designed for a subset of the spectrum. Jose From d019598 at polito.it Wed Apr 6 09:20:36 2016 From: d019598 at polito.it (FRANCAVILLA MATTEO ALESSANDRO) Date: Wed, 06 Apr 2016 16:20:36 +0200 Subject: [petsc-users] How to extract array portions In-Reply-To: References: Message-ID: Fantastic, it definitely solved my problem! Thank you! Should I always create the vectors with MatCreateVecs when I use them for matrix-vector multiplications? I used to create vectors with VecCreate, and it has always worked fine with "standard" matrices (matrices not created with MATNEST structure). In the example at http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html I read the following: "PETSc automatically generates appropriately partitioned matrices and vectors when MatCreate() and VecCreate() are used with the same communicator." It is not clear to me when I can simply use VecCreate (as I've always done so far), and when to use MatCreateVecs... On Wed, 6 Apr 2016 14:38:12 +0200 Dave May wrote: > On 6 April 2016 at 14:12, FRANCAVILLA MATTEO ALESSANDRO > > wrote: > >> Thanks Dave, that's exactly what I was looking for, and >>it (partially) >> solved my problem. >> >> The MatMult operation now works fine, unfortunately not >>for every number >> of MPI processes (while the MatMult involving each >>submatrix works fine for >> any number of processes): >> >> .... >> matArray(1) = L_discretized >> matArray(2) = K_discretized >> matArray(3) = L_MR >> matArray(4) = K_MR >> CALL >> MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,TheMatrix,IERR) >> CALL MatSetFromOptions(TheMatrix,IERR) >> ! >> CALL VecCreate(PETSC_COMM_WORLD,x,IERR) >> CALL VecSetSizes(x,PETSC_DECIDE,2*NumberOfUnknowns,IERR) >> CALL VecSetFromOptions(x,IERR) >> CALL VecSet(x,0.+0.*PETSC_i,IERR) >> CALL >>VecSetValues(x,1,0,1.+0.*PETSC_i,INSERT_VALUES,IERR) >> CALL VecAssemblyBegin(x,IERR) >> CALL VecAssemblyEnd(x,IERR) >> CALL VecCreate(PETSC_COMM_WORLD,y,IERR) >> CALL >>VecSetSizes(y,PETSC_DECIDE,NumberOfUnknowns+NumberOfMeasures,IERR) >> CALL VecSetFromOptions(y,IERR) >> CALL MatMult(TheMatrix,x,y,IERR) >> .... >> >> >> I guess there is something wrong with the distribution >>of vectors between >> the MPI processes. With the specific sizes of my problem >>everything works >> fine with 1-2-3-6 processes, otherwise I get >>"Nonconforming object sizes" >> error, e.g. with 8 processes: >> >> [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Nonconforming object sizes >> [2]PETSC ERROR: Mat mat,Vec y: local dim 383 382 >> [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>2015 >> [2]PETSC ERROR: Nonconforming object sizes >> ./ISS on a arch-linux2-c-debug named >>alessandro-HP-ProDesk-490-G2-MT by >> alessandro Wed Apr 6 13:31:45 2016 >> [2]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >> --with-scalar-type=complex >> [2]PETSC ERROR: #1 MatMult() line 2216 in >> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >> ... >> [6]PETSC ERROR: Mat mat,Vec y: local dim 381 382 >> ... >> [7]PETSC ERROR: Mat mat,Vec y: local dim 381 382 >> ... >> [1]PETSC ERROR: Mat mat,Vec y: local dim 383 382 >> ... >> >> >> I noticed that the code works (runs and provides correct >>results) when the >> number of processes divides the length of y2 (number of >>rows of A21 and >> A22); however when I apply MatMult separately on A21 and >>A22 everything >> works fine with any number of processes >> >> Any idea where I'm messing up here? >> > > Sounds like the large vector you are using to multiple >with the matnest > object wasn't created via MatCreateVecs(). If you use >this method you will > obtain a vector with the correct local and global sizes. > > > > >> >> >> Thanks, >> Alessandro >> >> >> >> >> On Wed, 6 Apr 2016 11:16:43 +0200 >> Dave May wrote: >> >>> On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO >>>< >>> d019598 at polito.it> >>> wrote: >>> >>> Hi, >>>> >>>> I'm trying to set a linear problem with a 2x2 block >>>>matrix (say A=[A11 >>>> A12; A21 A22] in matlab notation). Obviously I could >>>>generate a single >>>> matrix, but I would like to keep it as a 2x2 block >>>>matrix with the 4 >>>> blocks >>>> generated and used independently (e.g., because at some >>>>point I need to >>>> change some of those blocks to matrix-free). My plan was >>>>then to >>>> generate a >>>> MatShell and set a MATOP_MULT shell operation to >>>>explicitly compute y=A*x >>>> by splitting x=[x1;x2] and y=[y1;y2] and carrying out >>>>the single >>>> matrix-vector multiplications, and reassembling the >>>>final result. >>>> >>> >>> >>> Take a look at MatNest - it does exactly what you want. >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST >>> >>> Thanks, >>> Dave >>> >>> >>> My problem is that my efforts in extracting x1 and x2 >>>from the original x >>>> vector have failed. The following Fortran code is what >>>>seemed to me the >>>> most reasonable thing to do to set the matrix-vector >>>>multiplication, but >>>> it >>>> does not work >>>> >>>> >>>> !====================================================== >>>> subroutine MyMult(A,x,y,IERR) >>>> >>>> implicit none >>>> >>>> #include >>>> #include >>>> #include >>>> #include >>>> >>>> Mat A >>>> Vec x,y, x1, x2, y1, y2 >>>> IS :: ix1, ix2, iy1, iy2 >>>> PetscErrorCode ierr >>>> PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures >>>> PetscInt, ALLOCATABLE :: INDICES(:) >>>> PetscMPIInt :: RANK >>>> >>>> CALL VecGetSize(x,N,IERR) >>>> NumberOfUnknowns = N / 2 >>>> CALL VecGetSize(y,N,IERR) >>>> NumberOfMeasures = N - NumberOfUnknowns >>>> >>>> N = NumberOfUnknowns + >>>>MAX(NumberOfUnknowns,NumberOfMeasures) >>>> ALLOCATE(INDICES(N)) >>>> INDICES = (/ (ii, ii=0, N-1) /) >>>> >>>> CALL >>>> >>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) >>>> CALL VecGetSubVector(x,ix1,x1,IERR) >>>> CALL >>>> >>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) >>>> CALL VecGetSubVector(x,ix2,x2,IERR) >>>> CALL >>>> >>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) >>>> CALL VecGetSubVector(y,iy1,y1,IERR) >>>> CALL >>>> >>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) >>>> CALL VecGetSubVector(y,iy2,y2,IERR) >>>> >>>> CALL MatMult(L_discretized,x1,y1,IERR) >>>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>>> CALL MatMult(L_MR,x1,y2,IERR) >>>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>>> >>>> CALL VecRestoreSubVector(y,iy1,y1,IERR) >>>> CALL VecRestoreSubVector(y,iy2,y2,IERR) >>>> >>>> return >>>> end subroutine MyMult >>>> !====================================================== >>>> >>>> >>>> Obviously the sequence of calls to ISCreateGeneral and >>>>VecGetSubVector >>>> does not do what I expect, as the errors I'm getting are >>>>in the following >>>> MatMult multiplications (the global sizes of x1 and x2 >>>>SHOULD BE both >>>> 771, >>>> while y1 and y2 should be 771 and 2286): >>>> >>>> 1) executed with 1 MPI process: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:54:03 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #1 MatMult() line 2215 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:54:03 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #2 MatMultAdd() line 2396 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> >>>> >>>> >>>> 2) executed with 4 MPI processes (I'm copying the output >>>>of process 0 >>>> only, they're all the same): >>>> >>>> .... >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:49:13 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #1 MatMult() line 2214 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:49:13 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #2 MatMultAdd() line 2393 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:49:13 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #3 MatMult() line 2214 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>>> [0]PETSC ERROR: See >>>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, >>>>2015 >>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 >>>>10:49:13 2016 >>>> [0]PETSC ERROR: Configure options >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>> --with-scalar-type=complex >>>> [0]PETSC ERROR: #4 MatMultAdd() line 2393 in >>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>> .... >>>> >>>> >>>> Any idea? I would like to keep the flexibility of having >>>>a sequence of >>>> mat-vec multiplications as in >>>> >>>> CALL MatMult(L_discretized,x1,y1,IERR) >>>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>>> CALL MatMult(L_MR,x1,y2,IERR) >>>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>>> >>>> but for doing that I need to be able to extract x1 and >>>>x2, and to >>>> reassemble the partial results into y. >>>> >>>> Thanks in advance, >>>> Alessandro >>>> >>>> >> ___________________________________ >> Matteo Alessandro Francavilla, PhD >> Antenna and EMC Lab (LACE) >> Istituto Superiore Mario Boella (ISMB) >> Torino, Italy >> phone +39 011 2276 711 >> ___________________________________ Matteo Alessandro Francavilla, PhD Antenna and EMC Lab (LACE) Istituto Superiore Mario Boella (ISMB) Torino, Italy phone +39 011 2276 711 From dave.mayhem23 at gmail.com Wed Apr 6 09:27:55 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 6 Apr 2016 16:27:55 +0200 Subject: [petsc-users] How to extract array portions In-Reply-To: References: Message-ID: On 6 April 2016 at 16:20, FRANCAVILLA MATTEO ALESSANDRO wrote: > Fantastic, it definitely solved my problem! Thank you! > > Should I always create the vectors with MatCreateVecs when I use them for > matrix-vector multiplications? I used to create vectors with VecCreate, and > it has always worked fine with "standard" matrices (matrices not created > with MATNEST structure). > Yes > > In the example at > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html > I read the following: > > "PETSc automatically generates appropriately partitioned matrices and > vectors when MatCreate() and VecCreate() are used with the same > communicator." > This comment also assumes that you use PETSC_DECIDE for the local sizes. So if you create a matrix with global size M x M and a vector of global size M and you (i) use PETSC_DECIDE for the local size for the row/col with MatCreate() and PETSC_DECIDE for the local row for VecCreate(); and (ii) you use the same communicator - then the row partitioning adopted by the matrix will be consistent with the row partitioning for the vector. > It is not clear to me when I can simply use VecCreate (as I've always done > so far), and when to use MatCreateVecs... > You should always use MatCreateVecs(). The implementation of Mat will give you back a consistent vector. Thanks, Dave > > > > On Wed, 6 Apr 2016 14:38:12 +0200 > > Dave May wrote: > >> On 6 April 2016 at 14:12, FRANCAVILLA MATTEO ALESSANDRO < >> d019598 at polito.it> >> wrote: >> >> Thanks Dave, that's exactly what I was looking for, and it (partially) >>> solved my problem. >>> >>> The MatMult operation now works fine, unfortunately not for every number >>> of MPI processes (while the MatMult involving each submatrix works fine >>> for >>> any number of processes): >>> >>> .... >>> matArray(1) = L_discretized >>> matArray(2) = K_discretized >>> matArray(3) = L_MR >>> matArray(4) = K_MR >>> CALL >>> >>> MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,TheMatrix,IERR) >>> CALL MatSetFromOptions(TheMatrix,IERR) >>> ! >>> CALL VecCreate(PETSC_COMM_WORLD,x,IERR) >>> CALL VecSetSizes(x,PETSC_DECIDE,2*NumberOfUnknowns,IERR) >>> CALL VecSetFromOptions(x,IERR) >>> CALL VecSet(x,0.+0.*PETSC_i,IERR) >>> CALL VecSetValues(x,1,0,1.+0.*PETSC_i,INSERT_VALUES,IERR) >>> CALL VecAssemblyBegin(x,IERR) >>> CALL VecAssemblyEnd(x,IERR) >>> CALL VecCreate(PETSC_COMM_WORLD,y,IERR) >>> CALL VecSetSizes(y,PETSC_DECIDE,NumberOfUnknowns+NumberOfMeasures,IERR) >>> CALL VecSetFromOptions(y,IERR) >>> CALL MatMult(TheMatrix,x,y,IERR) >>> .... >>> >>> >>> I guess there is something wrong with the distribution of vectors between >>> the MPI processes. With the specific sizes of my problem everything works >>> fine with 1-2-3-6 processes, otherwise I get "Nonconforming object sizes" >>> error, e.g. with 8 processes: >>> >>> [2]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [2]PETSC ERROR: Nonconforming object sizes >>> [2]PETSC ERROR: Mat mat,Vec y: local dim 383 382 >>> [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [2]PETSC ERROR: Nonconforming object sizes >>> ./ISS on a arch-linux2-c-debug named alessandro-HP-ProDesk-490-G2-MT by >>> alessandro Wed Apr 6 13:31:45 2016 >>> [2]PETSC ERROR: Configure options >>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>> --with-scalar-type=complex >>> [2]PETSC ERROR: #1 MatMult() line 2216 in >>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>> ... >>> [6]PETSC ERROR: Mat mat,Vec y: local dim 381 382 >>> ... >>> [7]PETSC ERROR: Mat mat,Vec y: local dim 381 382 >>> ... >>> [1]PETSC ERROR: Mat mat,Vec y: local dim 383 382 >>> ... >>> >>> >>> I noticed that the code works (runs and provides correct results) when >>> the >>> number of processes divides the length of y2 (number of rows of A21 and >>> A22); however when I apply MatMult separately on A21 and A22 everything >>> works fine with any number of processes >>> >>> Any idea where I'm messing up here? >>> >>> >> Sounds like the large vector you are using to multiple with the matnest >> object wasn't created via MatCreateVecs(). If you use this method you will >> obtain a vector with the correct local and global sizes. >> >> >> >> >> >>> >>> Thanks, >>> Alessandro >>> >>> >>> >>> >>> On Wed, 6 Apr 2016 11:16:43 +0200 >>> Dave May wrote: >>> >>> On 6 April 2016 at 11:08, FRANCAVILLA MATTEO ALESSANDRO < >>>> d019598 at polito.it> >>>> wrote: >>>> >>>> Hi, >>>> >>>>> >>>>> I'm trying to set a linear problem with a 2x2 block matrix (say A=[A11 >>>>> A12; A21 A22] in matlab notation). Obviously I could generate a single >>>>> matrix, but I would like to keep it as a 2x2 block matrix with the 4 >>>>> blocks >>>>> generated and used independently (e.g., because at some point I need to >>>>> change some of those blocks to matrix-free). My plan was then to >>>>> generate a >>>>> MatShell and set a MATOP_MULT shell operation to explicitly compute >>>>> y=A*x >>>>> by splitting x=[x1;x2] and y=[y1;y2] and carrying out the single >>>>> matrix-vector multiplications, and reassembling the final result. >>>>> >>>>> >>>> >>>> Take a look at MatNest - it does exactly what you want. >>>> >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html >>>> >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATNEST.html#MATNEST >>>> >>>> Thanks, >>>> Dave >>>> >>>> >>>> My problem is that my efforts in extracting x1 and x2 from the original >>>> x >>>> >>>>> vector have failed. The following Fortran code is what seemed to me the >>>>> most reasonable thing to do to set the matrix-vector multiplication, >>>>> but >>>>> it >>>>> does not work >>>>> >>>>> >>>>> !====================================================== >>>>> subroutine MyMult(A,x,y,IERR) >>>>> >>>>> implicit none >>>>> >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> >>>>> Mat A >>>>> Vec x,y, x1, x2, y1, y2 >>>>> IS :: ix1, ix2, iy1, iy2 >>>>> PetscErrorCode ierr >>>>> PetscInt M, N, II, NumberOfUnknowns, NumberOfMeasures >>>>> PetscInt, ALLOCATABLE :: INDICES(:) >>>>> PetscMPIInt :: RANK >>>>> >>>>> CALL VecGetSize(x,N,IERR) >>>>> NumberOfUnknowns = N / 2 >>>>> CALL VecGetSize(y,N,IERR) >>>>> NumberOfMeasures = N - NumberOfUnknowns >>>>> >>>>> N = NumberOfUnknowns + MAX(NumberOfUnknowns,NumberOfMeasures) >>>>> ALLOCATE(INDICES(N)) >>>>> INDICES = (/ (ii, ii=0, N-1) /) >>>>> >>>>> CALL >>>>> >>>>> >>>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,ix1,IERR) >>>>> CALL VecGetSubVector(x,ix1,x1,IERR) >>>>> CALL >>>>> >>>>> >>>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:2*NumberOfUnknowns),PETSC_COPY_VALUES,ix2,IERR) >>>>> CALL VecGetSubVector(x,ix2,x2,IERR) >>>>> CALL >>>>> >>>>> >>>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(1:NumberOfUnknowns),PETSC_COPY_VALUES,iy1,IERR) >>>>> CALL VecGetSubVector(y,iy1,y1,IERR) >>>>> CALL >>>>> >>>>> >>>>> ISCreateGeneral(MPI_COMM_SELF,NumberOfUnknowns,INDICES(NumberOfUnknowns+1:NumberOfUnknowns+NumberOfMeasures),PETSC_COPY_VALUES,iy2,IERR) >>>>> CALL VecGetSubVector(y,iy2,y2,IERR) >>>>> >>>>> CALL MatMult(L_discretized,x1,y1,IERR) >>>>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>>>> CALL MatMult(L_MR,x1,y2,IERR) >>>>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>>>> >>>>> CALL VecRestoreSubVector(y,iy1,y1,IERR) >>>>> CALL VecRestoreSubVector(y,iy2,y2,IERR) >>>>> >>>>> return >>>>> end subroutine MyMult >>>>> !====================================================== >>>>> >>>>> >>>>> Obviously the sequence of calls to ISCreateGeneral and VecGetSubVector >>>>> does not do what I expect, as the errors I'm getting are in the >>>>> following >>>>> MatMult multiplications (the global sizes of x1 and x2 SHOULD BE both >>>>> 771, >>>>> while y1 and y2 should be 771 and 2286): >>>>> >>>>> 1) executed with 1 MPI process: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec y: global dim 2286 771 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #1 MatMult() line 2215 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec v3: local dim 2286 771 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:54:03 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #2 MatMultAdd() line 2396 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> >>>>> >>>>> >>>>> 2) executed with 4 MPI processes (I'm copying the output of process 0 >>>>> only, they're all the same): >>>>> >>>>> .... >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #1 MatMult() line 2214 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #2 MatMultAdd() line 2393 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec x: global dim 771 3084 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #3 MatMult() line 2214 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Mat mat,Vec v1: global dim 771 3084 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: ./ISS on a arch-linux2-c-debug named >>>>> alessandro-HP-ProDesk-490-G2-MT by alessandro Wed Apr 6 10:49:13 2016 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl >>>>> --with-mpi-cc=/home/alessandro/.openmpi-1.8.3/bin/mpicc >>>>> --with-mpi-f90=/home/alessandro/.openmpi-1.8.3/bin/mpif90 >>>>> --with-mpiexec=/home/alessandro/.openmpi-1.8.3/bin/mpiexec >>>>> --with-scalar-type=complex >>>>> [0]PETSC ERROR: #4 MatMultAdd() line 2393 in >>>>> /home/alessandro/CODE/PETSc/petsc-3.6.3/src/mat/interface/matrix.c >>>>> .... >>>>> >>>>> >>>>> Any idea? I would like to keep the flexibility of having a sequence of >>>>> mat-vec multiplications as in >>>>> >>>>> CALL MatMult(L_discretized,x1,y1,IERR) >>>>> CALL MatMultAdd(K_discretized,x2,y1,y1,IERR) >>>>> CALL MatMult(L_MR,x1,y2,IERR) >>>>> CALL MatMultAdd(K_MR,x2,y2,y2,IERR) >>>>> >>>>> but for doing that I need to be able to extract x1 and x2, and to >>>>> reassemble the partial results into y. >>>>> >>>>> Thanks in advance, >>>>> Alessandro >>>>> >>>>> >>>>> ___________________________________ >>> Matteo Alessandro Francavilla, PhD >>> Antenna and EMC Lab (LACE) >>> Istituto Superiore Mario Boella (ISMB) >>> Torino, Italy >>> phone +39 011 2276 711 >>> >>> > ___________________________________ > Matteo Alessandro Francavilla, PhD > Antenna and EMC Lab (LACE) > Istituto Superiore Mario Boella (ISMB) > Torino, Italy > phone +39 011 2276 711 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Wed Apr 6 10:12:43 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Wed, 06 Apr 2016 17:12:43 +0200 Subject: [petsc-users] Prelloc: Get coordinates of local diagonal submatrix Message-ID: <1505307.tmgfdB6JyE@asaru> Hello, in order to preallocate I have to know whether a non-zero will be in the local diagonal submatrix or not. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html says: The DIAGONAL portion of the local submatrix of a processor can be defined as the submatrix which is obtained by extraction the part corresponding to the rows r1-r2 and columns c1-c2 of the global matrix, where r1 is the first row that belongs to the processor, r2 is the last row belonging to the this processor, and c1-c2 is range of indices of the local part of a vector suitable for applying the matrix to. This is an mxn matrix. In the common case of a square matrix, the row and column ranges are the same and the DIAGONAL part is also square. The remaining portion of the local submatrix (mxN) constitute the OFF-DIAGONAL portion. Therefore I compute the begin and end rows and cols like that: int ownerRangeBeginA = _matrixA.ownerRange().first; int ownerRangeEndA = _matrixA.ownerRange().second; // Uses: MatGetVecs(_matrixA.matrix, &vector, nullptr); petsc::Vector diagTest{_matrixA, "test", petsc::Vector::RIGHT}; int localDiagColBegin = diagTest.ownerRange().first; int localDiagColEnd = diagTest.ownerRange().second; Debug("Local Submatrix Rows = " << ownerRangeBeginA << " / " << ownerRangeEndA << ", Local Submatrix Cols = " << localDiagColBegin << " / " << localDiagColEnd); It's a little bit tainted by my PETSc C++ helper function, but I think you get the code. Is there a way to do it more elegantly? Without instantiating a vector just for the purpose of gettings its owner range? And then testing a if an index is inside the local diagonal submatrix? Thanks, Florian From knepley at gmail.com Wed Apr 6 10:19:23 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Apr 2016 10:19:23 -0500 Subject: [petsc-users] Prelloc: Get coordinates of local diagonal submatrix In-Reply-To: <1505307.tmgfdB6JyE@asaru> References: <1505307.tmgfdB6JyE@asaru> Message-ID: On Wed, Apr 6, 2016 at 10:12 AM, Florian Lindner wrote: > Hello, > > in order to preallocate I have to know whether a non-zero will be in the > local diagonal submatrix or not. > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html > > says: > > The DIAGONAL portion of the local submatrix of a processor can be defined > as the submatrix which is obtained by extraction the part corresponding to > the rows r1-r2 and columns c1-c2 of the global matrix, where r1 is the > first row that belongs to the processor, r2 is the last row belonging to > the this processor, and c1-c2 is range of indices of the local part of a > vector suitable for applying the matrix to. This is an mxn matrix. In the > common case of a square matrix, the row and column ranges are the same and > the DIAGONAL part is also square. The remaining portion of the local > submatrix (mxN) constitute the OFF-DIAGONAL portion. > > Therefore I compute the begin and end rows and cols like that: > > int ownerRangeBeginA = _matrixA.ownerRange().first; > int ownerRangeEndA = _matrixA.ownerRange().second; > > // Uses: MatGetVecs(_matrixA.matrix, &vector, nullptr); > petsc::Vector diagTest{_matrixA, "test", petsc::Vector::RIGHT}; > > int localDiagColBegin = diagTest.ownerRange().first; > int localDiagColEnd = diagTest.ownerRange().second; > > Debug("Local Submatrix Rows = " << ownerRangeBeginA << " / " << > ownerRangeEndA << > ", Local Submatrix Cols = " << localDiagColBegin << " / " << > localDiagColEnd); > > > It's a little bit tainted by my PETSc C++ helper function, but I think you > get the code. > > Is there a way to do it more elegantly? Without instantiating a vector > just for the purpose of gettings its owner range? And then testing a if an > index is inside the local diagonal submatrix? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetOwnershipRangesColumn.html Thanks, Matt > Thanks, > Florian > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ppratapa at gatech.edu Wed Apr 6 12:07:34 2016 From: ppratapa at gatech.edu (Phanisri Pradeep Pratapa) Date: Wed, 6 Apr 2016 13:07:34 -0400 Subject: [petsc-users] VecMTDot vs MatMult Message-ID: Hi, I am trying to find out the product of a rectangular dense matrix and a vector. Could you please let me know if forming the rectangular matrix as an array of Vecs and then using VecMTDot is better (or worse/similar) than doing MatMult? (In both cases the matrix and the vectors are parallely distributed) Thank you, Regards, Pradeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 6 13:35:14 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 6 Apr 2016 13:35:14 -0500 Subject: [petsc-users] VecMTDot vs MatMult In-Reply-To: References: Message-ID: <39BF68EA-247D-4CBD-99B3-B858E1B0C09A@mcs.anl.gov> They should be pretty similar. So use whatever makes sense for the rest of the code with respect to the "rectangular dense matrix" or collection of vectors. Barry > On Apr 6, 2016, at 12:07 PM, Phanisri Pradeep Pratapa wrote: > > Hi, > > I am trying to find out the product of a rectangular dense matrix and a vector. Could you please let me know if forming the rectangular matrix as an array of Vecs and then using VecMTDot is better (or worse/similar) than doing MatMult? (In both cases the matrix and the vectors are parallely distributed) > > Thank you, > > Regards, > > Pradeep From Sander.Arens at UGent.be Wed Apr 6 14:06:14 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Wed, 6 Apr 2016 21:06:14 +0200 Subject: [petsc-users] Discretized fields at quadrature points Message-ID: Hi, I'd like to be able to represent some discretized fields at the quadrature points of a finite element, so I can easily use them with the plex residual/function evaluations. Is it possible to do this with PetscFECreateDefault and some command line options? I think what I need is PETSCSPACEDG for the space, but I'm not sure what type I should take for the dual space? Thanks, Sander -------------- next part -------------- An HTML attachment was scrubbed... URL: From d019598 at polito.it Thu Apr 7 05:31:51 2016 From: d019598 at polito.it (FRANCAVILLA MATTEO ALESSANDRO) Date: Thu, 07 Apr 2016 12:31:51 +0200 Subject: [petsc-users] Ordering with MatCreateNest and MPI Message-ID: Hi, I'm trying to use a matrix composed of different sub-matrix blocks; I create the matrix with MatCreateNest(). I'm finding it hard to understand the ordering (in MPI applications) of the result of MatMult operations. Apparently the result of the multiplication is correct, except that both input and output vectors go through some sort of permutation. How should I use the MatMult routine, i.e. how to pass the input vector with the correct ordering and how to get the correct ordering of the result? Below a simple Fortran example to demonstrate what I'm trying to do (running the example with 1 process works fine, with 2 or more processes I don't understand how to get the correct ordering). When using standard matrices (i.e., not MatNest) I've never had to care about orderings. Thanks, Alessandro PROGRAM Main !.....================================ implicit none #include #include #include !********************************************************************* ! PETSc variables !********************************************************************* Vec :: x, b Mat :: matArray(4), A PetscInt :: II, JJ, FromRow, ToRow, IMAT PetscErrorCode :: IERR PetscInt, ALLOCATABLE :: INDICES(:) PetscScalar, ALLOCATABLE :: VALUES(:) PetscInt, PARAMETER :: N = 2 ! size of sub-matrices CALL PetscInitialize(PETSC_NULL_CHARACTER,IERR) ALLOCATE(INDICES(N),VALUES(N)) INDICES = (/ (ii, ii=0, N-1) /) ! Form 4 blocks of size NxN DO IMAT = 1, 4 CALL MatCreateDense(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,PETSC_NULL_SCALAR,matArray(IMAT),IERR) CALL MatGetOwnershipRange(matArray(IMAT),FromRow,ToRow,IERR) DO II = FromRow, ToRow-1 DO JJ = 1, N VALUES(JJ) = (IMAT-1)*100 + N*II + JJ END DO CALL MatSetValues(matArray(IMAT),1,II,N,INDICES,VALUES,INSERT_VALUES,IERR) END DO CALL MatAssemblyBegin(matArray(IMAT),MAT_FINAL_ASSEMBLY,IERR) CALL MatAssemblyEnd(matArray(IMAT),MAT_FINAL_ASSEMBLY,IERR) END DO ! Form a 2x2 block matrix made as [ matArray(1) matArray(2) ; matArray(3) matArray(4) ] ! If N = 2 the matrix is: ! A = [ 1 2 | 101 102 ; ! 3 4 | 102 103 ; ! ___________________ ! 201 202 | 401 402 ; ! 203 204 | 403 404 ] CALL MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matArray,A,IERR) CALL MatCreateVecs(A,x,b,IERR) CALL VecSet(x,0.+0.*PETSC_i,IERR) CALL VecSetValues(x,1,2,(1.+0.*PETSC_i),INSERT_VALUES,IERR) CALL VecAssemblyBegin(x,IERR) CALL VecAssemblyEnd(x,IERR) CALL VecView(x,PETSC_VIEWER_STDOUT_WORLD,IERR) CALL MatMult(A,x,b,IERR) ! A*x extracts the 3rd column of A CALL VecView(b,PETSC_VIEWER_STDOUT_WORLD,IERR) DEALLOCATE(INDICES,VALUES) CALL PetscFinalize(IERR) END PROGRAM From jed at jedbrown.org Thu Apr 7 08:25:47 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 14:25:47 +0100 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: References: Message-ID: <87shyxzh2c.fsf@jedbrown.org> FRANCAVILLA MATTEO ALESSANDRO writes: > Hi, > > I'm trying to use a matrix composed of different sub-matrix blocks; I create > the matrix with MatCreateNest(). I'm finding it hard to understand the > ordering (in MPI applications) of the result of MatMult operations. > Apparently the result of the multiplication is correct, except that both > input and output vectors go through some sort of permutation. How should I > use the MatMult routine, i.e. how to pass the input vector with the correct > ordering and how to get the correct ordering of the result? > > Below a simple Fortran example to demonstrate what I'm trying to do (running > the example with 1 process works fine, with 2 or more processes I don't > understand how to get the correct ordering). When using standard matrices > (i.e., not MatNest) I've never had to care about orderings. MatNest is semantically identical to other matrix formats. In fact, we strongly recommend that you write your code so that your assembly and all subsequent operations do not depend in any way on the matrix being MATNEST of some other format. See src/snes/examples/tutorials/ex28.c for an example of this. But anyway, you can use VecGetSubVector to separate out the components (MatNestGetISs to get the index sets if you don't already have them). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mailinglists at xgm.de Thu Apr 7 08:56:25 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Thu, 07 Apr 2016 15:56:25 +0200 Subject: [petsc-users] Prelloc: Get coordinates of local diagonal submatrix In-Reply-To: References: <1505307.tmgfdB6JyE@asaru> Message-ID: <3274944.Nldat5oCoq@asaru> Hello, > MatGetOwnershipRangesColumn Thanks! That's what I was looking for. I have set a MatSetLocalToGlobalMapping on the matrix and I use MatMPIAIJSetPreallocation(_matrixA.matrix, 0, d_nnz, 0, o_nnz) to set the preallocation. Am I right to assume that nnz is to in the the PETSc ordering of rows and not application ordering? Best Thanks, Florian Am Mittwoch, 6. April 2016, 10:19:23 CEST schrieben Sie: > On Wed, Apr 6, 2016 at 10:12 AM, Florian Lindner > wrote: > > > Hello, > > > > in order to preallocate I have to know whether a non-zero will be in the > > local diagonal submatrix or not. > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html > > > > says: > > > > The DIAGONAL portion of the local submatrix of a processor can be defined > > as the submatrix which is obtained by extraction the part corresponding to > > the rows r1-r2 and columns c1-c2 of the global matrix, where r1 is the > > first row that belongs to the processor, r2 is the last row belonging to > > the this processor, and c1-c2 is range of indices of the local part of a > > vector suitable for applying the matrix to. This is an mxn matrix. In the > > common case of a square matrix, the row and column ranges are the same and > > the DIAGONAL part is also square. The remaining portion of the local > > submatrix (mxN) constitute the OFF-DIAGONAL portion. > > > > Therefore I compute the begin and end rows and cols like that: > > > > int ownerRangeBeginA = _matrixA.ownerRange().first; > > int ownerRangeEndA = _matrixA.ownerRange().second; > > > > // Uses: MatGetVecs(_matrixA.matrix, &vector, nullptr); > > petsc::Vector diagTest{_matrixA, "test", petsc::Vector::RIGHT}; > > > > int localDiagColBegin = diagTest.ownerRange().first; > > int localDiagColEnd = diagTest.ownerRange().second; > > > > Debug("Local Submatrix Rows = " << ownerRangeBeginA << " / " << > > ownerRangeEndA << > > ", Local Submatrix Cols = " << localDiagColBegin << " / " << > > localDiagColEnd); > > > > > > It's a little bit tainted by my PETSc C++ helper function, but I think you > > get the code. > > > > Is there a way to do it more elegantly? Without instantiating a vector > > just for the purpose of gettings its owner range? And then testing a if an > > index is inside the local diagonal submatrix? > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetOwnershipRangesColumn.html > > Thanks, > > Matt > > > > Thanks, > > Florian > > > > > > From jed at jedbrown.org Thu Apr 7 09:02:23 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 15:02:23 +0100 Subject: [petsc-users] Prelloc: Get coordinates of local diagonal submatrix In-Reply-To: <3274944.Nldat5oCoq@asaru> References: <1505307.tmgfdB6JyE@asaru> <3274944.Nldat5oCoq@asaru> Message-ID: <87mvp5zfdc.fsf@jedbrown.org> Florian Lindner writes: >Am I right to assume that nnz is to in the the PETSc ordering of rows and not application ordering? Yes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From matteo.francavilla at polito.it Thu Apr 7 07:23:36 2016 From: matteo.francavilla at polito.it (Alessandro Francavilla) Date: Thu, 7 Apr 2016 14:23:36 +0200 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: <87shyxzh2c.fsf@jedbrown.org> References: <87shyxzh2c.fsf@jedbrown.org> Message-ID: <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> Thanks for the reply; probably I've not been clear enough. I don't need to separate out components, I just need to understand the underlying orderings. For instance, suppose that I want to multiply the matrix with a null vector with a 1 in the i-th location (the result is basically the i-th column of the matrix): CALL MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matAr ray,A,IERR) CALL MatCreateVecs(A,x,b,IERR) CALL VecSet(x,0.+0.*PETSC_i,IERR) CALL VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) CALL VecAssemblyBegin(x,IERR) CALL VecAssemblyEnd(x,IERR) CALL MatMult(A,x,b,IERR) The above code works perfectly if I execute it single process (mpirun -np 1); if I run it on >1 MPI processes, the result is not what I would expect: rather than returning the i-th column, it returns (e.g., if I call VecView on b) some permutation of a different column (say a permutation of column j). Also, if I substitute the call to MatCreateNest() with A = matArray(1) Then everything works as expected, for arbitrary numbers of processes. By working as expected I mean that calling VecView on b prints the i-th column of A with its natural ordering. So, my question is: 1) how do I generate the input vector x such that the MatMult operation returns the i-th column of A? 2) how do I reorder the result of MatMult such that I get the i-th column with its "natural" ordering? Maybe I'm even misunderstanding the output of VecView, which prints the elements process by process, and thus the order corresponds to the natural ordering only if the elements are distributed consecutively among processes? Which would explain why I'm seeing a permutation of the column. However, I'm seeing the permutation of a DIFFERENT column of the matrix, which means that the input vector does not have a 1 in the i-th position when I fill it with VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR). Thanks, Alessandro -----Original Message----- From: Jed Brown [mailto:jed at jedbrown.org] Sent: gioved? 7 aprile 2016 15:26 To: FRANCAVILLA MATTEO ALESSANDRO; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Ordering with MatCreateNest and MPI FRANCAVILLA MATTEO ALESSANDRO writes: > Hi, > > I'm trying to use a matrix composed of different sub-matrix blocks; I > create the matrix with MatCreateNest(). I'm finding it hard to > understand the ordering (in MPI applications) of the result of MatMult operations. > Apparently the result of the multiplication is correct, except that > both input and output vectors go through some sort of permutation. How > should I use the MatMult routine, i.e. how to pass the input vector > with the correct ordering and how to get the correct ordering of the result? > > Below a simple Fortran example to demonstrate what I'm trying to do > (running the example with 1 process works fine, with 2 or more > processes I don't understand how to get the correct ordering). When > using standard matrices (i.e., not MatNest) I've never had to care about orderings. MatNest is semantically identical to other matrix formats. In fact, we strongly recommend that you write your code so that your assembly and all subsequent operations do not depend in any way on the matrix being MATNEST of some other format. See src/snes/examples/tutorials/ex28.c for an example of this. But anyway, you can use VecGetSubVector to separate out the components (MatNestGetISs to get the index sets if you don't already have them). From jed at jedbrown.org Thu Apr 7 13:03:58 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 19:03:58 +0100 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> References: <87shyxzh2c.fsf@jedbrown.org> <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> Message-ID: <871t6hz46p.fsf@jedbrown.org> Alessandro Francavilla writes: > Thanks for the reply; probably I've not been clear enough. I don't need to > separate out components, I just need to understand the underlying orderings. > > For instance, suppose that I want to multiply the matrix with a null vector > with a 1 in the i-th location (the result is basically the i-th column of > the matrix): > > CALL > MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,matAr > ray,A,IERR) > CALL MatCreateVecs(A,x,b,IERR) > CALL VecSet(x,0.+0.*PETSC_i,IERR) > CALL VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) > CALL VecAssemblyBegin(x,IERR) > CALL VecAssemblyEnd(x,IERR) > CALL MatMult(A,x,b,IERR) > > The above code works perfectly if I execute it single process (mpirun -np > 1); if I run it on >1 MPI processes, the result is not what I would expect: > rather than returning the i-th column, it returns (e.g., if I call VecView > on b) some permutation of a different column (say a permutation of column > j). > > Also, if I substitute the call to MatCreateNest() with > A = matArray(1) But this is a different matrix. Why would you expect that the MatNest has the same indices as its first Mat? > Then everything works as expected, for arbitrary numbers of processes. By > working as expected I mean that calling VecView on b prints the i-th column > of A with its natural ordering. > > So, my question is: > 1) how do I generate the input vector x such that the MatMult operation > returns the i-th column of A? It is returning the ith column of A. A is not just matArray(1). > 2) how do I reorder the result of MatMult such that I get the i-th column > with its "natural" ordering? > > Maybe I'm even misunderstanding the output of VecView, which prints the > elements process by process, and thus the order corresponds to the natural > ordering only if the elements are distributed consecutively among processes? > Which would explain why I'm seeing a permutation of the column. However, I'm > seeing the permutation of a DIFFERENT column of the matrix, which means that > the input vector does not have a 1 in the i-th position when I fill it with > VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR). PETSc Vecs are distributed with contiguous blocks assigned to each process. Mat's column and row indices are induced by their action on Vecs. MatNest does not move matrix data around, so its blocks produce the portions of Vecs that are owned on the current process. Consequently, MatNest induces an ordering in which one field gets the first chunk of a contiguous space on each process, with the intervening entries for the other field(s). If you have two matrices, B and C, each distributed over the same two processes, then the MatNest [B 0; 0 C] will keep both B and C distributed. I.e., it does not move all of B to rank 0 and all of C to rank 1, or some funny business like that. If you don't want to think about this ordering, extract the subvector the way I described in the previous email. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Thu Apr 7 13:30:05 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 19:30:05 +0100 Subject: [petsc-users] PDESoft 2016, UK, 4-8 Jul 2016 (abstract submission deadline extended) Message-ID: <87twjdxoeq.fsf@jedbrown.org> The abstract submission deadline has been extended to April 22. Please consider this small and interesting conference on your summer schedule. We are happy to announce that PDESoft 2016 will be held at the University of Warwick, UK. This main conference will run from 4-6 July and the remaining time will be dedicated to small group coding days. PDESoft 2016 provides a forum for the developers of open-source tools solving the diverse stages of the numerical PDE process to exchange the latest developments and future ideas. If you develop: - meshing tools; - HPC tools; - solvers for large systems of equations; - numerical PDE solvers; - visualisation systems for PDE; - coupling with or user interfaces to scientific software; or any other part of the PDE toolchain, this is the workshop for you. For more details please visit http://warwick.ac.uk/pdesoft2016 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From qiu.lingyun at gmail.com Thu Apr 7 14:13:24 2016 From: qiu.lingyun at gmail.com (Lingyun Qiu) Date: Thu, 7 Apr 2016 14:13:24 -0500 Subject: [petsc-users] L1 or projection type regularization with PETSc Message-ID: Dear all, I am working an optimization problem as min_x ||Ax - b||_2^2 + alpha ||x||_1 For the fidelity term, we use L2 norm. We use L1 norm for the regularization term. Without regularization term, i.e., alpha=0, we iteratively solve the problem as x_k+1 = KSP(x_k). I plan to use the split Bregman method to solve the regularized problem. It reads as, y_k+1 = KSP(x_k) x_k+1 = B(y_k+1) Here B() is the function related to the Bregman method. It works as a post-processing of the iterates. I am wondering is there a way to combine this post-processing with the KSP solver? A brute-force way is modify the initial guess and set the max iteration number to 1. This is also related to the projection type regularization: min_{x in subspace G} ||Ax-b||^2_2 The scheme is y_k+1 = KSP(x_k) x_k+1 = P_G(y_k+1) where P_G is the projection to subspace G. Lingyun Qiu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 7 14:24:12 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 20:24:12 +0100 Subject: [petsc-users] L1 or projection type regularization with PETSc In-Reply-To: References: Message-ID: <87h9fdxlwj.fsf@jedbrown.org> Lingyun Qiu writes: > Dear all, > > I am working an optimization problem as > min_x ||Ax - b||_2^2 + alpha ||x||_1 > > For the fidelity term, we use L2 norm. We use L1 norm for the > regularization term. Without regularization term, i.e., alpha=0, we > iteratively solve the problem as > x_k+1 = KSP(x_k). I.e., a single KSPSolve because the optimization problem is quadratic? > I plan to use the split Bregman method to solve the regularized problem. It > reads as, > y_k+1 = KSP(x_k) > x_k+1 = B(y_k+1) > Here B() is the function related to the Bregman method. It works as a > post-processing of the iterates. But outside the KSPSolve, not at each iteration of the KSPSolve... B is a nonlinear operation, so you can't put it inside KSP. You could put it inside SNES (perhaps with nonlinear preconditioning, NGMRES or QN), but I don't think that's what you're asking for here. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Thu Apr 7 14:24:17 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 7 Apr 2016 14:24:17 -0500 Subject: [petsc-users] L1 or projection type regularization with PETSc In-Reply-To: References: Message-ID: On Thu, Apr 7, 2016 at 2:13 PM, Lingyun Qiu wrote: > Dear all, > > I am working an optimization problem as > min_x ||Ax - b||_2^2 + alpha ||x||_1 > > For the fidelity term, we use L2 norm. We use L1 norm for the > regularization term. Without regularization term, i.e., alpha=0, we > iteratively solve the problem as > x_k+1 = KSP(x_k). > > I plan to use the split Bregman method to solve the regularized problem. > It reads as, > y_k+1 = KSP(x_k) > x_k+1 = B(y_k+1) > Here B() is the function related to the Bregman method. It works as a > post-processing of the iterates. > > I am wondering is there a way to combine this post-processing with the KSP > solver? A brute-force way is modify the initial guess and set the max > iteration number to 1. > Are you asking for something like this: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetPostSolve.html Thanks, Matt > This is also related to the projection type regularization: > min_{x in subspace G} ||Ax-b||^2_2 > The scheme is > y_k+1 = KSP(x_k) > x_k+1 = P_G(y_k+1) > where P_G is the projection to subspace G. > > Lingyun Qiu > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiu.lingyun at gmail.com Thu Apr 7 14:59:35 2016 From: qiu.lingyun at gmail.com (Lingyun Qiu) Date: Thu, 7 Apr 2016 14:59:35 -0500 Subject: [petsc-users] L1 or projection type regularization with PETSc In-Reply-To: <87h9fdxlwj.fsf@jedbrown.org> References: <87h9fdxlwj.fsf@jedbrown.org> Message-ID: On Thu, Apr 7, 2016 at 2:24 PM, Jed Brown wrote: > Lingyun Qiu writes: > > > Dear all, > > > > I am working an optimization problem as > > min_x ||Ax - b||_2^2 + alpha ||x||_1 > > > > For the fidelity term, we use L2 norm. We use L1 norm for the > > regularization term. Without regularization term, i.e., alpha=0, we > > iteratively solve the problem as > > x_k+1 = KSP(x_k). > > I.e., a single KSPSolve because the optimization problem is quadratic? Yes. A single KSPSolve because of the quadratic form. k stands for the iteration index inside KSPSolve. > > > I plan to use the split Bregman method to solve the regularized problem. > It > > reads as, > > y_k+1 = KSP(x_k) > > x_k+1 = B(y_k+1) > > Here B() is the function related to the Bregman method. It works as a > > post-processing of the iterates. > > But outside the KSPSolve, not at each iteration of the KSPSolve... > I need to modify the iterate after each iteration inside the KSPSolve. Yes, B is a nonlinear operation. > > B is a nonlinear operation, so you can't put it inside KSP. You could > put it inside SNES (perhaps with nonlinear preconditioning, NGMRES or > QN), but I don't think that's what you're asking for here. > I will check these functions. Thanks for the quick reply. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiu.lingyun at gmail.com Thu Apr 7 15:04:41 2016 From: qiu.lingyun at gmail.com (Lingyun Qiu) Date: Thu, 7 Apr 2016 15:04:41 -0500 Subject: [petsc-users] L1 or projection type regularization with PETSc In-Reply-To: References: Message-ID: I think *KSPSetPostSolve * is to add a post-processing after all the iterations inside KSPSolve finish. I need to modify the iterates inside KSPSolve or the linear solver. I just realize that this nonlinear operation will also affect the Krylov subspaces. Perhaps I should look for some other linear solver instead of KSPSolve. Comments are welcome. On Thu, Apr 7, 2016 at 2:24 PM, Matthew Knepley wrote: > On Thu, Apr 7, 2016 at 2:13 PM, Lingyun Qiu wrote: > >> Dear all, >> >> I am working an optimization problem as >> min_x ||Ax - b||_2^2 + alpha ||x||_1 >> >> For the fidelity term, we use L2 norm. We use L1 norm for the >> regularization term. Without regularization term, i.e., alpha=0, we >> iteratively solve the problem as >> x_k+1 = KSP(x_k). >> >> I plan to use the split Bregman method to solve the regularized problem. >> It reads as, >> y_k+1 = KSP(x_k) >> x_k+1 = B(y_k+1) >> Here B() is the function related to the Bregman method. It works as a >> post-processing of the iterates. >> >> I am wondering is there a way to combine this post-processing with the >> KSP solver? A brute-force way is modify the initial guess and set the max >> iteration number to 1. >> > > Are you asking for something like this: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetPostSolve.html > > Thanks, > > Matt > > >> This is also related to the projection type regularization: >> min_{x in subspace G} ||Ax-b||^2_2 >> The scheme is >> y_k+1 = KSP(x_k) >> x_k+1 = P_G(y_k+1) >> where P_G is the projection to subspace G. >> >> Lingyun Qiu >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 7 15:07:10 2016 From: jed at jedbrown.org (Jed Brown) Date: Thu, 07 Apr 2016 21:07:10 +0100 Subject: [petsc-users] L1 or projection type regularization with PETSc In-Reply-To: References: <87h9fdxlwj.fsf@jedbrown.org> Message-ID: <87bn5lxjwx.fsf@jedbrown.org> Lingyun Qiu writes: > On Thu, Apr 7, 2016 at 2:24 PM, Jed Brown wrote: > >> Lingyun Qiu writes: >> >> > Dear all, >> > >> > I am working an optimization problem as >> > min_x ||Ax - b||_2^2 + alpha ||x||_1 >> > >> > For the fidelity term, we use L2 norm. We use L1 norm for the >> > regularization term. Without regularization term, i.e., alpha=0, we >> > iteratively solve the problem as >> > x_k+1 = KSP(x_k). >> >> I.e., a single KSPSolve because the optimization problem is quadratic? > > Yes. A single KSPSolve because of the quadratic form. k stands for the > iteration index inside KSPSolve. But the k+1 iterate is not a function of just the previous iterate, it is a function of all previous iterates (sometimes via recurrence). >> > I plan to use the split Bregman method to solve the regularized problem. >> It >> > reads as, >> > y_k+1 = KSP(x_k) >> > x_k+1 = B(y_k+1) >> > Here B() is the function related to the Bregman method. It works as a >> > post-processing of the iterates. >> >> But outside the KSPSolve, not at each iteration of the KSPSolve... >> > I need to modify the iterate after each iteration inside the KSPSolve. Yes, > B is a nonlinear operation. That nonlinear operation breaks orthogonality, so your Krylov method won't work. You can't just tinker willy-nilly with the vectors inside the Krylov solver. There are specialized Krylov methods for solving problems with certain constraints, such as QCG for solving SPD systems subject to trust region constraints. But not in general. Call KSPSolve and then do your shrink operation, rinse and repeat, perhaps with nonlinear acceleration. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From matteo.francavilla at polito.it Fri Apr 8 03:22:29 2016 From: matteo.francavilla at polito.it (Alessandro Francavilla) Date: Fri, 8 Apr 2016 10:22:29 +0200 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: <871t6hz46p.fsf@jedbrown.org> References: <87shyxzh2c.fsf@jedbrown.org> <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> <871t6hz46p.fsf@jedbrown.org> Message-ID: <000601d1916f$caeb7140$60c253c0$@polito.it> In fact I don't expect anything special about indices, I just can't understand how to use them. If I set a matrix [A B; C D], and each of the blocks is distributed over some arbitrary number of processes, I want to know how to compute the product y = [A B; C D]*x and be able to see the result. I'm not concerned about how the elements are distributed, I just want to be able to execute the matmult operation and view the result. > It is returning the ith column of A. A is not just matArray(1). No, it is not. I know very well that A is not matArray(1), and this operation is returning one column of A, but definitely not i-th column. I'm definitely sure it is my mistake: 1) building the vector x the way I am setting it does not actually do what I'm expecting it to do; 2) even if I fix (1) I still need to permute the result in order to obtain y as defined by y=[A B;C D]*x . This is what's going on, which summarizes my 2 mistakes: (1) the result needs be permuted, and (2) the input vector x is not correct This is what the matrix looks like: ! A = [ 1 2 | 101 102 ; ! 3 4 | 102 103 ; ! ___________________ ! 201 202 | 401 402 ; ! 203 204 | 403 404 ] If I set the x vector with CALL VecSetValues(x,1,0,(1.+0.*PETSC_i),INSERT_VALUES,IERR) The result with 1 MPI process is what I expect (the first column of A): Vec Object: 1 MPI processes type: seq 1 3 201 203 With 2 MPI processes the results are distributed: the result is correct but I need to reorder it to get [1 3 201 203]' but don't know how: Vec Object: 2 MPI processes type: mpi Process [0] 1 201 Process [1] 3 203 If I set the x vector with CALL VecSetValues(x,1,1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) 1 MPI process correctly yields the 2nd column of A: Vec Object: 1 MPI processes type: seq 2 4 202 204 2 MPI processes DO NOT YIELD the 2nd column, but the 3rd column (apart from a permutation similarly to the previous case): Vec Object: 2 MPI processes type: mpi Process [0] 101 301 Process [1] 103 303 -----Original Message----- From: Jed Brown [mailto:jed at jedbrown.org] Sent: gioved? 7 aprile 2016 20:04 To: Alessandro Francavilla Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] Ordering with MatCreateNest and MPI Alessandro Francavilla writes: > Thanks for the reply; probably I've not been clear enough. I don't > need to separate out components, I just need to understand the underlying orderings. > > For instance, suppose that I want to multiply the matrix with a null > vector with a 1 in the i-th location (the result is basically the i-th > column of the matrix): > > CALL > MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT > ,matAr > ray,A,IERR) > CALL MatCreateVecs(A,x,b,IERR) > CALL VecSet(x,0.+0.*PETSC_i,IERR) > CALL VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) > CALL VecAssemblyBegin(x,IERR) > CALL VecAssemblyEnd(x,IERR) > CALL MatMult(A,x,b,IERR) > > The above code works perfectly if I execute it single process (mpirun > -np 1); if I run it on >1 MPI processes, the result is not what I would expect: > rather than returning the i-th column, it returns (e.g., if I call > VecView on b) some permutation of a different column (say a > permutation of column j). > > Also, if I substitute the call to MatCreateNest() with A = matArray(1) But this is a different matrix. Why would you expect that the MatNest has the same indices as its first Mat? > Then everything works as expected, for arbitrary numbers of processes. > By working as expected I mean that calling VecView on b prints the > i-th column of A with its natural ordering. > > So, my question is: > 1) how do I generate the input vector x such that the MatMult > operation returns the i-th column of A? It is returning the ith column of A. A is not just matArray(1). > 2) how do I reorder the result of MatMult such that I get the i-th > column with its "natural" ordering? > > Maybe I'm even misunderstanding the output of VecView, which prints > the elements process by process, and thus the order corresponds to the > natural ordering only if the elements are distributed consecutively among processes? > Which would explain why I'm seeing a permutation of the column. > However, I'm seeing the permutation of a DIFFERENT column of the > matrix, which means that the input vector does not have a 1 in the > i-th position when I fill it with VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR). PETSc Vecs are distributed with contiguous blocks assigned to each process. Mat's column and row indices are induced by their action on Vecs. MatNest does not move matrix data around, so its blocks produce the portions of Vecs that are owned on the current process. Consequently, MatNest induces an ordering in which one field gets the first chunk of a contiguous space on each process, with the intervening entries for the other field(s). If you have two matrices, B and C, each distributed over the same two processes, then the MatNest [B 0; 0 C] will keep both B and C distributed. I.e., it does not move all of B to rank 0 and all of C to rank 1, or some funny business like that. If you don't want to think about this ordering, extract the subvector the way I described in the previous email. From jed at jedbrown.org Fri Apr 8 03:59:28 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 08 Apr 2016 09:59:28 +0100 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: <000601d1916f$caeb7140$60c253c0$@polito.it> References: <87shyxzh2c.fsf@jedbrown.org> <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> <871t6hz46p.fsf@jedbrown.org> <000601d1916f$caeb7140$60c253c0$@polito.it> Message-ID: <878u0owk5r.fsf@jedbrown.org> Alessandro Francavilla writes: > In fact I don't expect anything special about indices, I just can't > understand how to use them. > > If I set a matrix [A B; C D], and each of the blocks is distributed over > some arbitrary number of processes, I want to know how to compute the > product > y = [A B; C D]*x > and be able to see the result. I'm not concerned about how the elements are > distributed, I just want to be able to execute the matmult operation and > view the result. MatNestGetISs give you the ISs to extract or set the portions of x and y corresponding to the first and second block. The index sets are not globally contiguous and can't be if you want sane parallelism. >> It is returning the ith column of A. A is not just matArray(1). > > No, it is not. I know very well that A is not matArray(1), and this > operation is returning one column of A, but definitely not i-th > column. Evidently the ith column is not defined the same way as your internal mental model. Use the index sets. > I'm definitely sure it is my mistake: 1) building the vector x the way > I am setting it does not actually do what I'm expecting it to do; 2) > even if I fix (1) I still need to permute the result in order to > obtain y as defined by y=[A B;C D]*x . > > This is what's going on, which summarizes my 2 mistakes: (1) the result > needs be permuted, and (2) the input vector x is not correct > > This is what the matrix looks like: > > ! A = [ 1 2 | 101 102 ; > ! 3 4 | 102 103 ; > ! ___________________ > ! 201 202 | 401 402 ; > ! 203 204 | 403 404 ] > > > If I set the x vector with CALL > VecSetValues(x,1,0,(1.+0.*PETSC_i),INSERT_VALUES,IERR) > > The result with 1 MPI process is what I expect (the first column of A): > > Vec Object: 1 MPI processes > type: seq > 1 > 3 > 201 > 203 > > With 2 MPI processes the results are distributed: the result is correct but > I need to reorder it to get [1 3 201 203]' but don't know how: > > Vec Object: 2 MPI processes > type: mpi > Process [0] > 1 > 201 > Process [1] > 3 > 203 > > > If I set the x vector with CALL > VecSetValues(x,1,1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) > > 1 MPI process correctly yields the 2nd column of A: > > Vec Object: 1 MPI processes > type: seq > 2 > 4 > 202 > 204 > > 2 MPI processes DO NOT YIELD the 2nd column, but the 3rd column (apart from > a permutation similarly to the previous case): > > Vec Object: 2 MPI processes > type: mpi > Process [0] > 101 > 301 > Process [1] > 103 > 303 > > > -----Original Message----- > From: Jed Brown [mailto:jed at jedbrown.org] > Sent: gioved? 7 aprile 2016 20:04 > To: Alessandro Francavilla > Cc: petsc-users at mcs.anl.gov > Subject: RE: [petsc-users] Ordering with MatCreateNest and MPI > > Alessandro Francavilla writes: > >> Thanks for the reply; probably I've not been clear enough. I don't >> need to separate out components, I just need to understand the underlying > orderings. >> >> For instance, suppose that I want to multiply the matrix with a null >> vector with a 1 in the i-th location (the result is basically the i-th >> column of the matrix): >> >> CALL >> MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT >> ,matAr >> ray,A,IERR) >> CALL MatCreateVecs(A,x,b,IERR) >> CALL VecSet(x,0.+0.*PETSC_i,IERR) >> CALL VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) >> CALL VecAssemblyBegin(x,IERR) >> CALL VecAssemblyEnd(x,IERR) >> CALL MatMult(A,x,b,IERR) >> >> The above code works perfectly if I execute it single process (mpirun >> -np 1); if I run it on >1 MPI processes, the result is not what I would > expect: >> rather than returning the i-th column, it returns (e.g., if I call >> VecView on b) some permutation of a different column (say a >> permutation of column j). >> >> Also, if I substitute the call to MatCreateNest() with A = matArray(1) > > But this is a different matrix. Why would you expect that the MatNest has > the same indices as its first Mat? > >> Then everything works as expected, for arbitrary numbers of processes. >> By working as expected I mean that calling VecView on b prints the >> i-th column of A with its natural ordering. >> >> So, my question is: >> 1) how do I generate the input vector x such that the MatMult >> operation returns the i-th column of A? > > It is returning the ith column of A. A is not just matArray(1). > > >> 2) how do I reorder the result of MatMult such that I get the i-th >> column with its "natural" ordering? >> >> Maybe I'm even misunderstanding the output of VecView, which prints >> the elements process by process, and thus the order corresponds to the >> natural ordering only if the elements are distributed consecutively among > processes? >> Which would explain why I'm seeing a permutation of the column. >> However, I'm seeing the permutation of a DIFFERENT column of the >> matrix, which means that the input vector does not have a 1 in the >> i-th position when I fill it with > VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR). > > PETSc Vecs are distributed with contiguous blocks assigned to each process. > Mat's column and row indices are induced by their action on Vecs. MatNest > does not move matrix data around, so its blocks produce the portions of Vecs > that are owned on the current process. > Consequently, MatNest induces an ordering in which one field gets the first > chunk of a contiguous space on each process, with the intervening entries > for the other field(s). If you have two matrices, B and C, each distributed > over the same two processes, then the MatNest > > [B 0; 0 C] > > will keep both B and C distributed. I.e., it does not move all of B to rank > 0 and all of C to rank 1, or some funny business like that. If you don't > want to think about this ordering, extract the subvector the way I described > in the previous email. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Fri Apr 8 09:48:17 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Apr 2016 09:48:17 -0500 Subject: [petsc-users] Discretized fields at quadrature points In-Reply-To: References: Message-ID: On Wed, Apr 6, 2016 at 2:06 PM, Sander Arens wrote: > Hi, > > I'd like to be able to represent some discretized fields at the quadrature > points of a finite element, so I can easily use them with the plex > residual/function evaluations. Is it possible to do this with > PetscFECreateDefault and some command line options? > I think what I need is PETSCSPACEDG for the space, but I'm not sure what > type I should take for the dual space? > This is why I put that in there. It really should not be named DG, but there are reasons I thought it made sense. I had not thought about a dual space, since I was not going to project into it. I think you can just use whatever is the default. Matt > Thanks, > Sander > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sander.Arens at UGent.be Fri Apr 8 10:35:02 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Fri, 8 Apr 2016 17:35:02 +0200 Subject: [petsc-users] Discretized fields at quadrature points In-Reply-To: References: Message-ID: What does it stand for? Discrete Gauss? Maybe PETSCSPACEQUADRATURE is a better name? The reason I want to use this is because these fields will be representing a bunch of ODE's at each point (not coupled spatially) and it would be nice if their mass matrices would be diagonal. So the dual space should be Dirac delta's at the quadrature points and that's different from the default (Lagrange). I'll see if I can code something up for this. Any suggestions for a name for this dual space? Thanks, Sander On 8 April 2016 at 16:48, Matthew Knepley wrote: > On Wed, Apr 6, 2016 at 2:06 PM, Sander Arens > wrote: > >> Hi, >> >> I'd like to be able to represent some discretized fields at the >> quadrature points of a finite element, so I can easily use them with the >> plex residual/function evaluations. Is it possible to do this with >> PetscFECreateDefault and some command line options? >> I think what I need is PETSCSPACEDG for the space, but I'm not sure what >> type I should take for the dual space? >> > > This is why I put that in there. It really should not be named DG, but > there are reasons I thought it made sense. > > I had not thought about a dual space, since I was not going to project > into it. I think you can just use whatever is the default. > > Matt > > >> Thanks, >> Sander >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sander.Arens at UGent.be Fri Apr 8 12:07:17 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Fri, 8 Apr 2016 19:07:17 +0200 Subject: [petsc-users] Discretized fields at quadrature points In-Reply-To: References: Message-ID: Or perhaps there should be a flag for PETSCDUALSPACELAGRANGE to evaluate at the quadrature points? On 8 April 2016 at 17:35, Sander Arens wrote: > What does it stand for? Discrete Gauss? Maybe PETSCSPACEQUADRATURE is a > better name? > > The reason I want to use this is because these fields will be representing > a bunch of ODE's at each point (not coupled spatially) and it would be nice > if their mass matrices would be diagonal. So the dual space should be Dirac > delta's at the quadrature points and that's different from the default > (Lagrange). > > I'll see if I can code something up for this. Any suggestions for a name > for this dual space? > > Thanks, > Sander > > On 8 April 2016 at 16:48, Matthew Knepley wrote: > >> On Wed, Apr 6, 2016 at 2:06 PM, Sander Arens >> wrote: >> >>> Hi, >>> >>> I'd like to be able to represent some discretized fields at the >>> quadrature points of a finite element, so I can easily use them with the >>> plex residual/function evaluations. Is it possible to do this with >>> PetscFECreateDefault and some command line options? >>> I think what I need is PETSCSPACEDG for the space, but I'm not sure what >>> type I should take for the dual space? >>> >> >> This is why I put that in there. It really should not be named DG, but >> there are reasons I thought it made sense. >> >> I had not thought about a dual space, since I was not going to project >> into it. I think you can just use whatever is the default. >> >> Matt >> >> >>> Thanks, >>> Sander >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huyaoyu1986 at gmail.com Sat Apr 9 03:34:35 2016 From: huyaoyu1986 at gmail.com (Yaoyu Hu) Date: Sat, 9 Apr 2016 16:34:35 +0800 Subject: [petsc-users] Simple questions about vector scattering Message-ID: Hi everyone, I am trying to write a simple CFD program with PETSc and I have encountered some simple problems with vector scattering. I use finite volume method. When calculating the flux of a scalar variable on a control volume face, the values of the control volumes straddling the face should be obtained. The scalar values are stored in a parallel Vec. I thought that the calculation of the flux values of different faces can be performed, somehow, in parallel. By 'in parallel', I mean let different processors calculate the fluxes of different faces. Because the scalar values are stored in a parallel Vec, I have to do vector scattering to retrieve values that not stored on the local processor. And different processes may require different portions of the parallel Vec. And here comes the problem. It happened that one processor encountered a boundary face, and no vector scattering was needed. And this processor omitted the scattering operation. Then the whole program just halted because, I thought, all other processors were waiting for the processor to trigger a vector scattering. I actually have three questions here: (1) Is it OK for different processors perform vector scattering to retrieve different portions of a parallel Vec in a single scattering operation? (By test I think it is OK?) (2) Should the processor, which is dealing with the boundary face flux, perform a sort of ?dummy scattering? operation? How it is done in the usual way? (3) Am I miss using PETSc to do stuff ?in parallel?? Should I perform the face flux calculation and other tasks alike in serial, and then only use PETSc to solve the discretized equations in parallel? Thanks in advance. HU Yaoyu School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China. From knepley at gmail.com Sat Apr 9 07:02:59 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 9 Apr 2016 07:02:59 -0500 Subject: [petsc-users] Simple questions about vector scattering In-Reply-To: References: Message-ID: On Sat, Apr 9, 2016 at 3:34 AM, Yaoyu Hu wrote: > Hi everyone, > > I am trying to write a simple CFD program with PETSc and I have > encountered some simple problems with vector scattering. > > I use finite volume method. When calculating the flux of a scalar > variable on a control volume face, the values of the control volumes > straddling the face should be obtained. The scalar values are stored > in a parallel Vec. I thought that the calculation of the flux values > of different faces can be performed, somehow, in parallel. By 'in > parallel', I mean let different processors calculate the fluxes of > different faces. Because the scalar values are stored in a parallel > Vec, I have to do vector scattering to retrieve values that not stored > on the local processor. And different processes may require different > portions of the parallel Vec. And here comes the problem. It happened > that one processor encountered a boundary face, and no vector > scattering was needed. And this processor omitted the scattering > operation. Then the whole program just halted because, I thought, all > other processors were waiting for the processor to trigger a vector > scattering. > > I actually have three questions here: > > (1) Is it OK for different processors perform vector scattering to > retrieve different portions of a parallel Vec in a single scattering > operation? (By test I think it is OK?) > Yes. Each process just asks for the values it needs. > (2) Should the processor, which is dealing with the boundary face > flux, perform a sort of ?dummy scattering? operation? How it is done > in the usual way? > Yes. VecScatter is collective, so you need all processes to call it. Some procs may request 0 entries. > (3) Am I miss using PETSc to do stuff ?in parallel?? Should I > perform the face flux calculation and other tasks alike in serial, and > then only use PETSc to solve the discretized equations in parallel? > No, everything can be done in parallel. Thanks, Matt > Thanks in advance. > > HU Yaoyu > School of Mechanical Engineering, > Shanghai Jiao Tong University, Shanghai, China. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 9 11:26:55 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 9 Apr 2016 11:26:55 -0500 Subject: [petsc-users] Discretized fields at quadrature points In-Reply-To: References: Message-ID: On Fri, Apr 8, 2016 at 12:07 PM, Sander Arens wrote: > Or perhaps there should be a flag for PETSCDUALSPACELAGRANGE to evaluate > at the quadrature points? > I think you can get what you want using DUALSPACESIMPLE and pass the quadrature. Matt > On 8 April 2016 at 17:35, Sander Arens wrote: > >> What does it stand for? Discrete Gauss? Maybe PETSCSPACEQUADRATURE is a >> better name? >> >> The reason I want to use this is because these fields will be >> representing a bunch of ODE's at each point (not coupled spatially) and it >> would be nice if their mass matrices would be diagonal. So the dual space >> should be Dirac delta's at the quadrature points and that's different from >> the default (Lagrange). >> >> I'll see if I can code something up for this. Any suggestions for a name >> for this dual space? >> >> Thanks, >> Sander >> >> On 8 April 2016 at 16:48, Matthew Knepley wrote: >> >>> On Wed, Apr 6, 2016 at 2:06 PM, Sander Arens >>> wrote: >>> >>>> Hi, >>>> >>>> I'd like to be able to represent some discretized fields at the >>>> quadrature points of a finite element, so I can easily use them with the >>>> plex residual/function evaluations. Is it possible to do this with >>>> PetscFECreateDefault and some command line options? >>>> I think what I need is PETSCSPACEDG for the space, but I'm not sure >>>> what type I should take for the dual space? >>>> >>> >>> This is why I put that in there. It really should not be named DG, but >>> there are reasons I thought it made sense. >>> >>> I had not thought about a dual space, since I was not going to project >>> into it. I think you can just use whatever is the default. >>> >>> Matt >>> >>> >>>> Thanks, >>>> Sander >>>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From d019598 at polito.it Mon Apr 11 08:16:18 2016 From: d019598 at polito.it (FRANCAVILLA MATTEO ALESSANDRO) Date: Mon, 11 Apr 2016 15:16:18 +0200 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: <878u0owk5r.fsf@jedbrown.org> References: <87shyxzh2c.fsf@jedbrown.org> <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> <871t6hz46p.fsf@jedbrown.org> <000601d1916f$caeb7140$60c253c0$@polito.it> <878u0owk5r.fsf@jedbrown.org> Message-ID: Dear Jed, thanks for the reply. Despite my efforts, I'm still unable to use the information carried by the index set. I'm able to extract the IS through MatNestGetISs but I don't really know how to interprete it and to exploit it to pursue my goal. I don't seem to find a tutorial explaining how to do the following: "MatNestGetISs give you the ISs to extract or set the portions of x and y corresponding to the first and second block. The index sets are not globally contiguous and can't be if you want sane parallelism." Can you please give me some references for the above operation? Thanks, Alessandro On Fri, 08 Apr 2016 09:59:28 +0100 Jed Brown wrote: > Alessandro Francavilla >writes: > >> In fact I don't expect anything special about indices, I >>just can't >> understand how to use them. >> >> If I set a matrix [A B; C D], and each of the blocks is >>distributed over >> some arbitrary number of processes, I want to know how >>to compute the >> product >> y = [A B; C D]*x >> and be able to see the result. I'm not concerned about >>how the elements are >> distributed, I just want to be able to execute the >>matmult operation and >> view the result. > > MatNestGetISs give you the ISs to extract or set the >portions of x and y > corresponding to the first and second block. The index >sets are not > globally contiguous and can't be if you want sane >parallelism. > >>> It is returning the ith column of A. A is not just >>>matArray(1). >> >> No, it is not. I know very well that A is not >>matArray(1), and this >> operation is returning one column of A, but definitely >>not i-th >> column. > > Evidently the ith column is not defined the same way as >your internal > mental model. Use the index sets. > >> I'm definitely sure it is my mistake: 1) building the >>vector x the way >> I am setting it does not actually do what I'm expecting >>it to do; 2) >> even if I fix (1) I still need to permute the result in >>order to >> obtain y as defined by y=[A B;C D]*x . >> >> This is what's going on, which summarizes my 2 mistakes: >>(1) the result >> needs be permuted, and (2) the input vector x is not >>correct >> >> This is what the matrix looks like: >> >> ! A = [ 1 2 | 101 102 ; >> ! 3 4 | 102 103 ; >> ! ___________________ >> ! 201 202 | 401 402 ; >> ! 203 204 | 403 404 ] >> >> >> If I set the x vector with CALL >> VecSetValues(x,1,0,(1.+0.*PETSC_i),INSERT_VALUES,IERR) >> >> The result with 1 MPI process is what I expect (the >>first column of A): >> >> Vec Object: 1 MPI processes >> type: seq >> 1 >> 3 >> 201 >> 203 >> >> With 2 MPI processes the results are distributed: the >>result is correct but >> I need to reorder it to get [1 3 201 203]' but don't >>know how: >> >> Vec Object: 2 MPI processes >> type: mpi >> Process [0] >> 1 >> 201 >> Process [1] >> 3 >> 203 >> >> >> If I set the x vector with CALL >> VecSetValues(x,1,1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) >> >> 1 MPI process correctly yields the 2nd column of A: >> >> Vec Object: 1 MPI processes >> type: seq >> 2 >> 4 >> 202 >> 204 >> >> 2 MPI processes DO NOT YIELD the 2nd column, but the 3rd >>column (apart from >> a permutation similarly to the previous case): >> >> Vec Object: 2 MPI processes >> type: mpi >> Process [0] >> 101 >> 301 >> Process [1] >> 103 >> 303 >> >> >> -----Original Message----- >> From: Jed Brown [mailto:jed at jedbrown.org] >> Sent: gioved? 7 aprile 2016 20:04 >> To: Alessandro Francavilla >> Cc: petsc-users at mcs.anl.gov >> Subject: RE: [petsc-users] Ordering with MatCreateNest >>and MPI >> >> Alessandro Francavilla >>writes: >> >>> Thanks for the reply; probably I've not been clear >>>enough. I don't >>> need to separate out components, I just need to >>>understand the underlying >> orderings. >>> >>> For instance, suppose that I want to multiply the matrix >>>with a null >>> vector with a 1 in the i-th location (the result is >>>basically the i-th >>> column of the matrix): >>> >>> CALL >>> MatCreateNest(PETSC_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT >>> ,matAr >>> ray,A,IERR) >>> CALL MatCreateVecs(A,x,b,IERR) >>> CALL VecSet(x,0.+0.*PETSC_i,IERR) >>> CALL >>>VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR) >>> CALL VecAssemblyBegin(x,IERR) >>> CALL VecAssemblyEnd(x,IERR) >>> CALL MatMult(A,x,b,IERR) >>> >>> The above code works perfectly if I execute it single >>>process (mpirun >>> -np 1); if I run it on >1 MPI processes, the result is >>>not what I would >> expect: >>> rather than returning the i-th column, it returns (e.g., >>>if I call >>> VecView on b) some permutation of a different column >>>(say a >>> permutation of column j). >>> >>> Also, if I substitute the call to MatCreateNest() with A >>>= matArray(1) >> >> But this is a different matrix. Why would you expect >>that the MatNest has >> the same indices as its first Mat? >> >>> Then everything works as expected, for arbitrary numbers >>>of processes. >>> By working as expected I mean that calling VecView on b >>>prints the >>> i-th column of A with its natural ordering. >>> >>> So, my question is: >>> 1) how do I generate the input vector x such that the >>>MatMult >>> operation returns the i-th column of A? >> >> It is returning the ith column of A. A is not just >>matArray(1). >> >> >>> 2) how do I reorder the result of MatMult such that I >>>get the i-th >>> column with its "natural" ordering? >>> >>> Maybe I'm even misunderstanding the output of VecView, >>>which prints >>> the elements process by process, and thus the order >>>corresponds to the >>> natural ordering only if the elements are distributed >>>consecutively among >> processes? >>> Which would explain why I'm seeing a permutation of the >>>column. >>> However, I'm seeing the permutation of a DIFFERENT >>>column of the >>> matrix, which means that the input vector does not have >>>a 1 in the >>> i-th position when I fill it with >> VecSetValues(x,1,i-1,(1.+0.*PETSC_i),INSERT_VALUES,IERR). >> >> PETSc Vecs are distributed with contiguous blocks >>assigned to each process. >> Mat's column and row indices are induced by their action >>on Vecs. MatNest >> does not move matrix data around, so its blocks produce >>the portions of Vecs >> that are owned on the current process. >> Consequently, MatNest induces an ordering in which one >>field gets the first >> chunk of a contiguous space on each process, with the >>intervening entries >> for the other field(s). If you have two matrices, B and >>C, each distributed >> over the same two processes, then the MatNest >> >> [B 0; 0 C] >> >> will keep both B and C distributed. I.e., it does not >>move all of B to rank >> 0 and all of C to rank 1, or some funny business like >>that. If you don't >> want to think about this ordering, extract the subvector >>the way I described >> in the previous email. ___________________________________ Matteo Alessandro Francavilla, PhD Antenna and EMC Lab (LACE) Istituto Superiore Mario Boella (ISMB) Torino, Italy phone +39 011 2276 711 From cnathaide at yahoo.com Mon Apr 11 14:04:16 2016 From: cnathaide at yahoo.com (Christopher Athaide) Date: Mon, 11 Apr 2016 14:04:16 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem Message-ID: <004001d19424$f2117960$d6346c20$@yahoo.com> Hi, I am trying to solve a mixed complementarity problem using the TAO SSILS Solver . C(x) = A x + d , l <= x <= u where A = n x n matrix D = n x 1 vector The Jacobian matrix is equal to the matrix A and is constant at each iteration. I have setup the Jacobian update function PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { AppCtx *data = (AppCtx *)ptr; PetscErrorCode ierr; PetscInt N; PetscInt zero = 0; PetscScalar *d, v; PetscBool assembled; PetscFunctionBegin; ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { ierr = MatZeroEntries(H); CHKERRQ(ierr); } /* Get pointers to vector data */ VecGetSize(X, &N); for (auto i = zero; i != N; ++i) { for (auto j = zero; j != N; ++j) { ierr = MatGetValue(data->A, i, j, &v); ierr = MatSetValue(H, i, j, v, INSERT_VALUES); } } /* Assemble the matrix */ ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, PETSC_TRUE); CHKERRQ(ierr); ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); PetscFunctionReturn(0); } In this case, after iterating over all the row and column indexes and assembling the matrix the function seems to run correctly. But when I try to copy the matrix PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { PetscFunctionReturn(0); AppCtx *data = (AppCtx *)ptr; PetscErrorCode ierr; ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = MatConvert(data->A, MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); /* Assemble the matrix */ ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); PetscFunctionReturn(0); } I get the following error messages. Problem size: 259 rows [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for unassembled matrix [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 MatMult() line 2210 in C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects \petsc-3.6.3\src\tao\util\tao_util.c [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in C:\Apps\Projects \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line 941 in C:\Apps\Projects \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ petsc-3.6.3\src\tao\interface\taosolver.c [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects \complementarity.cpp I am new to PETSC. Can anyone help me? Thanks Chris Athiade -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 11 15:16:00 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 11 Apr 2016 15:16:00 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: <004001d19424$f2117960$d6346c20$@yahoo.com> References: <004001d19424$f2117960$d6346c20$@yahoo.com> Message-ID: > On Apr 11, 2016, at 2:04 PM, Christopher Athaide wrote: > > Hi, > > I am trying to solve a mixed complementarity problem using the TAO SSILS Solver . > > C(x) = A x + d , l <= x <= u > > where A = n x n matrix > D = n x 1 vector > > The Jacobian matrix is equal to the matrix A and is constant at each iteration. > > I have setup the Jacobian update function > > PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) > { > AppCtx *data = (AppCtx *)ptr; > PetscErrorCode ierr; > PetscInt N; > PetscInt zero = 0; > PetscScalar *d, v; > PetscBool assembled; > > PetscFunctionBegin; > ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); > if (assembled) { ierr = MatZeroEntries(H); CHKERRQ(ierr); } > > /* Get pointers to vector data */ > VecGetSize(X, &N); > for (auto i = zero; i != N; ++i) > { > for (auto j = zero; j != N; ++j) > { > ierr = MatGetValue(data->A, i, j, &v); > ierr = MatSetValue(H, i, j, v, INSERT_VALUES); > } > } > > /* Assemble the matrix */ > ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, PETSC_TRUE); CHKERRQ(ierr); > ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > PetscFunctionReturn(0); > > } > > In this case, after iterating over all the row and column indexes and assembling the matrix the function seems to run correctly. > > > But when I try to copy the matrix > > PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) > { > > PetscFunctionReturn(0); > AppCtx *data = (AppCtx *)ptr; > PetscErrorCode ierr; > ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); > if (assembled) { ierr = MatZeroEntries(H); CHKERRQ(ierr); } > ierr = MatConvert(data->A, MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); You cannot do this because you are creating an entirely new H matrix here in the subroutine that TAO won't know about. You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); But since H is constant why have a copy? Just set the values in H initially and don't have a data->A matrix at all. Then your FormJacobian() doesn't need to do anything since H is already set with proper values and H remains the same through all iterations. Barry > > > > /* Assemble the matrix */ > ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > I get the following error messages. > > Problem size: 259 rows > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by Christopher Mon Apr 11 19:57:37 2016 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 --download-f2cblaslapack=1 --with-debugging=1 > [0]PETSC ERROR: #1 MatMult() line 2210 in C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c > [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects \petsc-3.6.3\src\tao\util\tao_util.c > [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in C:\Apps\Projects \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c > [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line 941 in C:\Apps\Projects \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c > [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c > [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ petsc-3.6.3\src\tao\interface\taosolver.c > [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects \complementarity.cpp > > I am new to PETSC. Can anyone help me? > > Thanks > > Chris Athiade From cnathaide at yahoo.com Mon Apr 11 15:59:30 2016 From: cnathaide at yahoo.com (Christopher Athaide) Date: Mon, 11 Apr 2016 15:59:30 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: References: <004001d19424$f2117960$d6346c20$@yahoo.com> Message-ID: <005301d19435$0b1b2180$21516480$@yahoo.com> Hi Barry, Thank you for taking a look at the issue I am having. I followed your suggestion and made the following changes: main() { ierr = MatDuplicate(data.A, MAT_SHARE_NONZERO_PATTERN, &J); CHKERRQ(ierr); ierr = MatCopy(data.A, J, SAME_NONZERO_PATTERN); CHKERRQ(ierr); } PetscErrorCode FormJacobian(Tao, Vec, Mat, Mat, void*) { PetscFunctionBegin; PetscFunctionReturn(0); } But I get a bunch of errors - I am not sure why. Problem size : 259 rows [0]PETSC ERROR : -------------------- - Error Message-------------------------------------------------------------- [0]PETSC ERROR : Argument out of range [0]PETSC ERROR : New nonzero at(0, 0) caused a malloc Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check [0]PETSC ERROR : See http ://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR : Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR : Unknown Name on a arch - mswin - c - debug named LAGRANGE by Christopher Mon Apr 11 21 : 50 : 06 2016 [0]PETSC ERROR : Configure options --with - cc = "win32fe cl" --with - fc = 0 --with - mpi = 0 --download - f2cblaslapack = 1 --with - debugging = 1 [0]PETSC ERROR : #1 MatSetValues_SeqAIJ() line 485 in C : \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC ERROR : #2 MatSetValues() line 1173 in C : \Apps\Projects\petsc - 3.6.3\src\mat\interface\matrix.c [0]PETSC ERROR : #3 MatDiagonalSet_Default() line 198 in C : \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #4 MatDiagonalSet_SeqAIJ() line 194 in C : \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC ERROR : #5 MatDiagonalSet() line 242 in C : \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #6 Tao_SSLS_FunctionGradient() line 66 in C : \Apps\Projects\petsc - 3.6.3\src\tao\complementarity\impls\ssls\ssls.c [0]PETSC ERROR : #7 TaoLineSearchComputeObjectiveAndGradient() line 941 in C : \Apps\Projects\petsc - 3.6.3\src\tao\linesearch\interface\taolinesearch.c [0]PETSC ERROR : #8 TaoSolve_SSILS() line 64 in C : \Apps\Projects\petsc - 3.6.3\src\tao\complementarity\impls\ssls\ssils.c [0]PETSC ERROR : #9 TaoSolve() line 204 in C : \Apps\Projects\petsc - 3.6.3\src\tao\interface\taosolver.c [0]PETSC ERROR : #10 run_complementarity() line 290 in c : \apps\projects\complementarity.cpp I appreciate your help in this matter. Thanks Chris -----Original Message----- From: Barry Smith [mailto:bsmith at mcs.anl.gov] Sent: Monday, April 11, 2016 3:16 PM To: Christopher Athaide Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Problem setting Jacobian matrix in complementarity problem > On Apr 11, 2016, at 2:04 PM, Christopher Athaide wrote: > > Hi, > > I am trying to solve a mixed complementarity problem using the TAO SSILS Solver . > > C(x) = A x + d , l <= x <= u > > where A = n x n matrix > D = n x 1 vector > > The Jacobian matrix is equal to the matrix A and is constant at each iteration. > > I have setup the Jacobian update function > > PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { > AppCtx *data = (AppCtx *)ptr; > PetscErrorCode ierr; > PetscInt N; > PetscInt zero = 0; > PetscScalar *d, v; > PetscBool assembled; > > PetscFunctionBegin; > ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); > if (assembled) { ierr = MatZeroEntries(H); > CHKERRQ(ierr); } > > /* Get pointers to vector data */ > VecGetSize(X, &N); > for (auto i = zero; i != N; ++i) > { > for (auto j = zero; j != N; ++j) > { > ierr = MatGetValue(data->A, i, j, &v); > ierr = MatSetValue(H, i, j, v, INSERT_VALUES); > } > } > > /* Assemble the matrix */ > ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, PETSC_TRUE); CHKERRQ(ierr); > ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > PetscFunctionReturn(0); > > } > > In this case, after iterating over all the row and column indexes and assembling the matrix the function seems to run correctly. > > > But when I try to copy the matrix > > PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { > > PetscFunctionReturn(0); > AppCtx *data = (AppCtx *)ptr; > PetscErrorCode ierr; > ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { > ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = MatConvert(data->A, > MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); You cannot do this because you are creating an entirely new H matrix here in the subroutine that TAO won't know about. You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); But since H is constant why have a copy? Just set the values in H initially and don't have a data->A matrix at all. Then your FormJacobian() doesn't need to do anything since H is already set with proper values and H remains the same through all iterations. Barry > > > > /* Assemble the matrix */ > ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > PetscFunctionReturn(0); } > > I get the following error messages. > > Problem size: 259 rows > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for > unassembled matrix [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC > ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by > Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure options > --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 > --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 > MatMult() line 2210 in > C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c > [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects > \petsc-3.6.3\src\tao\util\tao_util.c > [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in > C:\Apps\Projects > \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c > [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line 941 > in C:\Apps\Projects > \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c > [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects > \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c > [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ > petsc-3.6.3\src\tao\interface\taosolver.c > [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects > \complementarity.cpp > > I am new to PETSC. Can anyone help me? > > Thanks > > Chris Athiade From bsmith at mcs.anl.gov Mon Apr 11 16:04:39 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 11 Apr 2016 16:04:39 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: <005301d19435$0b1b2180$21516480$@yahoo.com> References: <004001d19424$f2117960$d6346c20$@yahoo.com> <005301d19435$0b1b2180$21516480$@yahoo.com> Message-ID: Does your matrix have entries all along the diagonal? If not you need to preallocate for even the 0 locations and set the value of 0 in there specifically. Barry > On Apr 11, 2016, at 3:59 PM, Christopher Athaide wrote: > > Hi Barry, > > Thank you for taking a look at the issue I am having. I followed your > suggestion and made the following changes: > > main() > { > ierr = MatDuplicate(data.A, MAT_SHARE_NONZERO_PATTERN, &J); > CHKERRQ(ierr); > ierr = MatCopy(data.A, J, SAME_NONZERO_PATTERN); CHKERRQ(ierr); > > } > > PetscErrorCode FormJacobian(Tao, Vec, Mat, Mat, void*) > { > > PetscFunctionBegin; > > PetscFunctionReturn(0); > } > > But I get a bunch of errors - I am not sure why. > > Problem size : 259 rows > [0]PETSC ERROR : -------------------- - Error > Message-------------------------------------------------------------- > [0]PETSC ERROR : Argument out of range > [0]PETSC ERROR : New nonzero at(0, 0) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off > this check > [0]PETSC ERROR : See http ://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR : Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR : Unknown Name on a arch - mswin - c - debug named LAGRANGE > by Christopher Mon Apr 11 21 : 50 : 06 2016 > [0]PETSC ERROR : Configure options --with - cc = "win32fe cl" --with - fc = > 0 --with - mpi = 0 --download - f2cblaslapack = 1 --with - debugging = 1 > [0]PETSC ERROR : #1 MatSetValues_SeqAIJ() line 485 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c > [0]PETSC ERROR : #2 MatSetValues() line 1173 in C : \Apps\Projects\petsc - > 3.6.3\src\mat\interface\matrix.c > [0]PETSC ERROR : #3 MatDiagonalSet_Default() line 198 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c > [0]PETSC ERROR : #4 MatDiagonalSet_SeqAIJ() line 194 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c > [0]PETSC ERROR : #5 MatDiagonalSet() line 242 in C : \Apps\Projects\petsc - > 3.6.3\src\mat\utils\axpy.c > [0]PETSC ERROR : #6 Tao_SSLS_FunctionGradient() line 66 in C : > \Apps\Projects\petsc - 3.6.3\src\tao\complementarity\impls\ssls\ssls.c > [0]PETSC ERROR : #7 TaoLineSearchComputeObjectiveAndGradient() line 941 in C > : \Apps\Projects\petsc - 3.6.3\src\tao\linesearch\interface\taolinesearch.c > [0]PETSC ERROR : #8 TaoSolve_SSILS() line 64 in C : \Apps\Projects\petsc - > 3.6.3\src\tao\complementarity\impls\ssls\ssils.c > [0]PETSC ERROR : #9 TaoSolve() line 204 in C : \Apps\Projects\petsc - > 3.6.3\src\tao\interface\taosolver.c > [0]PETSC ERROR : #10 run_complementarity() line 290 in c : > \apps\projects\complementarity.cpp > > I appreciate your help in this matter. > > Thanks > > Chris > > -----Original Message----- > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > Sent: Monday, April 11, 2016 3:16 PM > To: Christopher Athaide > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Problem setting Jacobian matrix in > complementarity problem > > >> On Apr 11, 2016, at 2:04 PM, Christopher Athaide > wrote: >> >> Hi, >> >> I am trying to solve a mixed complementarity problem using the TAO SSILS > Solver . >> >> C(x) = A x + d , l <= x <= u >> >> where A = n x n matrix >> D = n x 1 vector >> >> The Jacobian matrix is equal to the matrix A and is constant at each > iteration. >> >> I have setup the Jacobian update function >> >> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >> AppCtx *data = (AppCtx *)ptr; >> PetscErrorCode ierr; >> PetscInt N; >> PetscInt zero = 0; >> PetscScalar *d, v; >> PetscBool assembled; >> >> PetscFunctionBegin; >> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); >> if (assembled) { ierr = MatZeroEntries(H); >> CHKERRQ(ierr); } >> >> /* Get pointers to vector data */ >> VecGetSize(X, &N); >> for (auto i = zero; i != N; ++i) >> { >> for (auto j = zero; j != N; ++j) >> { >> ierr = > MatGetValue(data->A, i, j, &v); >> ierr = MatSetValue(H, i, > j, v, INSERT_VALUES); >> } >> } >> >> /* Assemble the matrix */ >> ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, > PETSC_TRUE); CHKERRQ(ierr); >> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> PetscFunctionReturn(0); >> >> } >> >> In this case, after iterating over all the row and column indexes and > assembling the matrix the function seems to run correctly. >> >> >> But when I try to copy the matrix >> >> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >> >> PetscFunctionReturn(0); >> AppCtx *data = (AppCtx *)ptr; >> PetscErrorCode ierr; >> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { >> ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = MatConvert(data->A, >> MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); > > You cannot do this because you are creating an entirely new H matrix here > in the subroutine that TAO won't know about. > > You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); > > But since H is constant why have a copy? Just set the values in H > initially and don't have a data->A matrix at all. Then your FormJacobian() > doesn't need to do anything since H is already set with proper values and H > remains the same through all iterations. > > > > Barry > > >> >> >> >> /* Assemble the matrix */ >> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> PetscFunctionReturn(0); } >> >> I get the following error messages. >> >> Problem size: 259 rows >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for >> unassembled matrix [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >> ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by >> Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure options >> --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 >> --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 >> MatMult() line 2210 in >> C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c >> [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects >> \petsc-3.6.3\src\tao\util\tao_util.c >> [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in >> C:\Apps\Projects >> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c >> [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line 941 >> in C:\Apps\Projects >> \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c >> [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects >> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c >> [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ >> petsc-3.6.3\src\tao\interface\taosolver.c >> [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects >> \complementarity.cpp >> >> I am new to PETSC. Can anyone help me? >> >> Thanks >> >> Chris Athiade > From eugenio.aulisa at ttu.edu Tue Apr 12 08:39:33 2016 From: eugenio.aulisa at ttu.edu (Aulisa, Eugenio) Date: Tue, 12 Apr 2016 13:39:33 +0000 Subject: [petsc-users] Understanding the -ksp_monitor_true_residual output Message-ID: Hi, I am trying to understand better the meaning of the output obtained using the option -ksp_monitor_true_residual For this particular ksp gmres solver I used KSPSetTolerances( ksp, 1.0e-4, 1.0e-20, 1.0e50, 60); Below I am reporting the output for 5 different solutions In each of them the convergence reason is 2, the relative tolerance. If I well understood the manual I would expect the solver to exit when the output of the third column ||r(i)||/||b|| would fall below 1.0e-04, but it is not. Only in the last solution this is the case. So probably I did not understood well the manual, and there is a rescaling involved. Can anybody clarify? Thanks, Eugenio Aulisa ********* solution 1 ********* 0 KSP preconditioned resid norm 3.410562043865e+01 true resid norm 2.677893657847e-02 ||r(i)||/||b|| 3.993927934126e+00 1 KSP preconditioned resid norm 5.824255715066e+00 true resid norm 1.393228808527e-02 ||r(i)||/||b|| 2.077922489826e+00 2 KSP preconditioned resid norm 2.044834259922e+00 true resid norm 9.998998076659e-03 ||r(i)||/||b|| 1.491294384100e+00 3 KSP preconditioned resid norm 7.983786718994e-01 true resid norm 7.993995927780e-03 ||r(i)||/||b|| 1.192259578632e+00 4 KSP preconditioned resid norm 5.088731995172e-01 true resid norm 8.593150583543e-03 ||r(i)||/||b|| 1.281620129208e+00 5 KSP preconditioned resid norm 4.857006661570e-01 true resid norm 9.145941228163e-03 ||r(i)||/||b|| 1.364065748018e+00 6 KSP preconditioned resid norm 4.709624056986e-01 true resid norm 8.186977687270e-03 ||r(i)||/||b|| 1.221041723798e+00 7 KSP preconditioned resid norm 3.220045108449e-01 true resid norm 4.352636436074e-03 ||r(i)||/||b|| 6.491712692992e-01 8 KSP preconditioned resid norm 2.317017366518e-01 true resid norm 3.068984741076e-03 ||r(i)||/||b|| 4.577218311441e-01 9 KSP preconditioned resid norm 1.910797150631e-01 true resid norm 2.564718555990e-03 ||r(i)||/||b|| 3.825133628412e-01 10 KSP preconditioned resid norm 1.729613253747e-01 true resid norm 2.555884137267e-03 ||r(i)||/||b|| 3.811957589244e-01 11 KSP preconditioned resid norm 1.708323617518e-01 true resid norm 2.699641560429e-03 ||r(i)||/||b|| 4.026363708929e-01 12 KSP preconditioned resid norm 1.606003434286e-01 true resid norm 2.148175491821e-03 ||r(i)||/||b|| 3.203883051534e-01 13 KSP preconditioned resid norm 1.205953154320e-01 true resid norm 1.319733567910e-03 ||r(i)||/||b|| 1.968308467752e-01 14 KSP preconditioned resid norm 6.473668392252e-02 true resid norm 7.750007977497e-04 ||r(i)||/||b|| 1.155870146685e-01 15 KSP preconditioned resid norm 4.210976512647e-02 true resid norm 3.627137178803e-04 ||r(i)||/||b|| 5.409671312704e-02 16 KSP preconditioned resid norm 2.167079981806e-02 true resid norm 1.885937330161e-04 ||r(i)||/||b|| 2.812769567181e-02 *************** MG linear solver time: 1.710646e+01 *************** Number of outer ksp solver iterations = 16 *************** Convergence reason = 2 *************** Residual norm = 0.0216708 ********* solution 2 ********* ********* Level Max 4 MG PROJECTION MATRICES TIME: 1.199155 ********* Level Max 4 MGINIT TIME: 0.691589 ********* Level Max 4 ASSEMBLY TIME: 11.470600 ********* Linear Cycle + Residual Update iteration 1 *************** Linear iteration 1 *********** 0 KSP preconditioned resid norm 5.264510580035e+01 true resid norm 4.446470526095e-03 ||r(i)||/||b|| 1.343097318343e+00 1 KSP preconditioned resid norm 1.194632734776e+00 true resid norm 2.650838633150e-03 ||r(i)||/||b|| 8.007101899472e-01 2 KSP preconditioned resid norm 2.514505382950e-01 true resid norm 8.961867887367e-04 ||r(i)||/||b|| 2.707014621199e-01 3 KSP preconditioned resid norm 4.126701684642e-02 true resid norm 2.828561525607e-04 ||r(i)||/||b|| 8.543930242012e-02 4 KSP preconditioned resid norm 2.990078801994e-02 true resid norm 3.012123557665e-04 ||r(i)||/||b|| 9.098396242766e-02 5 KSP preconditioned resid norm 2.498986435717e-02 true resid norm 3.492907889867e-04 ||r(i)||/||b|| 1.055064953780e-01 6 KSP preconditioned resid norm 2.212280334220e-02 true resid norm 2.649097730890e-04 ||r(i)||/||b|| 8.001843344081e-02 7 KSP preconditioned resid norm 1.270611663385e-02 true resid norm 1.114378752133e-04 ||r(i)||/||b|| 3.366083514611e-02 *************** MG linear solver time: 1.032488e+01 *************** Number of outer ksp solver iterations = 7 *************** Convergence reason = 2 *************** Residual norm = 0.012706117 ********* solution 3 ********* 0 KSP preconditioned resid norm 8.243929775966e-02 true resid norm 1.154408051359e-04 ||r(i)||/||b|| 1.020198211331e+00 1 KSP preconditioned resid norm 9.646734506593e-03 true resid norm 1.275532346161e-04 ||r(i)||/||b|| 1.127240767696e+00 2 KSP preconditioned resid norm 9.036645252734e-03 true resid norm 1.308042358485e-04 ||r(i)||/||b|| 1.155971212173e+00 3 KSP preconditioned resid norm 9.031542604401e-03 true resid norm 1.291290364679e-04 ||r(i)||/||b|| 1.141166781368e+00 4 KSP preconditioned resid norm 5.171448589573e-03 true resid norm 8.206502754373e-05 ||r(i)||/||b|| 7.252426402812e-01 5 KSP preconditioned resid norm 2.891929663032e-03 true resid norm 7.598041857685e-05 ||r(i)||/||b|| 6.714704305557e-01 6 KSP preconditioned resid norm 2.343001795062e-03 true resid norm 7.025844324392e-05 ||r(i)||/||b|| 6.209029644586e-01 7 KSP preconditioned resid norm 2.185509183830e-03 true resid norm 6.344926688255e-05 ||r(i)||/||b|| 5.607274525472e-01 8 KSP preconditioned resid norm 2.058441165493e-03 true resid norm 5.444619590411e-05 ||r(i)||/||b|| 4.811635851791e-01 9 KSP preconditioned resid norm 1.932842861175e-03 true resid norm 4.852637232344e-05 ||r(i)||/||b|| 4.288476521667e-01 10 KSP preconditioned resid norm 1.898825969952e-03 true resid norm 4.918129150547e-05 ||r(i)||/||b|| 4.346354442501e-01 11 KSP preconditioned resid norm 1.891602735935e-03 true resid norm 4.710536993558e-05 ||r(i)||/||b|| 4.162896654766e-01 12 KSP preconditioned resid norm 1.781187329134e-03 true resid norm 3.777014072473e-05 ||r(i)||/||b|| 3.337903782266e-01 13 KSP preconditioned resid norm 1.099066136430e-03 true resid norm 1.424597383325e-05 ||r(i)||/||b|| 1.258975715410e-01 14 KSP preconditioned resid norm 6.280243703281e-04 true resid norm 5.012017417525e-06 ||r(i)||/||b|| 4.429327392944e-02 15 KSP preconditioned resid norm 3.076374051756e-04 true resid norm 2.605020117635e-06 ||r(i)||/||b|| 2.302164179611e-02 16 KSP preconditioned resid norm 1.368303935548e-04 true resid norm 1.393123702916e-06 ||r(i)||/||b|| 1.231161120372e-02 17 KSP preconditioned resid norm 7.228043655761e-05 true resid norm 7.895171494272e-07 ||r(i)||/||b|| 6.977290072715e-03 18 KSP preconditioned resid norm 3.866647775994e-05 true resid norm 4.528577419728e-07 ||r(i)||/||b|| 4.002091442486e-03 *************** MG linear solver time: 1.798489e+01 *************** Number of outer ksp solver iterations = 18 *************** Convergence reason = 2 *************** Residual norm = 3.8666478e-05 ********* solution 4 ********* 0 KSP preconditioned resid norm 1.580191013568e-04 true resid norm 4.329063484560e-07 ||r(i)||/||b|| 9.556132759754e-01 1 KSP preconditioned resid norm 3.193492170557e-05 true resid norm 3.833183963598e-07 ||r(i)||/||b|| 8.461510204077e-01 2 KSP preconditioned resid norm 1.771193813393e-05 true resid norm 3.779057891921e-07 ||r(i)||/||b|| 8.342030337691e-01 3 KSP preconditioned resid norm 1.457271790528e-05 true resid norm 3.908277498762e-07 ||r(i)||/||b|| 8.627274414739e-01 4 KSP preconditioned resid norm 1.450745561248e-05 true resid norm 3.973502432864e-07 ||r(i)||/||b|| 8.771254315185e-01 5 KSP preconditioned resid norm 1.413655961039e-05 true resid norm 3.690661275865e-07 ||r(i)||/||b|| 8.146900420664e-01 6 KSP preconditioned resid norm 1.250367472770e-05 true resid norm 2.896966438369e-07 ||r(i)||/||b|| 6.394869464109e-01 7 KSP preconditioned resid norm 8.752862209538e-06 true resid norm 1.583120202644e-07 ||r(i)||/||b|| 3.494637324001e-01 8 KSP preconditioned resid norm 5.705999073439e-06 true resid norm 9.765760786693e-08 ||r(i)||/||b|| 2.155729684041e-01 9 KSP preconditioned resid norm 4.192178998873e-06 true resid norm 8.201685117361e-08 ||r(i)||/||b|| 1.810469911442e-01 10 KSP preconditioned resid norm 3.787794183624e-06 true resid norm 8.005205894911e-08 ||r(i)||/||b|| 1.767098370670e-01 11 KSP preconditioned resid norm 3.786646339736e-06 true resid norm 8.038622787957e-08 ||r(i)||/||b|| 1.774474937623e-01 12 KSP preconditioned resid norm 3.485518782082e-06 true resid norm 6.939667236683e-08 ||r(i)||/||b|| 1.531887477714e-01 13 KSP preconditioned resid norm 2.521214592414e-06 true resid norm 4.378155291157e-08 ||r(i)||/||b|| 9.664499805635e-02 14 KSP preconditioned resid norm 1.284994591466e-06 true resid norm 2.240582241084e-08 ||r(i)||/||b|| 4.945943027010e-02 15 KSP preconditioned resid norm 6.133028069300e-07 true resid norm 1.361309969809e-08 ||r(i)||/||b|| 3.005005319295e-02 16 KSP preconditioned resid norm 3.455523081821e-07 true resid norm 9.658743967776e-09 ||r(i)||/||b|| 2.132106400789e-02 17 KSP preconditioned resid norm 2.505109883585e-07 true resid norm 6.870409301721e-09 ||r(i)||/||b|| 1.516599228338e-02 18 KSP preconditioned resid norm 1.443203299992e-07 true resid norm 2.920265170395e-09 ||r(i)||/||b|| 6.446299935659e-03 19 KSP preconditioned resid norm 6.636071800703e-08 true resid norm 8.697314690164e-10 ||r(i)||/||b|| 1.919876992541e-03 20 KSP preconditioned resid norm 2.714859523602e-08 true resid norm 3.148984178618e-10 ||r(i)||/||b|| 6.951182623347e-04 21 KSP preconditioned resid norm 1.117207550910e-08 true resid norm 9.396722225928e-11 ||r(i)||/||b|| 2.074266765035e-04 *************** MG linear solver time: 2.005832e+01 *************** Number of outer ksp solver iterations = 21 *************** Convergence reason = 2 *************** Residual norm = 1.1172076e-08 ********* solution 5 ********* 0 KSP preconditioned resid norm 8.454401619840e-09 true resid norm 8.452738510071e-11 ||r(i)||/||b|| 8.995416179570e-01 1 KSP preconditioned resid norm 4.580659647172e-09 true resid norm 8.265563180408e-11 ||r(i)||/||b|| 8.796223931180e-01 2 KSP preconditioned resid norm 4.256081515265e-09 true resid norm 8.549582845321e-11 ||r(i)||/||b|| 9.098478056991e-01 3 KSP preconditioned resid norm 3.976359790239e-09 true resid norm 8.007959832870e-11 ||r(i)||/||b|| 8.522082087375e-01 4 KSP preconditioned resid norm 2.913165841066e-09 true resid norm 6.211100493580e-11 ||r(i)||/||b|| 6.609861857942e-01 5 KSP preconditioned resid norm 2.130900661500e-09 true resid norm 4.739159015704e-11 ||r(i)||/||b|| 5.043419672409e-01 6 KSP preconditioned resid norm 1.606110624311e-09 true resid norm 3.793019756253e-11 ||r(i)||/||b|| 4.036536945295e-01 7 KSP preconditioned resid norm 1.194061891991e-09 true resid norm 3.460284393549e-11 ||r(i)||/||b|| 3.682439505559e-01 8 KSP preconditioned resid norm 1.116001295092e-09 true resid norm 3.338332136751e-11 ||r(i)||/||b|| 3.552657742805e-01 9 KSP preconditioned resid norm 1.101694508395e-09 true resid norm 3.242096748193e-11 ||r(i)||/||b|| 3.450243907306e-01 10 KSP preconditioned resid norm 1.101137265122e-09 true resid norm 3.225308396705e-11 ||r(i)||/||b|| 3.432377720103e-01 11 KSP preconditioned resid norm 1.100600624436e-09 true resid norm 3.226535049788e-11 ||r(i)||/||b|| 3.433683126034e-01 12 KSP preconditioned resid norm 1.092907627060e-09 true resid norm 3.134764252264e-11 ||r(i)||/||b|| 3.336020514577e-01 13 KSP preconditioned resid norm 9.870087224964e-10 true resid norm 2.277199172927e-11 ||r(i)||/||b|| 2.423398554190e-01 14 KSP preconditioned resid norm 6.855374335133e-10 true resid norm 8.872613630455e-12 ||r(i)||/||b|| 9.442247871666e-02 15 KSP preconditioned resid norm 3.960223027036e-10 true resid norm 3.394176790592e-12 ||r(i)||/||b|| 3.612087701758e-02 16 KSP preconditioned resid norm 1.977336550448e-10 true resid norm 1.661499967555e-12 ||r(i)||/||b|| 1.768170596156e-02 17 KSP preconditioned resid norm 8.560605344251e-11 true resid norm 7.411831253461e-13 ||r(i)||/||b|| 7.887681216944e-03 18 KSP preconditioned resid norm 3.819157888961e-11 true resid norm 4.556531922343e-13 ||r(i)||/||b|| 4.849067663472e-03 19 KSP preconditioned resid norm 2.031569059250e-11 true resid norm 2.030960472074e-13 ||r(i)||/||b|| 2.161350983329e-03 20 KSP preconditioned resid norm 9.966436828718e-12 true resid norm 9.192887548370e-14 ||r(i)||/||b|| 9.783083824378e-04 21 KSP preconditioned resid norm 4.416012066006e-12 true resid norm 4.607062767741e-14 ||r(i)||/||b|| 4.902842660027e-04 22 KSP preconditioned resid norm 1.740999775043e-12 true resid norm 1.452842765632e-14 ||r(i)||/||b|| 1.546117309173e-04 23 KSP preconditioned resid norm 6.762885582183e-13 true resid norm 3.253813922193e-15 ||r(i)||/||b|| 3.462713340314e-05 *************** MG linear solver time: 2.151817e+01 *************** Number of outer ksp solver iterations = 23 *************** Convergence reason = 2 *************** Residual norm = 6.7628856e-13 Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 From cnathaide at yahoo.com Tue Apr 12 09:32:02 2016 From: cnathaide at yahoo.com (Christopher Athaide) Date: Tue, 12 Apr 2016 09:32:02 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: References: <004001d19424$f2117960$d6346c20$@yahoo.com> <005301d19435$0b1b2180$21516480$@yahoo.com> Message-ID: <004201d194c8$15871780$40954680$@yahoo.com> Barry, Thank you for your suggestions. I implemented the changes that you suggested. int main() { PetscScalar v; ierr = MatCreateSeqAIJ(PETSC_COMM_SELF, N, N, 5 * N, NULL, &J); CHKERRQ(ierr); for (auto i = 0; i != N; ++i) { for (auto j = 0; j != N; ++j) { ierr = MatGetValue(data.A, i, j, &v); CHKERRQ(ierr); ierr = MatSetValue(J, i, j, v, INSERT_VALUES); CHKERRQ(ierr); } } /* Assemble the matrix */ ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); compare_matrices(data.A, J); } PetscErrorCode FormJacobian(Tao, Vec, Mat H, Mat, void* ptr) { PetscFunctionBegin; // AppCtx *data = (AppCtx *)ptr; // Is there any way I can do a copy data->A to H using a matrix or a batch operation? PetscFunctionReturn(0); } The good news is that the model runs without generating any errors. The bad news is that the program terminates without the optimality conditions being satisfied. In the original program, the model takes about 70 seconds to run and the optimality conditions are satisfied. In this version, it takes about 5 seconds. Is there any way I can copy the matrix data->A to H in the FormJacobian using a matrix function instead of iterating through all the rows and columns and then assembling the matrix at every iteration. Thank you for your help. Chris -----Original Message----- From: Barry Smith [mailto:bsmith at mcs.anl.gov] Sent: Monday, April 11, 2016 4:05 PM To: Christopher Athaide Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Problem setting Jacobian matrix in complementarity problem Does your matrix have entries all along the diagonal? If not you need to preallocate for even the 0 locations and set the value of 0 in there specifically. Barry > On Apr 11, 2016, at 3:59 PM, Christopher Athaide wrote: > > Hi Barry, > > Thank you for taking a look at the issue I am having. I followed your > suggestion and made the following changes: > > main() > { > ierr = MatDuplicate(data.A, MAT_SHARE_NONZERO_PATTERN, &J); > CHKERRQ(ierr); > ierr = MatCopy(data.A, J, SAME_NONZERO_PATTERN); CHKERRQ(ierr); > > } > > PetscErrorCode FormJacobian(Tao, Vec, Mat, Mat, void*) { > > PetscFunctionBegin; > > PetscFunctionReturn(0); > } > > But I get a bunch of errors - I am not sure why. > > Problem size : 259 rows > [0]PETSC ERROR : -------------------- - Error > Message-------------------------------------------------------------- > [0]PETSC ERROR : Argument out of range [0]PETSC ERROR : New nonzero > at(0, 0) caused a malloc Use MatSetOption(A, > MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [0]PETSC ERROR : See http > ://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR : Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC > ERROR : Unknown Name on a arch - mswin - c - debug named LAGRANGE by > Christopher Mon Apr 11 21 : 50 : 06 2016 [0]PETSC ERROR : Configure > options --with - cc = "win32fe cl" --with - fc = > 0 --with - mpi = 0 --download - f2cblaslapack = 1 --with - debugging = > 1 [0]PETSC ERROR : #1 MatSetValues_SeqAIJ() line 485 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC > ERROR : #2 MatSetValues() line 1173 in C : \Apps\Projects\petsc - > 3.6.3\src\mat\interface\matrix.c [0]PETSC ERROR : #3 > MatDiagonalSet_Default() line 198 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #4 > MatDiagonalSet_SeqAIJ() line 194 in C : > \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC > ERROR : #5 MatDiagonalSet() line 242 in C : \Apps\Projects\petsc - > 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #6 > Tao_SSLS_FunctionGradient() line 66 in C : > \Apps\Projects\petsc - 3.6.3\src\tao\complementarity\impls\ssls\ssls.c > [0]PETSC ERROR : #7 TaoLineSearchComputeObjectiveAndGradient() line > 941 in C > : \Apps\Projects\petsc - > 3.6.3\src\tao\linesearch\interface\taolinesearch.c > [0]PETSC ERROR : #8 TaoSolve_SSILS() line 64 in C : > \Apps\Projects\petsc - > 3.6.3\src\tao\complementarity\impls\ssls\ssils.c > [0]PETSC ERROR : #9 TaoSolve() line 204 in C : \Apps\Projects\petsc - > 3.6.3\src\tao\interface\taosolver.c > [0]PETSC ERROR : #10 run_complementarity() line 290 in c : > \apps\projects\complementarity.cpp > > I appreciate your help in this matter. > > Thanks > > Chris > > -----Original Message----- > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > Sent: Monday, April 11, 2016 3:16 PM > To: Christopher Athaide > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Problem setting Jacobian matrix in > complementarity problem > > >> On Apr 11, 2016, at 2:04 PM, Christopher Athaide >> > wrote: >> >> Hi, >> >> I am trying to solve a mixed complementarity problem using the TAO >> SSILS > Solver . >> >> C(x) = A x + d , l <= x <= u >> >> where A = n x n matrix >> D = n x 1 vector >> >> The Jacobian matrix is equal to the matrix A and is constant at each > iteration. >> >> I have setup the Jacobian update function >> >> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >> AppCtx *data = (AppCtx *)ptr; >> PetscErrorCode ierr; >> PetscInt N; >> PetscInt zero = 0; >> PetscScalar *d, v; >> PetscBool assembled; >> >> PetscFunctionBegin; >> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); >> if (assembled) { ierr = MatZeroEntries(H); >> CHKERRQ(ierr); } >> >> /* Get pointers to vector data */ >> VecGetSize(X, &N); >> for (auto i = zero; i != N; ++i) >> { >> for (auto j = zero; j != N; ++j) >> { >> ierr = > MatGetValue(data->A, i, j, &v); >> ierr = MatSetValue(H, >> i, > j, v, INSERT_VALUES); >> } >> } >> >> /* Assemble the matrix */ >> ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, > PETSC_TRUE); CHKERRQ(ierr); >> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> PetscFunctionReturn(0); >> >> } >> >> In this case, after iterating over all the row and column indexes and > assembling the matrix the function seems to run correctly. >> >> >> But when I try to copy the matrix >> >> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >> >> PetscFunctionReturn(0); >> AppCtx *data = (AppCtx *)ptr; >> PetscErrorCode ierr; >> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { >> ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = MatConvert(data->A, >> MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); > > You cannot do this because you are creating an entirely new H matrix > here in the subroutine that TAO won't know about. > > You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); > > But since H is constant why have a copy? Just set the values in H > initially and don't have a data->A matrix at all. Then your > FormJacobian() doesn't need to do anything since H is already set with > proper values and H remains the same through all iterations. > > > > Barry > > >> >> >> >> /* Assemble the matrix */ >> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); > CHKERRQ(ierr); >> PetscFunctionReturn(0); } >> >> I get the following error messages. >> >> Problem size: 259 rows >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for >> unassembled matrix [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >> ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by >> Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure >> options --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 >> --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 >> MatMult() line 2210 in >> C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c >> [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects >> \petsc-3.6.3\src\tao\util\tao_util.c >> [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in >> C:\Apps\Projects >> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c >> [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line >> 941 in C:\Apps\Projects >> \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c >> [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects >> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c >> [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ >> petsc-3.6.3\src\tao\interface\taosolver.c >> [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects >> \complementarity.cpp >> >> I am new to PETSC. Can anyone help me? >> >> Thanks >> >> Chris Athiade > From natacha.bereux at gmail.com Tue Apr 12 10:22:11 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Tue, 12 Apr 2016 17:22:11 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory Message-ID: Hello, I am trying to compile Petsc (3.6.3) with external packages (MUMPS and its prerequisites). More precisely I would like PETSc to use a pre-installed version of MUMPS. Petsc downloads and compiles the prerequisites (parmetis, scalapack etc) : this works fine. I define MUMPS location by --with-mumps-dir=top-directory of MUMPS install, but the configure step fails with the followiing message: UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work I do not understand what is wrong. I have attached the configure.log file. Any hint would be greatly appreciated ! Best regards, Natacha -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2024806 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Apr 12 11:19:20 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 12 Apr 2016 11:19:20 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: On Tue, 12 Apr 2016, Natacha BEREUX wrote: > Hello, > I am trying to compile Petsc (3.6.3) with external packages (MUMPS and its > prerequisites). > More precisely I would like PETSc to use a pre-installed version of MUMPS. > > Petsc downloads and compiles the prerequisites (parmetis, scalapack etc) : > this works fine. What metis/parmetis/slcalapack is this MUMPS installed with? What version of MUMPS did you install? Why could you not use --download-mumps? Using a different metis/parmetis/slcalapack to install MUMPS - and then specifying --download-metis --download-parmetis --download-scalapack [i.e different versions/builds of the same libraries] can result in conflcits. > > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS install, > but the > configure step fails with the followiing message: > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > I do not understand what is wrong. > I have attached the configure.log file. > > Any hint would be greatly appreciated ! >>>> Executing: mpicc -show stdout: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc Defined make macro "MPICC_SHOW" to "gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" <<<< Ok - so you are using system openmpi with gcc. >>>> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O /tmp/petsc-0u_4WI/config.libraries/conftest.o -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lscalapack -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lparmetis -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lmetis -lm -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl Possible ERROR while running linker: exit code 256 stderr: /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': dlr_stats.F:(.text+0x3079): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x30fa): undefined reference to `GOMP_critical_name_end' dlr_stats.F:(.text+0x310e): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x318f): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': dlr_stats.F:(.text+0x33a9): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x33f9): undefined reference to `GOMP_critical_name_end' dlr_stats.F:(.text+0x340a): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x345a): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_panel': dlr_stats.F:(.text+0x3576): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x35a7): undefined reference to `GOMP_critical_name_end' dlr_stats.F:(.text+0x35b8): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x35e9): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_demote': dlr_stats.F:(.text+0x36ac): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x36ce): undefined reference to `GOMP_critical_name_end' dlr_stats.F:(.text+0x36df): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x3701): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': dlr_stats.F:(.text+0x37c1): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x37e3): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': dlr_stats.F:(.text+0x3839): undefined reference to `GOMP_critical_name_start' dlr_stats.F:(.text+0x3856): undefined reference to `GOMP_critical_name_end' /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In function `__dmumps_ana_lr_MOD_mumps_scotch_kway': dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' Looks like this MUMPS is built with ptscotch and openmp. You can specify -lgomp for openmp. This can be done with the configure option LIBS=-lgomp Wrt and PTSCOTCH stuff depending upon how its installed we'll have to figureout how to specify it. Its best to specify all the packages [mumps and its dependencies] you built manually with the options: --with-package-include --with-package-lib Satish > Best regards, > > Natacha > From dave.mayhem23 at gmail.com Tue Apr 12 12:21:14 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 12 Apr 2016 19:21:14 +0200 Subject: [petsc-users] Understanding the -ksp_monitor_true_residual output In-Reply-To: References: Message-ID: On 12 April 2016 at 15:39, Aulisa, Eugenio wrote: > Hi, > > I am trying to understand better the meaning of > the output obtained using the option > > -ksp_monitor_true_residual > > For this particular ksp gmres solver I used > > KSPSetTolerances( ksp, 1.0e-4, 1.0e-20, 1.0e50, 60); > > Below I am reporting the output for 5 different solutions > > In each of them the convergence reason is 2, the relative tolerance. > > If I well understood the manual I would expect the solver to exit when the > output of the third column ||r(i)||/||b|| would fall below 1.0e-04, but it > is not. > By default, GMRES uses left preconditioning, thus iterations are terminated when || P^{-1} r_i || / || P^{-1} b || < 1e-4 With GMRES, you can change the side of the preconditioning via -ksp_pc_side right (note that not all methods support both left and right preconditioning) FGMRES and GCR use right preconditioning by default Thanks, Dave Only in the last solution this is the case. > So probably I did not understood well the manual, and there is a rescaling > involved. > > Can anybody clarify? > > Thanks, > Eugenio Aulisa > > > > ********* solution 1 ********* > 0 KSP preconditioned resid norm 3.410562043865e+01 true resid norm > 2.677893657847e-02 ||r(i)||/||b|| 3.993927934126e+00 > 1 KSP preconditioned resid norm 5.824255715066e+00 true resid norm > 1.393228808527e-02 ||r(i)||/||b|| 2.077922489826e+00 > 2 KSP preconditioned resid norm 2.044834259922e+00 true resid norm > 9.998998076659e-03 ||r(i)||/||b|| 1.491294384100e+00 > 3 KSP preconditioned resid norm 7.983786718994e-01 true resid norm > 7.993995927780e-03 ||r(i)||/||b|| 1.192259578632e+00 > 4 KSP preconditioned resid norm 5.088731995172e-01 true resid norm > 8.593150583543e-03 ||r(i)||/||b|| 1.281620129208e+00 > 5 KSP preconditioned resid norm 4.857006661570e-01 true resid norm > 9.145941228163e-03 ||r(i)||/||b|| 1.364065748018e+00 > 6 KSP preconditioned resid norm 4.709624056986e-01 true resid norm > 8.186977687270e-03 ||r(i)||/||b|| 1.221041723798e+00 > 7 KSP preconditioned resid norm 3.220045108449e-01 true resid norm > 4.352636436074e-03 ||r(i)||/||b|| 6.491712692992e-01 > 8 KSP preconditioned resid norm 2.317017366518e-01 true resid norm > 3.068984741076e-03 ||r(i)||/||b|| 4.577218311441e-01 > 9 KSP preconditioned resid norm 1.910797150631e-01 true resid norm > 2.564718555990e-03 ||r(i)||/||b|| 3.825133628412e-01 > 10 KSP preconditioned resid norm 1.729613253747e-01 true resid norm > 2.555884137267e-03 ||r(i)||/||b|| 3.811957589244e-01 > 11 KSP preconditioned resid norm 1.708323617518e-01 true resid norm > 2.699641560429e-03 ||r(i)||/||b|| 4.026363708929e-01 > 12 KSP preconditioned resid norm 1.606003434286e-01 true resid norm > 2.148175491821e-03 ||r(i)||/||b|| 3.203883051534e-01 > 13 KSP preconditioned resid norm 1.205953154320e-01 true resid norm > 1.319733567910e-03 ||r(i)||/||b|| 1.968308467752e-01 > 14 KSP preconditioned resid norm 6.473668392252e-02 true resid norm > 7.750007977497e-04 ||r(i)||/||b|| 1.155870146685e-01 > 15 KSP preconditioned resid norm 4.210976512647e-02 true resid norm > 3.627137178803e-04 ||r(i)||/||b|| 5.409671312704e-02 > 16 KSP preconditioned resid norm 2.167079981806e-02 true resid norm > 1.885937330161e-04 ||r(i)||/||b|| 2.812769567181e-02 > *************** MG linear solver time: 1.710646e+01 > *************** Number of outer ksp solver iterations = 16 > *************** Convergence reason = 2 > *************** Residual norm = 0.0216708 > > > > > > ********* solution 2 ********* > ********* Level Max 4 MG PROJECTION MATRICES TIME: 1.199155 > ********* Level Max 4 MGINIT TIME: 0.691589 > ********* Level Max 4 ASSEMBLY TIME: 11.470600 > ********* Linear Cycle + Residual Update iteration 1 > *************** Linear iteration 1 *********** > 0 KSP preconditioned resid norm 5.264510580035e+01 true resid norm > 4.446470526095e-03 ||r(i)||/||b|| 1.343097318343e+00 > 1 KSP preconditioned resid norm 1.194632734776e+00 true resid norm > 2.650838633150e-03 ||r(i)||/||b|| 8.007101899472e-01 > 2 KSP preconditioned resid norm 2.514505382950e-01 true resid norm > 8.961867887367e-04 ||r(i)||/||b|| 2.707014621199e-01 > 3 KSP preconditioned resid norm 4.126701684642e-02 true resid norm > 2.828561525607e-04 ||r(i)||/||b|| 8.543930242012e-02 > 4 KSP preconditioned resid norm 2.990078801994e-02 true resid norm > 3.012123557665e-04 ||r(i)||/||b|| 9.098396242766e-02 > 5 KSP preconditioned resid norm 2.498986435717e-02 true resid norm > 3.492907889867e-04 ||r(i)||/||b|| 1.055064953780e-01 > 6 KSP preconditioned resid norm 2.212280334220e-02 true resid norm > 2.649097730890e-04 ||r(i)||/||b|| 8.001843344081e-02 > 7 KSP preconditioned resid norm 1.270611663385e-02 true resid norm > 1.114378752133e-04 ||r(i)||/||b|| 3.366083514611e-02 > *************** MG linear solver time: 1.032488e+01 > *************** Number of outer ksp solver iterations = 7 > *************** Convergence reason = 2 > *************** Residual norm = 0.012706117 > > > > > ********* solution 3 ********* > 0 KSP preconditioned resid norm 8.243929775966e-02 true resid norm > 1.154408051359e-04 ||r(i)||/||b|| 1.020198211331e+00 > 1 KSP preconditioned resid norm 9.646734506593e-03 true resid norm > 1.275532346161e-04 ||r(i)||/||b|| 1.127240767696e+00 > 2 KSP preconditioned resid norm 9.036645252734e-03 true resid norm > 1.308042358485e-04 ||r(i)||/||b|| 1.155971212173e+00 > 3 KSP preconditioned resid norm 9.031542604401e-03 true resid norm > 1.291290364679e-04 ||r(i)||/||b|| 1.141166781368e+00 > 4 KSP preconditioned resid norm 5.171448589573e-03 true resid norm > 8.206502754373e-05 ||r(i)||/||b|| 7.252426402812e-01 > 5 KSP preconditioned resid norm 2.891929663032e-03 true resid norm > 7.598041857685e-05 ||r(i)||/||b|| 6.714704305557e-01 > 6 KSP preconditioned resid norm 2.343001795062e-03 true resid norm > 7.025844324392e-05 ||r(i)||/||b|| 6.209029644586e-01 > 7 KSP preconditioned resid norm 2.185509183830e-03 true resid norm > 6.344926688255e-05 ||r(i)||/||b|| 5.607274525472e-01 > 8 KSP preconditioned resid norm 2.058441165493e-03 true resid norm > 5.444619590411e-05 ||r(i)||/||b|| 4.811635851791e-01 > 9 KSP preconditioned resid norm 1.932842861175e-03 true resid norm > 4.852637232344e-05 ||r(i)||/||b|| 4.288476521667e-01 > 10 KSP preconditioned resid norm 1.898825969952e-03 true resid norm > 4.918129150547e-05 ||r(i)||/||b|| 4.346354442501e-01 > 11 KSP preconditioned resid norm 1.891602735935e-03 true resid norm > 4.710536993558e-05 ||r(i)||/||b|| 4.162896654766e-01 > 12 KSP preconditioned resid norm 1.781187329134e-03 true resid norm > 3.777014072473e-05 ||r(i)||/||b|| 3.337903782266e-01 > 13 KSP preconditioned resid norm 1.099066136430e-03 true resid norm > 1.424597383325e-05 ||r(i)||/||b|| 1.258975715410e-01 > 14 KSP preconditioned resid norm 6.280243703281e-04 true resid norm > 5.012017417525e-06 ||r(i)||/||b|| 4.429327392944e-02 > 15 KSP preconditioned resid norm 3.076374051756e-04 true resid norm > 2.605020117635e-06 ||r(i)||/||b|| 2.302164179611e-02 > 16 KSP preconditioned resid norm 1.368303935548e-04 true resid norm > 1.393123702916e-06 ||r(i)||/||b|| 1.231161120372e-02 > 17 KSP preconditioned resid norm 7.228043655761e-05 true resid norm > 7.895171494272e-07 ||r(i)||/||b|| 6.977290072715e-03 > 18 KSP preconditioned resid norm 3.866647775994e-05 true resid norm > 4.528577419728e-07 ||r(i)||/||b|| 4.002091442486e-03 > *************** MG linear solver time: 1.798489e+01 > *************** Number of outer ksp solver iterations = 18 > *************** Convergence reason = 2 > *************** Residual norm = 3.8666478e-05 > > > > > ********* solution 4 ********* > 0 KSP preconditioned resid norm 1.580191013568e-04 true resid norm > 4.329063484560e-07 ||r(i)||/||b|| 9.556132759754e-01 > 1 KSP preconditioned resid norm 3.193492170557e-05 true resid norm > 3.833183963598e-07 ||r(i)||/||b|| 8.461510204077e-01 > 2 KSP preconditioned resid norm 1.771193813393e-05 true resid norm > 3.779057891921e-07 ||r(i)||/||b|| 8.342030337691e-01 > 3 KSP preconditioned resid norm 1.457271790528e-05 true resid norm > 3.908277498762e-07 ||r(i)||/||b|| 8.627274414739e-01 > 4 KSP preconditioned resid norm 1.450745561248e-05 true resid norm > 3.973502432864e-07 ||r(i)||/||b|| 8.771254315185e-01 > 5 KSP preconditioned resid norm 1.413655961039e-05 true resid norm > 3.690661275865e-07 ||r(i)||/||b|| 8.146900420664e-01 > 6 KSP preconditioned resid norm 1.250367472770e-05 true resid norm > 2.896966438369e-07 ||r(i)||/||b|| 6.394869464109e-01 > 7 KSP preconditioned resid norm 8.752862209538e-06 true resid norm > 1.583120202644e-07 ||r(i)||/||b|| 3.494637324001e-01 > 8 KSP preconditioned resid norm 5.705999073439e-06 true resid norm > 9.765760786693e-08 ||r(i)||/||b|| 2.155729684041e-01 > 9 KSP preconditioned resid norm 4.192178998873e-06 true resid norm > 8.201685117361e-08 ||r(i)||/||b|| 1.810469911442e-01 > 10 KSP preconditioned resid norm 3.787794183624e-06 true resid norm > 8.005205894911e-08 ||r(i)||/||b|| 1.767098370670e-01 > 11 KSP preconditioned resid norm 3.786646339736e-06 true resid norm > 8.038622787957e-08 ||r(i)||/||b|| 1.774474937623e-01 > 12 KSP preconditioned resid norm 3.485518782082e-06 true resid norm > 6.939667236683e-08 ||r(i)||/||b|| 1.531887477714e-01 > 13 KSP preconditioned resid norm 2.521214592414e-06 true resid norm > 4.378155291157e-08 ||r(i)||/||b|| 9.664499805635e-02 > 14 KSP preconditioned resid norm 1.284994591466e-06 true resid norm > 2.240582241084e-08 ||r(i)||/||b|| 4.945943027010e-02 > 15 KSP preconditioned resid norm 6.133028069300e-07 true resid norm > 1.361309969809e-08 ||r(i)||/||b|| 3.005005319295e-02 > 16 KSP preconditioned resid norm 3.455523081821e-07 true resid norm > 9.658743967776e-09 ||r(i)||/||b|| 2.132106400789e-02 > 17 KSP preconditioned resid norm 2.505109883585e-07 true resid norm > 6.870409301721e-09 ||r(i)||/||b|| 1.516599228338e-02 > 18 KSP preconditioned resid norm 1.443203299992e-07 true resid norm > 2.920265170395e-09 ||r(i)||/||b|| 6.446299935659e-03 > 19 KSP preconditioned resid norm 6.636071800703e-08 true resid norm > 8.697314690164e-10 ||r(i)||/||b|| 1.919876992541e-03 > 20 KSP preconditioned resid norm 2.714859523602e-08 true resid norm > 3.148984178618e-10 ||r(i)||/||b|| 6.951182623347e-04 > 21 KSP preconditioned resid norm 1.117207550910e-08 true resid norm > 9.396722225928e-11 ||r(i)||/||b|| 2.074266765035e-04 > *************** MG linear solver time: 2.005832e+01 > *************** Number of outer ksp solver iterations = 21 > *************** Convergence reason = 2 > *************** Residual norm = 1.1172076e-08 > > > > > ********* solution 5 ********* > 0 KSP preconditioned resid norm 8.454401619840e-09 true resid norm > 8.452738510071e-11 ||r(i)||/||b|| 8.995416179570e-01 > 1 KSP preconditioned resid norm 4.580659647172e-09 true resid norm > 8.265563180408e-11 ||r(i)||/||b|| 8.796223931180e-01 > 2 KSP preconditioned resid norm 4.256081515265e-09 true resid norm > 8.549582845321e-11 ||r(i)||/||b|| 9.098478056991e-01 > 3 KSP preconditioned resid norm 3.976359790239e-09 true resid norm > 8.007959832870e-11 ||r(i)||/||b|| 8.522082087375e-01 > 4 KSP preconditioned resid norm 2.913165841066e-09 true resid norm > 6.211100493580e-11 ||r(i)||/||b|| 6.609861857942e-01 > 5 KSP preconditioned resid norm 2.130900661500e-09 true resid norm > 4.739159015704e-11 ||r(i)||/||b|| 5.043419672409e-01 > 6 KSP preconditioned resid norm 1.606110624311e-09 true resid norm > 3.793019756253e-11 ||r(i)||/||b|| 4.036536945295e-01 > 7 KSP preconditioned resid norm 1.194061891991e-09 true resid norm > 3.460284393549e-11 ||r(i)||/||b|| 3.682439505559e-01 > 8 KSP preconditioned resid norm 1.116001295092e-09 true resid norm > 3.338332136751e-11 ||r(i)||/||b|| 3.552657742805e-01 > 9 KSP preconditioned resid norm 1.101694508395e-09 true resid norm > 3.242096748193e-11 ||r(i)||/||b|| 3.450243907306e-01 > 10 KSP preconditioned resid norm 1.101137265122e-09 true resid norm > 3.225308396705e-11 ||r(i)||/||b|| 3.432377720103e-01 > 11 KSP preconditioned resid norm 1.100600624436e-09 true resid norm > 3.226535049788e-11 ||r(i)||/||b|| 3.433683126034e-01 > 12 KSP preconditioned resid norm 1.092907627060e-09 true resid norm > 3.134764252264e-11 ||r(i)||/||b|| 3.336020514577e-01 > 13 KSP preconditioned resid norm 9.870087224964e-10 true resid norm > 2.277199172927e-11 ||r(i)||/||b|| 2.423398554190e-01 > 14 KSP preconditioned resid norm 6.855374335133e-10 true resid norm > 8.872613630455e-12 ||r(i)||/||b|| 9.442247871666e-02 > 15 KSP preconditioned resid norm 3.960223027036e-10 true resid norm > 3.394176790592e-12 ||r(i)||/||b|| 3.612087701758e-02 > 16 KSP preconditioned resid norm 1.977336550448e-10 true resid norm > 1.661499967555e-12 ||r(i)||/||b|| 1.768170596156e-02 > 17 KSP preconditioned resid norm 8.560605344251e-11 true resid norm > 7.411831253461e-13 ||r(i)||/||b|| 7.887681216944e-03 > 18 KSP preconditioned resid norm 3.819157888961e-11 true resid norm > 4.556531922343e-13 ||r(i)||/||b|| 4.849067663472e-03 > 19 KSP preconditioned resid norm 2.031569059250e-11 true resid norm > 2.030960472074e-13 ||r(i)||/||b|| 2.161350983329e-03 > 20 KSP preconditioned resid norm 9.966436828718e-12 true resid norm > 9.192887548370e-14 ||r(i)||/||b|| 9.783083824378e-04 > 21 KSP preconditioned resid norm 4.416012066006e-12 true resid norm > 4.607062767741e-14 ||r(i)||/||b|| 4.902842660027e-04 > 22 KSP preconditioned resid norm 1.740999775043e-12 true resid norm > 1.452842765632e-14 ||r(i)||/||b|| 1.546117309173e-04 > 23 KSP preconditioned resid norm 6.762885582183e-13 true resid norm > 3.253813922193e-15 ||r(i)||/||b|| 3.462713340314e-05 > *************** MG linear solver time: 2.151817e+01 > *************** Number of outer ksp solver iterations = 23 > *************** Convergence reason = 2 > *************** Residual norm = 6.7628856e-13 > > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugenio.aulisa at ttu.edu Tue Apr 12 12:45:17 2016 From: eugenio.aulisa at ttu.edu (Aulisa, Eugenio) Date: Tue, 12 Apr 2016 17:45:17 +0000 Subject: [petsc-users] Understanding the -ksp_monitor_true_residual output In-Reply-To: References: , Message-ID: Thanks Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Tuesday, April 12, 2016 12:21 PM To: Aulisa, Eugenio Cc: PETSc users list ?[petsc-users at mcs.anl.gov]? Subject: Re: [petsc-users] Understanding the -ksp_monitor_true_residual output On 12 April 2016 at 15:39, Aulisa, Eugenio > wrote: Hi, I am trying to understand better the meaning of the output obtained using the option -ksp_monitor_true_residual For this particular ksp gmres solver I used KSPSetTolerances( ksp, 1.0e-4, 1.0e-20, 1.0e50, 60); Below I am reporting the output for 5 different solutions In each of them the convergence reason is 2, the relative tolerance. If I well understood the manual I would expect the solver to exit when the output of the third column ||r(i)||/||b|| would fall below 1.0e-04, but it is not. By default, GMRES uses left preconditioning, thus iterations are terminated when || P^{-1} r_i || / || P^{-1} b || < 1e-4 With GMRES, you can change the side of the preconditioning via -ksp_pc_side right (note that not all methods support both left and right preconditioning) FGMRES and GCR use right preconditioning by default Thanks, Dave Only in the last solution this is the case. So probably I did not understood well the manual, and there is a rescaling involved. Can anybody clarify? Thanks, Eugenio Aulisa ********* solution 1 ********* 0 KSP preconditioned resid norm 3.410562043865e+01 true resid norm 2.677893657847e-02 ||r(i)||/||b|| 3.993927934126e+00 1 KSP preconditioned resid norm 5.824255715066e+00 true resid norm 1.393228808527e-02 ||r(i)||/||b|| 2.077922489826e+00 2 KSP preconditioned resid norm 2.044834259922e+00 true resid norm 9.998998076659e-03 ||r(i)||/||b|| 1.491294384100e+00 3 KSP preconditioned resid norm 7.983786718994e-01 true resid norm 7.993995927780e-03 ||r(i)||/||b|| 1.192259578632e+00 4 KSP preconditioned resid norm 5.088731995172e-01 true resid norm 8.593150583543e-03 ||r(i)||/||b|| 1.281620129208e+00 5 KSP preconditioned resid norm 4.857006661570e-01 true resid norm 9.145941228163e-03 ||r(i)||/||b|| 1.364065748018e+00 6 KSP preconditioned resid norm 4.709624056986e-01 true resid norm 8.186977687270e-03 ||r(i)||/||b|| 1.221041723798e+00 7 KSP preconditioned resid norm 3.220045108449e-01 true resid norm 4.352636436074e-03 ||r(i)||/||b|| 6.491712692992e-01 8 KSP preconditioned resid norm 2.317017366518e-01 true resid norm 3.068984741076e-03 ||r(i)||/||b|| 4.577218311441e-01 9 KSP preconditioned resid norm 1.910797150631e-01 true resid norm 2.564718555990e-03 ||r(i)||/||b|| 3.825133628412e-01 10 KSP preconditioned resid norm 1.729613253747e-01 true resid norm 2.555884137267e-03 ||r(i)||/||b|| 3.811957589244e-01 11 KSP preconditioned resid norm 1.708323617518e-01 true resid norm 2.699641560429e-03 ||r(i)||/||b|| 4.026363708929e-01 12 KSP preconditioned resid norm 1.606003434286e-01 true resid norm 2.148175491821e-03 ||r(i)||/||b|| 3.203883051534e-01 13 KSP preconditioned resid norm 1.205953154320e-01 true resid norm 1.319733567910e-03 ||r(i)||/||b|| 1.968308467752e-01 14 KSP preconditioned resid norm 6.473668392252e-02 true resid norm 7.750007977497e-04 ||r(i)||/||b|| 1.155870146685e-01 15 KSP preconditioned resid norm 4.210976512647e-02 true resid norm 3.627137178803e-04 ||r(i)||/||b|| 5.409671312704e-02 16 KSP preconditioned resid norm 2.167079981806e-02 true resid norm 1.885937330161e-04 ||r(i)||/||b|| 2.812769567181e-02 *************** MG linear solver time: 1.710646e+01 *************** Number of outer ksp solver iterations = 16 *************** Convergence reason = 2 *************** Residual norm = 0.0216708 ********* solution 2 ********* ********* Level Max 4 MG PROJECTION MATRICES TIME: 1.199155 ********* Level Max 4 MGINIT TIME: 0.691589 ********* Level Max 4 ASSEMBLY TIME: 11.470600 ********* Linear Cycle + Residual Update iteration 1 *************** Linear iteration 1 *********** 0 KSP preconditioned resid norm 5.264510580035e+01 true resid norm 4.446470526095e-03 ||r(i)||/||b|| 1.343097318343e+00 1 KSP preconditioned resid norm 1.194632734776e+00 true resid norm 2.650838633150e-03 ||r(i)||/||b|| 8.007101899472e-01 2 KSP preconditioned resid norm 2.514505382950e-01 true resid norm 8.961867887367e-04 ||r(i)||/||b|| 2.707014621199e-01 3 KSP preconditioned resid norm 4.126701684642e-02 true resid norm 2.828561525607e-04 ||r(i)||/||b|| 8.543930242012e-02 4 KSP preconditioned resid norm 2.990078801994e-02 true resid norm 3.012123557665e-04 ||r(i)||/||b|| 9.098396242766e-02 5 KSP preconditioned resid norm 2.498986435717e-02 true resid norm 3.492907889867e-04 ||r(i)||/||b|| 1.055064953780e-01 6 KSP preconditioned resid norm 2.212280334220e-02 true resid norm 2.649097730890e-04 ||r(i)||/||b|| 8.001843344081e-02 7 KSP preconditioned resid norm 1.270611663385e-02 true resid norm 1.114378752133e-04 ||r(i)||/||b|| 3.366083514611e-02 *************** MG linear solver time: 1.032488e+01 *************** Number of outer ksp solver iterations = 7 *************** Convergence reason = 2 *************** Residual norm = 0.012706117 ********* solution 3 ********* 0 KSP preconditioned resid norm 8.243929775966e-02 true resid norm 1.154408051359e-04 ||r(i)||/||b|| 1.020198211331e+00 1 KSP preconditioned resid norm 9.646734506593e-03 true resid norm 1.275532346161e-04 ||r(i)||/||b|| 1.127240767696e+00 2 KSP preconditioned resid norm 9.036645252734e-03 true resid norm 1.308042358485e-04 ||r(i)||/||b|| 1.155971212173e+00 3 KSP preconditioned resid norm 9.031542604401e-03 true resid norm 1.291290364679e-04 ||r(i)||/||b|| 1.141166781368e+00 4 KSP preconditioned resid norm 5.171448589573e-03 true resid norm 8.206502754373e-05 ||r(i)||/||b|| 7.252426402812e-01 5 KSP preconditioned resid norm 2.891929663032e-03 true resid norm 7.598041857685e-05 ||r(i)||/||b|| 6.714704305557e-01 6 KSP preconditioned resid norm 2.343001795062e-03 true resid norm 7.025844324392e-05 ||r(i)||/||b|| 6.209029644586e-01 7 KSP preconditioned resid norm 2.185509183830e-03 true resid norm 6.344926688255e-05 ||r(i)||/||b|| 5.607274525472e-01 8 KSP preconditioned resid norm 2.058441165493e-03 true resid norm 5.444619590411e-05 ||r(i)||/||b|| 4.811635851791e-01 9 KSP preconditioned resid norm 1.932842861175e-03 true resid norm 4.852637232344e-05 ||r(i)||/||b|| 4.288476521667e-01 10 KSP preconditioned resid norm 1.898825969952e-03 true resid norm 4.918129150547e-05 ||r(i)||/||b|| 4.346354442501e-01 11 KSP preconditioned resid norm 1.891602735935e-03 true resid norm 4.710536993558e-05 ||r(i)||/||b|| 4.162896654766e-01 12 KSP preconditioned resid norm 1.781187329134e-03 true resid norm 3.777014072473e-05 ||r(i)||/||b|| 3.337903782266e-01 13 KSP preconditioned resid norm 1.099066136430e-03 true resid norm 1.424597383325e-05 ||r(i)||/||b|| 1.258975715410e-01 14 KSP preconditioned resid norm 6.280243703281e-04 true resid norm 5.012017417525e-06 ||r(i)||/||b|| 4.429327392944e-02 15 KSP preconditioned resid norm 3.076374051756e-04 true resid norm 2.605020117635e-06 ||r(i)||/||b|| 2.302164179611e-02 16 KSP preconditioned resid norm 1.368303935548e-04 true resid norm 1.393123702916e-06 ||r(i)||/||b|| 1.231161120372e-02 17 KSP preconditioned resid norm 7.228043655761e-05 true resid norm 7.895171494272e-07 ||r(i)||/||b|| 6.977290072715e-03 18 KSP preconditioned resid norm 3.866647775994e-05 true resid norm 4.528577419728e-07 ||r(i)||/||b|| 4.002091442486e-03 *************** MG linear solver time: 1.798489e+01 *************** Number of outer ksp solver iterations = 18 *************** Convergence reason = 2 *************** Residual norm = 3.8666478e-05 ********* solution 4 ********* 0 KSP preconditioned resid norm 1.580191013568e-04 true resid norm 4.329063484560e-07 ||r(i)||/||b|| 9.556132759754e-01 1 KSP preconditioned resid norm 3.193492170557e-05 true resid norm 3.833183963598e-07 ||r(i)||/||b|| 8.461510204077e-01 2 KSP preconditioned resid norm 1.771193813393e-05 true resid norm 3.779057891921e-07 ||r(i)||/||b|| 8.342030337691e-01 3 KSP preconditioned resid norm 1.457271790528e-05 true resid norm 3.908277498762e-07 ||r(i)||/||b|| 8.627274414739e-01 4 KSP preconditioned resid norm 1.450745561248e-05 true resid norm 3.973502432864e-07 ||r(i)||/||b|| 8.771254315185e-01 5 KSP preconditioned resid norm 1.413655961039e-05 true resid norm 3.690661275865e-07 ||r(i)||/||b|| 8.146900420664e-01 6 KSP preconditioned resid norm 1.250367472770e-05 true resid norm 2.896966438369e-07 ||r(i)||/||b|| 6.394869464109e-01 7 KSP preconditioned resid norm 8.752862209538e-06 true resid norm 1.583120202644e-07 ||r(i)||/||b|| 3.494637324001e-01 8 KSP preconditioned resid norm 5.705999073439e-06 true resid norm 9.765760786693e-08 ||r(i)||/||b|| 2.155729684041e-01 9 KSP preconditioned resid norm 4.192178998873e-06 true resid norm 8.201685117361e-08 ||r(i)||/||b|| 1.810469911442e-01 10 KSP preconditioned resid norm 3.787794183624e-06 true resid norm 8.005205894911e-08 ||r(i)||/||b|| 1.767098370670e-01 11 KSP preconditioned resid norm 3.786646339736e-06 true resid norm 8.038622787957e-08 ||r(i)||/||b|| 1.774474937623e-01 12 KSP preconditioned resid norm 3.485518782082e-06 true resid norm 6.939667236683e-08 ||r(i)||/||b|| 1.531887477714e-01 13 KSP preconditioned resid norm 2.521214592414e-06 true resid norm 4.378155291157e-08 ||r(i)||/||b|| 9.664499805635e-02 14 KSP preconditioned resid norm 1.284994591466e-06 true resid norm 2.240582241084e-08 ||r(i)||/||b|| 4.945943027010e-02 15 KSP preconditioned resid norm 6.133028069300e-07 true resid norm 1.361309969809e-08 ||r(i)||/||b|| 3.005005319295e-02 16 KSP preconditioned resid norm 3.455523081821e-07 true resid norm 9.658743967776e-09 ||r(i)||/||b|| 2.132106400789e-02 17 KSP preconditioned resid norm 2.505109883585e-07 true resid norm 6.870409301721e-09 ||r(i)||/||b|| 1.516599228338e-02 18 KSP preconditioned resid norm 1.443203299992e-07 true resid norm 2.920265170395e-09 ||r(i)||/||b|| 6.446299935659e-03 19 KSP preconditioned resid norm 6.636071800703e-08 true resid norm 8.697314690164e-10 ||r(i)||/||b|| 1.919876992541e-03 20 KSP preconditioned resid norm 2.714859523602e-08 true resid norm 3.148984178618e-10 ||r(i)||/||b|| 6.951182623347e-04 21 KSP preconditioned resid norm 1.117207550910e-08 true resid norm 9.396722225928e-11 ||r(i)||/||b|| 2.074266765035e-04 *************** MG linear solver time: 2.005832e+01 *************** Number of outer ksp solver iterations = 21 *************** Convergence reason = 2 *************** Residual norm = 1.1172076e-08 ********* solution 5 ********* 0 KSP preconditioned resid norm 8.454401619840e-09 true resid norm 8.452738510071e-11 ||r(i)||/||b|| 8.995416179570e-01 1 KSP preconditioned resid norm 4.580659647172e-09 true resid norm 8.265563180408e-11 ||r(i)||/||b|| 8.796223931180e-01 2 KSP preconditioned resid norm 4.256081515265e-09 true resid norm 8.549582845321e-11 ||r(i)||/||b|| 9.098478056991e-01 3 KSP preconditioned resid norm 3.976359790239e-09 true resid norm 8.007959832870e-11 ||r(i)||/||b|| 8.522082087375e-01 4 KSP preconditioned resid norm 2.913165841066e-09 true resid norm 6.211100493580e-11 ||r(i)||/||b|| 6.609861857942e-01 5 KSP preconditioned resid norm 2.130900661500e-09 true resid norm 4.739159015704e-11 ||r(i)||/||b|| 5.043419672409e-01 6 KSP preconditioned resid norm 1.606110624311e-09 true resid norm 3.793019756253e-11 ||r(i)||/||b|| 4.036536945295e-01 7 KSP preconditioned resid norm 1.194061891991e-09 true resid norm 3.460284393549e-11 ||r(i)||/||b|| 3.682439505559e-01 8 KSP preconditioned resid norm 1.116001295092e-09 true resid norm 3.338332136751e-11 ||r(i)||/||b|| 3.552657742805e-01 9 KSP preconditioned resid norm 1.101694508395e-09 true resid norm 3.242096748193e-11 ||r(i)||/||b|| 3.450243907306e-01 10 KSP preconditioned resid norm 1.101137265122e-09 true resid norm 3.225308396705e-11 ||r(i)||/||b|| 3.432377720103e-01 11 KSP preconditioned resid norm 1.100600624436e-09 true resid norm 3.226535049788e-11 ||r(i)||/||b|| 3.433683126034e-01 12 KSP preconditioned resid norm 1.092907627060e-09 true resid norm 3.134764252264e-11 ||r(i)||/||b|| 3.336020514577e-01 13 KSP preconditioned resid norm 9.870087224964e-10 true resid norm 2.277199172927e-11 ||r(i)||/||b|| 2.423398554190e-01 14 KSP preconditioned resid norm 6.855374335133e-10 true resid norm 8.872613630455e-12 ||r(i)||/||b|| 9.442247871666e-02 15 KSP preconditioned resid norm 3.960223027036e-10 true resid norm 3.394176790592e-12 ||r(i)||/||b|| 3.612087701758e-02 16 KSP preconditioned resid norm 1.977336550448e-10 true resid norm 1.661499967555e-12 ||r(i)||/||b|| 1.768170596156e-02 17 KSP preconditioned resid norm 8.560605344251e-11 true resid norm 7.411831253461e-13 ||r(i)||/||b|| 7.887681216944e-03 18 KSP preconditioned resid norm 3.819157888961e-11 true resid norm 4.556531922343e-13 ||r(i)||/||b|| 4.849067663472e-03 19 KSP preconditioned resid norm 2.031569059250e-11 true resid norm 2.030960472074e-13 ||r(i)||/||b|| 2.161350983329e-03 20 KSP preconditioned resid norm 9.966436828718e-12 true resid norm 9.192887548370e-14 ||r(i)||/||b|| 9.783083824378e-04 21 KSP preconditioned resid norm 4.416012066006e-12 true resid norm 4.607062767741e-14 ||r(i)||/||b|| 4.902842660027e-04 22 KSP preconditioned resid norm 1.740999775043e-12 true resid norm 1.452842765632e-14 ||r(i)||/||b|| 1.546117309173e-04 23 KSP preconditioned resid norm 6.762885582183e-13 true resid norm 3.253813922193e-15 ||r(i)||/||b|| 3.462713340314e-05 *************** MG linear solver time: 2.151817e+01 *************** Number of outer ksp solver iterations = 23 *************** Convergence reason = 2 *************** Residual norm = 6.7628856e-13 Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 12 13:07:50 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Apr 2016 13:07:50 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: <004201d194c8$15871780$40954680$@yahoo.com> References: <004001d19424$f2117960$d6346c20$@yahoo.com> <005301d19435$0b1b2180$21516480$@yahoo.com> <004201d194c8$15871780$40954680$@yahoo.com> Message-ID: <7A9CBB55-5B66-427D-839E-9CD321EBAC64@mcs.anl.gov> Yes call the MatCopy() inside the FormJacobian > On Apr 12, 2016, at 9:32 AM, Christopher Athaide wrote: > > Barry, > > Thank you for your suggestions. I implemented the changes that you > suggested. > > int main() > { > PetscScalar v; > ierr = MatCreateSeqAIJ(PETSC_COMM_SELF, N, N, 5 * N, NULL, &J); > CHKERRQ(ierr); > for (auto i = 0; i != N; ++i) > { > for (auto j = 0; j != N; ++j) > { > ierr = MatGetValue(data.A, i, j, &v); CHKERRQ(ierr); > ierr = MatSetValue(J, i, j, v, INSERT_VALUES); > CHKERRQ(ierr); > } > } > > /* Assemble the matrix */ > ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > compare_matrices(data.A, J); > > } > > PetscErrorCode FormJacobian(Tao, Vec, Mat H, Mat, void* ptr) > { > > PetscFunctionBegin; > > // AppCtx *data = (AppCtx *)ptr; > // Is there any way I can do a copy data->A to H using a matrix or a > batch operation? > > PetscFunctionReturn(0); > } > > > The good news is that the model runs without generating any errors. The bad > news is that the program terminates without the optimality conditions being > satisfied. > > In the original program, the model takes about 70 seconds to run and the > optimality conditions are satisfied. In this version, it takes about 5 > seconds. > > Is there any way I can copy the matrix data->A to H in the FormJacobian > using a matrix function instead of iterating through all the rows and > columns and then assembling the matrix at every iteration. > > Thank you for your help. > > Chris > > -----Original Message----- > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > Sent: Monday, April 11, 2016 4:05 PM > To: Christopher Athaide > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Problem setting Jacobian matrix in > complementarity problem > > > Does your matrix have entries all along the diagonal? If not you need to > preallocate for even the 0 locations and set the value of 0 in there > specifically. > > Barry > >> On Apr 11, 2016, at 3:59 PM, Christopher Athaide > wrote: >> >> Hi Barry, >> >> Thank you for taking a look at the issue I am having. I followed your >> suggestion and made the following changes: >> >> main() >> { >> ierr = MatDuplicate(data.A, MAT_SHARE_NONZERO_PATTERN, &J); >> CHKERRQ(ierr); >> ierr = MatCopy(data.A, J, SAME_NONZERO_PATTERN); CHKERRQ(ierr); >> >> } >> >> PetscErrorCode FormJacobian(Tao, Vec, Mat, Mat, void*) { >> >> PetscFunctionBegin; >> >> PetscFunctionReturn(0); >> } >> >> But I get a bunch of errors - I am not sure why. >> >> Problem size : 259 rows >> [0]PETSC ERROR : -------------------- - Error >> Message-------------------------------------------------------------- >> [0]PETSC ERROR : Argument out of range [0]PETSC ERROR : New nonzero >> at(0, 0) caused a malloc Use MatSetOption(A, >> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >> [0]PETSC ERROR : See http >> ://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR : Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >> ERROR : Unknown Name on a arch - mswin - c - debug named LAGRANGE by >> Christopher Mon Apr 11 21 : 50 : 06 2016 [0]PETSC ERROR : Configure >> options --with - cc = "win32fe cl" --with - fc = >> 0 --with - mpi = 0 --download - f2cblaslapack = 1 --with - debugging = >> 1 [0]PETSC ERROR : #1 MatSetValues_SeqAIJ() line 485 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC >> ERROR : #2 MatSetValues() line 1173 in C : \Apps\Projects\petsc - >> 3.6.3\src\mat\interface\matrix.c [0]PETSC ERROR : #3 >> MatDiagonalSet_Default() line 198 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #4 >> MatDiagonalSet_SeqAIJ() line 194 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC >> ERROR : #5 MatDiagonalSet() line 242 in C : \Apps\Projects\petsc - >> 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #6 >> Tao_SSLS_FunctionGradient() line 66 in C : >> \Apps\Projects\petsc - 3.6.3\src\tao\complementarity\impls\ssls\ssls.c >> [0]PETSC ERROR : #7 TaoLineSearchComputeObjectiveAndGradient() line >> 941 in C >> : \Apps\Projects\petsc - >> 3.6.3\src\tao\linesearch\interface\taolinesearch.c >> [0]PETSC ERROR : #8 TaoSolve_SSILS() line 64 in C : >> \Apps\Projects\petsc - >> 3.6.3\src\tao\complementarity\impls\ssls\ssils.c >> [0]PETSC ERROR : #9 TaoSolve() line 204 in C : \Apps\Projects\petsc - >> 3.6.3\src\tao\interface\taosolver.c >> [0]PETSC ERROR : #10 run_complementarity() line 290 in c : >> \apps\projects\complementarity.cpp >> >> I appreciate your help in this matter. >> >> Thanks >> >> Chris >> >> -----Original Message----- >> From: Barry Smith [mailto:bsmith at mcs.anl.gov] >> Sent: Monday, April 11, 2016 3:16 PM >> To: Christopher Athaide >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] Problem setting Jacobian matrix in >> complementarity problem >> >> >>> On Apr 11, 2016, at 2:04 PM, Christopher Athaide >>> >> wrote: >>> >>> Hi, >>> >>> I am trying to solve a mixed complementarity problem using the TAO >>> SSILS >> Solver . >>> >>> C(x) = A x + d , l <= x <= u >>> >>> where A = n x n matrix >>> D = n x 1 vector >>> >>> The Jacobian matrix is equal to the matrix A and is constant at each >> iteration. >>> >>> I have setup the Jacobian update function >>> >>> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >>> AppCtx *data = (AppCtx *)ptr; >>> PetscErrorCode ierr; >>> PetscInt N; >>> PetscInt zero = 0; >>> PetscScalar *d, v; >>> PetscBool assembled; >>> >>> PetscFunctionBegin; >>> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); >>> if (assembled) { ierr = MatZeroEntries(H); >>> CHKERRQ(ierr); } >>> >>> /* Get pointers to vector data */ >>> VecGetSize(X, &N); >>> for (auto i = zero; i != N; ++i) >>> { >>> for (auto j = zero; j != N; ++j) >>> { >>> ierr = >> MatGetValue(data->A, i, j, &v); >>> ierr = MatSetValue(H, >>> i, >> j, v, INSERT_VALUES); >>> } >>> } >>> >>> /* Assemble the matrix */ >>> ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, >> PETSC_TRUE); CHKERRQ(ierr); >>> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> PetscFunctionReturn(0); >>> >>> } >>> >>> In this case, after iterating over all the row and column indexes and >> assembling the matrix the function seems to run correctly. >>> >>> >>> But when I try to copy the matrix >>> >>> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >>> >>> PetscFunctionReturn(0); >>> AppCtx *data = (AppCtx *)ptr; >>> PetscErrorCode ierr; >>> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { >>> ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = MatConvert(data->A, >>> MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); CHKERRQ(ierr); >> >> You cannot do this because you are creating an entirely new H matrix >> here in the subroutine that TAO won't know about. >> >> You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); >> >> But since H is constant why have a copy? Just set the values in H >> initially and don't have a data->A matrix at all. Then your >> FormJacobian() doesn't need to do anything since H is already set with >> proper values and H remains the same through all iterations. >> >> >> >> Barry >> >> >>> >>> >>> >>> /* Assemble the matrix */ >>> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> PetscFunctionReturn(0); } >>> >>> I get the following error messages. >>> >>> Problem size: 259 rows >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for >>> unassembled matrix [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >>> ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by >>> Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure >>> options --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 >>> --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 >>> MatMult() line 2210 in >>> C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c >>> [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\util\tao_util.c >>> [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in >>> C:\Apps\Projects >>> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c >>> [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line >>> 941 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c >>> [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c >>> [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ >>> petsc-3.6.3\src\tao\interface\taosolver.c >>> [0]PETSC ERROR: #7 run_complementarity() line 394 in c:\apps\projects >>> \complementarity.cpp >>> >>> I am new to PETSC. Can anyone help me? >>> >>> Thanks >>> >>> Chris Athiade >> > From norihiro.w at gmail.com Wed Apr 13 00:36:05 2016 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Wed, 13 Apr 2016 07:36:05 +0200 Subject: [petsc-users] SCALAPACK compilation error Message-ID: Hi, I'm trying to compile PETSc v3.5.2 with "--download-scalapack". The full command is ./configure PETSC_ARCH=gcc-release --download-f2cblaslapack=1 -with-debugging=0 --download-superlu_dist --download-hypre=1 --download-ml=1 --download-parmetis --download-metis --download-mumps --download-suitesparse --download-scalapack --with-fc=mpif90 --with-cc=mpicc --with-cxx=mpicxx Unfortunately it fails because linking to gfortran library is missing (see partial log below). Do you have any idea how to fix it? I found "-lgfortran" is used in several other places in configure.log but not for the below particular compilation. conf/scalapack also doesn't have "-lgfortran". Executing: mpicc -o /tmp/petsc-THHgOH/config.libraries/conftest -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O /tmp/petsc-THHgOH/config.libraries/conftest.o -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 -lscalapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2clapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2cblas -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2 -L/home/localadmin/tools/petsc/petsc-3.5.2 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl Possible ERROR while running linker: exit code 256 stderr: /home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib/libscalapack.a(pxerbla.o): In function `pxerbla_': pxerbla.f:(.text+0x7d): undefined reference to `_gfortran_st_write' pxerbla.f:(.text+0x8d): undefined reference to `_gfortran_transfer_integer_write' pxerbla.f:(.text+0x9d): undefined reference to `_gfortran_transfer_integer_write' pxerbla.f:(.text+0xab): undefined reference to `_gfortran_transfer_character_write' pxerbla.f:(.text+0xbb): undefined reference to `_gfortran_transfer_integer_write' pxerbla.f:(.text+0xc3): undefined reference to `_gfortran_st_write_done' collect2: error: ld returned 1 exit status Thanks, Nori -- Norihiro Watanabe From ztdepyahoo at 163.com Wed Apr 13 00:38:37 2016 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Wed, 13 Apr 2016 13:38:37 +0800 (CST) Subject: [petsc-users] which solver should i use to solve a dense matrix occured from spectral method Message-ID: <6bfab000.65be.1540e20e55a.Coremail.ztdepyahoo@163.com> Dear friends: the matrix from spectral method is a dense matrix, and some entries at diagonal may have a zero value. which solver and preconditioner should i use to solve it. Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 13 00:42:49 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 13 Apr 2016 00:42:49 -0500 Subject: [petsc-users] SCALAPACK compilation error In-Reply-To: References: Message-ID: Any perticular reason for using 3.5 instead of currently supported 3.6 releasE? If you have system fortran blas installed - suggest using it [i.e don't specify any blas/lapack option] or use --download-fblaslapack [not --download-f2cblaslapack=1] or use LIBS=-lgfortran Satish On Wed, 13 Apr 2016, Norihiro Watanabe wrote: > Hi, > > I'm trying to compile PETSc v3.5.2 with "--download-scalapack". The > full command is > > ./configure PETSC_ARCH=gcc-release --download-f2cblaslapack=1 > -with-debugging=0 --download-superlu_dist --download-hypre=1 > --download-ml=1 --download-parmetis --download-metis --download-mumps > --download-suitesparse --download-scalapack --with-fc=mpif90 > --with-cc=mpicc --with-cxx=mpicxx > > Unfortunately it fails because linking to gfortran library is missing > (see partial log below). Do you have any idea how to fix it? I found > "-lgfortran" is used in several other places in configure.log but not > for the below particular compilation. conf/scalapack also doesn't have > "-lgfortran". > > > Executing: mpicc -o /tmp/petsc-THHgOH/config.libraries/conftest > -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -O /tmp/petsc-THHgOH/config.libraries/conftest.o > -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 > -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 > -lscalapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib > -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2clapack > -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib > -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2cblas > -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 > -L/usr/lib/gcc/x86_64-linux-gnu/4.8 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2 > -L/home/localadmin/tools/petsc/petsc-3.5.2 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl > -lmpi -lhwloc -lgcc_s -lpthread -ldl > Possible ERROR while running linker: exit code 256 > stderr: > /home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib/libscalapack.a(pxerbla.o): > In function `pxerbla_': > pxerbla.f:(.text+0x7d): undefined reference to `_gfortran_st_write' > pxerbla.f:(.text+0x8d): undefined reference to > `_gfortran_transfer_integer_write' > pxerbla.f:(.text+0x9d): undefined reference to > `_gfortran_transfer_integer_write' > pxerbla.f:(.text+0xab): undefined reference to > `_gfortran_transfer_character_write' > pxerbla.f:(.text+0xbb): undefined reference to > `_gfortran_transfer_integer_write' > pxerbla.f:(.text+0xc3): undefined reference to `_gfortran_st_write_done' > collect2: error: ld returned 1 exit status > > > > Thanks, > Nori > > From norihiro.w at gmail.com Wed Apr 13 01:05:21 2016 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Wed, 13 Apr 2016 08:05:21 +0200 Subject: [petsc-users] SCALAPACK compilation error In-Reply-To: References: Message-ID: Thank you, Satish! LIBS=-lgfortran worked for me. There is no critical reason to keep using the old version. I'm just lazy to update it because the PETSc interface sometimes changes (didn't check 3.6 yet) and I need to change my program as well to support multiple PETSc versions (we need to maintain multiple computers and cannot update PETSc on every computer at once). Best, Nori On Wed, Apr 13, 2016 at 7:42 AM, Satish Balay wrote: > Any perticular reason for using 3.5 instead of currently supported 3.6 releasE? > > > If you have system fortran blas installed - suggest using it > [i.e don't specify any blas/lapack option] > > or use --download-fblaslapack [not --download-f2cblaslapack=1] > > or use LIBS=-lgfortran > > Satish > > On Wed, 13 Apr 2016, Norihiro Watanabe wrote: > >> Hi, >> >> I'm trying to compile PETSc v3.5.2 with "--download-scalapack". The >> full command is >> >> ./configure PETSC_ARCH=gcc-release --download-f2cblaslapack=1 >> -with-debugging=0 --download-superlu_dist --download-hypre=1 >> --download-ml=1 --download-parmetis --download-metis --download-mumps >> --download-suitesparse --download-scalapack --with-fc=mpif90 >> --with-cc=mpicc --with-cxx=mpicxx >> >> Unfortunately it fails because linking to gfortran library is missing >> (see partial log below). Do you have any idea how to fix it? I found >> "-lgfortran" is used in several other places in configure.log but not >> for the below particular compilation. conf/scalapack also doesn't have >> "-lgfortran". >> >> >> Executing: mpicc -o /tmp/petsc-THHgOH/config.libraries/conftest >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -O /tmp/petsc-THHgOH/config.libraries/conftest.o >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 >> -lscalapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2clapack >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2cblas >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu >> -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2 >> -L/home/localadmin/tools/petsc/petsc-3.5.2 >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl >> -lmpi -lhwloc -lgcc_s -lpthread -ldl >> Possible ERROR while running linker: exit code 256 >> stderr: >> /home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib/libscalapack.a(pxerbla.o): >> In function `pxerbla_': >> pxerbla.f:(.text+0x7d): undefined reference to `_gfortran_st_write' >> pxerbla.f:(.text+0x8d): undefined reference to >> `_gfortran_transfer_integer_write' >> pxerbla.f:(.text+0x9d): undefined reference to >> `_gfortran_transfer_integer_write' >> pxerbla.f:(.text+0xab): undefined reference to >> `_gfortran_transfer_character_write' >> pxerbla.f:(.text+0xbb): undefined reference to >> `_gfortran_transfer_integer_write' >> pxerbla.f:(.text+0xc3): undefined reference to `_gfortran_st_write_done' >> collect2: error: ld returned 1 exit status >> >> >> >> Thanks, >> Nori >> >> > -- Norihiro Watanabe From john.mousel at gmail.com Wed Apr 13 07:11:18 2016 From: john.mousel at gmail.com (John Mousel) Date: Wed, 13 Apr 2016 08:11:18 -0400 Subject: [petsc-users] Zeroed solution after 0 iterations Message-ID: I have a colleague who is trying to run some simple test cases. He noticed if he provides the solution vector as the exact solution with a non-zero guess tagged, that the solution vector is zeroed upon return after 0 iterations. The system is just a 5x5 matrix corresponding to a heat equation with Dirichlet BCs. I've copied the KSPView below. He's using petsc-3.6.3. John solution before kspsolve:Vec Object: 1 MPI processes type: mpi Process [0] 300 300 300 300 300 KSP Object: 1 MPI processes type: bcgs maximum iterations=100 tolerances: relative=1e-15, absolute=1e-50, divergence=10000 right preconditioning using nonzero initial guess using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=5, cols=5 package used to perform factorization: petsc total: nonzeros=25, allocated nonzeros=25 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=5, cols=5 total: nonzeros=25, allocated nonzeros=25 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 solution after kspsolve:Vec Object: 1 MPI processes type: mpi Process [0] 0 0 0 0 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 13 08:25:23 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 13 Apr 2016 08:25:23 -0500 Subject: [petsc-users] Zeroed solution after 0 iterations In-Reply-To: References: Message-ID: <4DD2FC75-458D-4DF3-83A7-DBCB77056E92@mcs.anl.gov> John, We'll need the full code to debug this. It doesn't see this should happen. Barry > On Apr 13, 2016, at 7:11 AM, John Mousel wrote: > > I have a colleague who is trying to run some simple test cases. He noticed if he provides the solution vector as the exact solution with a non-zero guess tagged, that the solution vector is zeroed upon return after 0 iterations. The system is just a 5x5 matrix corresponding to a heat equation with Dirichlet BCs. I've copied the KSPView below. He's using petsc-3.6.3. > > John > > solution before kspsolve:Vec Object: 1 MPI processes > type: mpi > Process [0] > 300 > 300 > 300 > 300 > 300 > KSP Object: 1 MPI processes > type: bcgs > maximum iterations=100 > tolerances: relative=1e-15, absolute=1e-50, divergence=10000 > right preconditioning > using nonzero initial guess > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=5, cols=5 > package used to perform factorization: petsc > total: nonzeros=25, allocated nonzeros=25 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 1 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=5, cols=5 > total: nonzeros=25, allocated nonzeros=25 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 1 nodes, limit used is 5 > solution after kspsolve:Vec Object: 1 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > 0 > 0 From john.mousel at gmail.com Wed Apr 13 09:13:20 2016 From: john.mousel at gmail.com (John Mousel) Date: Wed, 13 Apr 2016 10:13:20 -0400 Subject: [petsc-users] Zeroed solution after 0 iterations In-Reply-To: <4DD2FC75-458D-4DF3-83A7-DBCB77056E92@mcs.anl.gov> References: <4DD2FC75-458D-4DF3-83A7-DBCB77056E92@mcs.anl.gov> Message-ID: Barry, I've attached a small example that replicates this behavior. John On Wed, Apr 13, 2016 at 9:25 AM, Barry Smith wrote: > > John, > > We'll need the full code to debug this. It doesn't see this should > happen. > > Barry > > > On Apr 13, 2016, at 7:11 AM, John Mousel wrote: > > > > I have a colleague who is trying to run some simple test cases. He > noticed if he provides the solution vector as the exact solution with a > non-zero guess tagged, that the solution vector is zeroed upon return after > 0 iterations. The system is just a 5x5 matrix corresponding to a heat > equation with Dirichlet BCs. I've copied the KSPView below. He's using > petsc-3.6.3. > > > > John > > > > solution before kspsolve:Vec Object: 1 MPI processes > > type: mpi > > Process [0] > > 300 > > 300 > > 300 > > 300 > > 300 > > KSP Object: 1 MPI processes > > type: bcgs > > maximum iterations=100 > > tolerances: relative=1e-15, absolute=1e-50, divergence=10000 > > right preconditioning > > using nonzero initial guess > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: ilu > > ILU: out-of-place factorization > > 0 levels of fill > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: natural > > factor fill ratio given 1, needed 1 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > package used to perform factorization: petsc > > total: nonzeros=25, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 1 nodes, limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > total: nonzeros=25, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 1 nodes, limit used is 5 > > solution after kspsolve:Vec Object: 1 MPI processes > > type: mpi > > Process [0] > > 0 > > 0 > > 0 > > 0 > > 0 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1.c Type: text/x-csrc Size: 2795 bytes Desc: not available URL: From hzhang at mcs.anl.gov Wed Apr 13 10:03:53 2016 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 13 Apr 2016 10:03:53 -0500 Subject: [petsc-users] which solver should i use to solve a dense matrix occured from spectral method In-Reply-To: <6bfab000.65be.1540e20e55a.Coremail.ztdepyahoo@163.com> References: <6bfab000.65be.1540e20e55a.Coremail.ztdepyahoo@163.com> Message-ID: ??? : > Dear friends: > the matrix from spectral method is a dense matrix, and some entries > at diagonal may have a zero value. > which solver and preconditioner should i use to solve it. > For dense matrix, you should use dense matrix format. LU/Cholesky preconditioner are provided by Lapack or Elemental. Petsc supports interface to these packages. Hong > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 13 10:14:33 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 13 Apr 2016 10:14:33 -0500 Subject: [petsc-users] SCALAPACK compilation error In-Reply-To: References: Message-ID: Glad it worked. Generally we recommend upgrading to latest release of PETSc - as thats what we primarily support. And some of the issues such as this one might be fixed in it [so its best not to revisit such issues with older versions]. If not already fixed - we would look at fixing the issue in the latest version. Wrt your specific usage - we recommend using --download-fblaslapack [and not --download-f2cblaslapack=1] when you have a fortran compiler available [except when using quad precision build] It does mean you would have to put some effort updating your code to use the latest version. We do help with installation issues that you might encounter. Usually you can grab the 'reconfigure_PETSC_ARCH.py' script from the previous install and reuse it to build any new versions. [yes there can be API changes with configure aswell - so there could be corner cases with this process] Satish On Wed, 13 Apr 2016, Norihiro Watanabe wrote: > Thank you, Satish! LIBS=-lgfortran worked for me. There is no critical > reason to keep using the old version. I'm just lazy to update it > because the PETSc interface sometimes changes (didn't check 3.6 yet) > and I need to change my program as well to support multiple PETSc > versions (we need to maintain multiple computers and cannot update > PETSc on every computer at once). > > Best, > Nori > > > On Wed, Apr 13, 2016 at 7:42 AM, Satish Balay wrote: > > Any perticular reason for using 3.5 instead of currently supported 3.6 releasE? > > > > > > If you have system fortran blas installed - suggest using it > > [i.e don't specify any blas/lapack option] > > > > or use --download-fblaslapack [not --download-f2cblaslapack=1] > > > > or use LIBS=-lgfortran > > > > Satish > > > > On Wed, 13 Apr 2016, Norihiro Watanabe wrote: > > > >> Hi, > >> > >> I'm trying to compile PETSc v3.5.2 with "--download-scalapack". The > >> full command is > >> > >> ./configure PETSC_ARCH=gcc-release --download-f2cblaslapack=1 > >> -with-debugging=0 --download-superlu_dist --download-hypre=1 > >> --download-ml=1 --download-parmetis --download-metis --download-mumps > >> --download-suitesparse --download-scalapack --with-fc=mpif90 > >> --with-cc=mpicc --with-cxx=mpicxx > >> > >> Unfortunately it fails because linking to gfortran library is missing > >> (see partial log below). Do you have any idea how to fix it? I found > >> "-lgfortran" is used in several other places in configure.log but not > >> for the below particular compilation. conf/scalapack also doesn't have > >> "-lgfortran". > >> > >> > >> Executing: mpicc -o /tmp/petsc-THHgOH/config.libraries/conftest > >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > >> -O /tmp/petsc-THHgOH/config.libraries/conftest.o > >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 > >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 > >> -lscalapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib > >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2clapack > >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib > >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2cblas > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu > >> -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu > >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2 > >> -L/home/localadmin/tools/petsc/petsc-3.5.2 > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl > >> -lmpi -lhwloc -lgcc_s -lpthread -ldl > >> Possible ERROR while running linker: exit code 256 > >> stderr: > >> /home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib/libscalapack.a(pxerbla.o): > >> In function `pxerbla_': > >> pxerbla.f:(.text+0x7d): undefined reference to `_gfortran_st_write' > >> pxerbla.f:(.text+0x8d): undefined reference to > >> `_gfortran_transfer_integer_write' > >> pxerbla.f:(.text+0x9d): undefined reference to > >> `_gfortran_transfer_integer_write' > >> pxerbla.f:(.text+0xab): undefined reference to > >> `_gfortran_transfer_character_write' > >> pxerbla.f:(.text+0xbb): undefined reference to > >> `_gfortran_transfer_integer_write' > >> pxerbla.f:(.text+0xc3): undefined reference to `_gfortran_st_write_done' > >> collect2: error: ld returned 1 exit status > >> > >> > >> > >> Thanks, > >> Nori > >> > >> > > > > > > From huyaoyu1986 at gmail.com Wed Apr 13 10:25:37 2016 From: huyaoyu1986 at gmail.com (Yaoyu Hu) Date: Wed, 13 Apr 2016 23:25:37 +0800 Subject: [petsc-users] Problem when getting matrix values owned by other processor Message-ID: Hi everyone, I am trying to get values owned by other processors of a parallel matrix. I tried to create a sub-matrix by using MatGetSubMatrices(), and then MatGetRow() on the sub-matrix. But MatGetRow() give me the following error message: ===== Error message begins ===== No support for this operation for this object type only local rows ===== Error message ends ===== The parallel matrix is a parallel dense matrix. The ISs for MatGetSubMatrices() are created using ISCreateGeneral() and PETSC_COMM_WORLD. The row IS is sorted by ISSort(). What did I mistake while using the above functions? Is there a better way to get access to matrix values owned by other processor? Thanks! HU Yaoyu From hzhang at mcs.anl.gov Wed Apr 13 10:36:32 2016 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 13 Apr 2016 10:36:32 -0500 Subject: [petsc-users] Problem when getting matrix values owned by other processor In-Reply-To: References: Message-ID: Yaoyu : MatGetSubMatrices() returns sequential matrices. IS must be sequential, created with PETSC_COMM_SELF. See petsc/src/mat/examples/tests/ex42.c Check your submatrices. Hong Hi everyone, > > I am trying to get values owned by other processors of a parallel matrix. > > I tried to create a sub-matrix by using MatGetSubMatrices(), and then > MatGetRow() on the sub-matrix. But MatGetRow() give me the following > error message: > > ===== Error message begins ===== > > No support for this operation for this object type > only local rows > > ===== Error message ends ===== > > The parallel matrix is a parallel dense matrix. The ISs for > MatGetSubMatrices() are created using ISCreateGeneral() and > PETSC_COMM_WORLD. The row IS is sorted by ISSort(). > > What did I mistake while using the above functions? Is there a better > way to get access to matrix values owned by other processor? > > Thanks! > > HU Yaoyu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 13 11:42:09 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 13 Apr 2016 11:42:09 -0500 Subject: [petsc-users] Zeroed solution after 0 iterations In-Reply-To: References: <4DD2FC75-458D-4DF3-83A7-DBCB77056E92@mcs.anl.gov> Message-ID: John, Thank you for the report. It was a bug in our code. I have fixed the code in the master and next branch of the PETSc repository. We will be making a PETSc release in the next few days that will have the fix. Barry > On Apr 13, 2016, at 9:13 AM, John Mousel wrote: > > Barry, > > I've attached a small example that replicates this behavior. > > John > > On Wed, Apr 13, 2016 at 9:25 AM, Barry Smith wrote: > > John, > > We'll need the full code to debug this. It doesn't see this should happen. > > Barry > > > On Apr 13, 2016, at 7:11 AM, John Mousel wrote: > > > > I have a colleague who is trying to run some simple test cases. He noticed if he provides the solution vector as the exact solution with a non-zero guess tagged, that the solution vector is zeroed upon return after 0 iterations. The system is just a 5x5 matrix corresponding to a heat equation with Dirichlet BCs. I've copied the KSPView below. He's using petsc-3.6.3. > > > > John > > > > solution before kspsolve:Vec Object: 1 MPI processes > > type: mpi > > Process [0] > > 300 > > 300 > > 300 > > 300 > > 300 > > KSP Object: 1 MPI processes > > type: bcgs > > maximum iterations=100 > > tolerances: relative=1e-15, absolute=1e-50, divergence=10000 > > right preconditioning > > using nonzero initial guess > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: ilu > > ILU: out-of-place factorization > > 0 levels of fill > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: natural > > factor fill ratio given 1, needed 1 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > package used to perform factorization: petsc > > total: nonzeros=25, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 1 nodes, limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > total: nonzeros=25, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 1 nodes, limit used is 5 > > solution after kspsolve:Vec Object: 1 MPI processes > > type: mpi > > Process [0] > > 0 > > 0 > > 0 > > 0 > > 0 > > > From huyaoyu1986 at gmail.com Wed Apr 13 12:41:27 2016 From: huyaoyu1986 at gmail.com (Yaoyu Hu) Date: Thu, 14 Apr 2016 01:41:27 +0800 Subject: [petsc-users] petsc-users Digest, Vol 88, Issue 41 In-Reply-To: References: Message-ID: Hi Hong, Thank you for your rapid response. I figured out that it was my mistake. I did not use the last argument of MatGetSubMatrices() in the right way. It turns out that, just like VecGetArray(), there is no need to explicitly allocate memory for the last argument passed to MatGetSubMatrices(). Now I can get the right submatrices and the rows of the submatrices. But the reason for the original error message is unclear, and I haven't debug it. And further. I checked the submatrices I obtained by MatGetSubMatrices() with both parallel ISs and sequential ISs. The submatrices are identical. Is it means that I could actually use parallel IS (created with PETSC_COMM_WORLD) ? > Yaoyu : > MatGetSubMatrices() returns sequential matrices. > IS must be sequential, created with PETSC_COMM_SELF. > See petsc/src/mat/examples/tests/ex42.c > > Check your submatrices. > > Hong > > Hi everyone, >> >> I am trying to get values owned by other processors of a parallel matrix. >> >> I tried to create a sub-matrix by using MatGetSubMatrices(), and then >> MatGetRow() on the sub-matrix. But MatGetRow() give me the following >> error message: >> >> ===== Error message begins ===== >> >> No support for this operation for this object type >> only local rows >> >> ===== Error message ends ===== >> >> The parallel matrix is a parallel dense matrix. The ISs for >> MatGetSubMatrices() are created using ISCreateGeneral() and >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). >> >> What did I mistake while using the above functions? Is there a better >> way to get access to matrix values owned by other processor? >> >> Thanks! >> >> HU Yaoyu >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 88, Issue 41 > ******************************************* From john.mousel at gmail.com Wed Apr 13 13:13:17 2016 From: john.mousel at gmail.com (John Mousel) Date: Wed, 13 Apr 2016 14:13:17 -0400 Subject: [petsc-users] Zeroed solution after 0 iterations In-Reply-To: References: <4DD2FC75-458D-4DF3-83A7-DBCB77056E92@mcs.anl.gov> Message-ID: Thanks Barry. On Wed, Apr 13, 2016 at 12:42 PM, Barry Smith wrote: > > John, > > Thank you for the report. It was a bug in our code. I have fixed the > code in the master and next branch of the PETSc repository. We will be > making a PETSc release in the next few days that will have the fix. > > Barry > > > On Apr 13, 2016, at 9:13 AM, John Mousel wrote: > > > > Barry, > > > > I've attached a small example that replicates this behavior. > > > > John > > > > On Wed, Apr 13, 2016 at 9:25 AM, Barry Smith wrote: > > > > John, > > > > We'll need the full code to debug this. It doesn't see this should > happen. > > > > Barry > > > > > On Apr 13, 2016, at 7:11 AM, John Mousel > wrote: > > > > > > I have a colleague who is trying to run some simple test cases. He > noticed if he provides the solution vector as the exact solution with a > non-zero guess tagged, that the solution vector is zeroed upon return after > 0 iterations. The system is just a 5x5 matrix corresponding to a heat > equation with Dirichlet BCs. I've copied the KSPView below. He's using > petsc-3.6.3. > > > > > > John > > > > > > solution before kspsolve:Vec Object: 1 MPI processes > > > type: mpi > > > Process [0] > > > 300 > > > 300 > > > 300 > > > 300 > > > 300 > > > KSP Object: 1 MPI processes > > > type: bcgs > > > maximum iterations=100 > > > tolerances: relative=1e-15, absolute=1e-50, divergence=10000 > > > right preconditioning > > > using nonzero initial guess > > > using UNPRECONDITIONED norm type for convergence test > > > PC Object: 1 MPI processes > > > type: ilu > > > ILU: out-of-place factorization > > > 0 levels of fill > > > tolerance for zero pivot 2.22045e-14 > > > matrix ordering: natural > > > factor fill ratio given 1, needed 1 > > > Factored matrix follows: > > > Mat Object: 1 MPI processes > > > type: seqaij > > > rows=5, cols=5 > > > package used to perform factorization: petsc > > > total: nonzeros=25, allocated nonzeros=25 > > > total number of mallocs used during MatSetValues calls =0 > > > using I-node routines: found 1 nodes, limit used is 5 > > > linear system matrix = precond matrix: > > > Mat Object: 1 MPI processes > > > type: seqaij > > > rows=5, cols=5 > > > total: nonzeros=25, allocated nonzeros=25 > > > total number of mallocs used during MatSetValues calls =0 > > > using I-node routines: found 1 nodes, limit used is 5 > > > solution after kspsolve:Vec Object: 1 MPI processes > > > type: mpi > > > Process [0] > > > 0 > > > 0 > > > 0 > > > 0 > > > 0 > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cnathaide at yahoo.com Wed Apr 13 13:38:15 2016 From: cnathaide at yahoo.com (Christopher Athaide) Date: Wed, 13 Apr 2016 13:38:15 -0500 Subject: [petsc-users] Problem setting Jacobian matrix in complementarity problem In-Reply-To: <7A9CBB55-5B66-427D-839E-9CD321EBAC64@mcs.anl.gov> References: <004001d19424$f2117960$d6346c20$@yahoo.com> <005301d19435$0b1b2180$21516480$@yahoo.com> <004201d194c8$15871780$40954680$@yahoo.com> <7A9CBB55-5B66-427D-839E-9CD321EBAC64@mcs.anl.gov> Message-ID: <005101d195b3$a510eed0$ef32cc70$@yahoo.com> Barry, Thanks. I was able to get the MatCopy to work inside the Jacobian and produce the optimal results. I initially created a copy of the Jacobian matrix element-by-element and stored a copy in the AppCtx data structure. In this way, the diagonal elements were saved even if they were zero. There was drastic improvement in the running time. In the original run, the total run time was 73 seconds, and 57 seconds of these was spent in the FormJacobian function. Using a MatCopy, the total run time was about 18 seconds. Thanks again for your assistance. Chris -----Original Message----- From: Barry Smith [mailto:bsmith at mcs.anl.gov] Sent: Tuesday, April 12, 2016 1:08 PM To: Christopher Athaide Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Problem setting Jacobian matrix in complementarity problem Yes call the MatCopy() inside the FormJacobian > On Apr 12, 2016, at 9:32 AM, Christopher Athaide wrote: > > Barry, > > Thank you for your suggestions. I implemented the changes that you > suggested. > > int main() > { > PetscScalar v; > ierr = MatCreateSeqAIJ(PETSC_COMM_SELF, N, N, 5 * N, NULL, &J); > CHKERRQ(ierr); > for (auto i = 0; i != N; ++i) > { > for (auto j = 0; j != N; ++j) > { > ierr = MatGetValue(data.A, i, j, &v); CHKERRQ(ierr); > ierr = MatSetValue(J, i, j, v, INSERT_VALUES); CHKERRQ(ierr); > } > } > > /* Assemble the matrix */ > ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > compare_matrices(data.A, J); > > } > > PetscErrorCode FormJacobian(Tao, Vec, Mat H, Mat, void* ptr) { > > PetscFunctionBegin; > > // AppCtx *data = (AppCtx *)ptr; > // Is there any way I can do a copy data->A to H using a matrix or a > batch operation? > > PetscFunctionReturn(0); > } > > > The good news is that the model runs without generating any errors. > The bad news is that the program terminates without the optimality > conditions being satisfied. > > In the original program, the model takes about 70 seconds to run and > the optimality conditions are satisfied. In this version, it takes > about 5 seconds. > > Is there any way I can copy the matrix data->A to H in the > FormJacobian using a matrix function instead of iterating through all > the rows and columns and then assembling the matrix at every iteration. > > Thank you for your help. > > Chris > > -----Original Message----- > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > Sent: Monday, April 11, 2016 4:05 PM > To: Christopher Athaide > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Problem setting Jacobian matrix in > complementarity problem > > > Does your matrix have entries all along the diagonal? If not you > need to preallocate for even the 0 locations and set the value of 0 in > there specifically. > > Barry > >> On Apr 11, 2016, at 3:59 PM, Christopher Athaide >> > wrote: >> >> Hi Barry, >> >> Thank you for taking a look at the issue I am having. I followed your >> suggestion and made the following changes: >> >> main() >> { >> ierr = MatDuplicate(data.A, MAT_SHARE_NONZERO_PATTERN, &J); >> CHKERRQ(ierr); >> ierr = MatCopy(data.A, J, SAME_NONZERO_PATTERN); CHKERRQ(ierr); >> >> } >> >> PetscErrorCode FormJacobian(Tao, Vec, Mat, Mat, void*) { >> >> PetscFunctionBegin; >> >> PetscFunctionReturn(0); >> } >> >> But I get a bunch of errors - I am not sure why. >> >> Problem size : 259 rows >> [0]PETSC ERROR : -------------------- - Error >> Message-------------------------------------------------------------- >> [0]PETSC ERROR : Argument out of range [0]PETSC ERROR : New nonzero >> at(0, 0) caused a malloc Use MatSetOption(A, >> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >> [0]PETSC ERROR : See http >> ://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR : Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >> ERROR : Unknown Name on a arch - mswin - c - debug named LAGRANGE by >> Christopher Mon Apr 11 21 : 50 : 06 2016 [0]PETSC ERROR : Configure >> options --with - cc = "win32fe cl" --with - fc = >> 0 --with - mpi = 0 --download - f2cblaslapack = 1 --with - debugging >> = >> 1 [0]PETSC ERROR : #1 MatSetValues_SeqAIJ() line 485 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC >> ERROR : #2 MatSetValues() line 1173 in C : \Apps\Projects\petsc - >> 3.6.3\src\mat\interface\matrix.c [0]PETSC ERROR : #3 >> MatDiagonalSet_Default() line 198 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #4 >> MatDiagonalSet_SeqAIJ() line 194 in C : >> \Apps\Projects\petsc - 3.6.3\src\mat\impls\aij\seq\aij.c [0]PETSC >> ERROR : #5 MatDiagonalSet() line 242 in C : \Apps\Projects\petsc - >> 3.6.3\src\mat\utils\axpy.c [0]PETSC ERROR : #6 >> Tao_SSLS_FunctionGradient() line 66 in C : >> \Apps\Projects\petsc - >> 3.6.3\src\tao\complementarity\impls\ssls\ssls.c >> [0]PETSC ERROR : #7 TaoLineSearchComputeObjectiveAndGradient() line >> 941 in C >> : \Apps\Projects\petsc - >> 3.6.3\src\tao\linesearch\interface\taolinesearch.c >> [0]PETSC ERROR : #8 TaoSolve_SSILS() line 64 in C : >> \Apps\Projects\petsc - >> 3.6.3\src\tao\complementarity\impls\ssls\ssils.c >> [0]PETSC ERROR : #9 TaoSolve() line 204 in C : \Apps\Projects\petsc - >> 3.6.3\src\tao\interface\taosolver.c >> [0]PETSC ERROR : #10 run_complementarity() line 290 in c : >> \apps\projects\complementarity.cpp >> >> I appreciate your help in this matter. >> >> Thanks >> >> Chris >> >> -----Original Message----- >> From: Barry Smith [mailto:bsmith at mcs.anl.gov] >> Sent: Monday, April 11, 2016 3:16 PM >> To: Christopher Athaide >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] Problem setting Jacobian matrix in >> complementarity problem >> >> >>> On Apr 11, 2016, at 2:04 PM, Christopher Athaide >>> >> wrote: >>> >>> Hi, >>> >>> I am trying to solve a mixed complementarity problem using the TAO >>> SSILS >> Solver . >>> >>> C(x) = A x + d , l <= x <= u >>> >>> where A = n x n matrix >>> D = n x 1 vector >>> >>> The Jacobian matrix is equal to the matrix A and is constant at each >> iteration. >>> >>> I have setup the Jacobian update function >>> >>> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >>> AppCtx *data = (AppCtx *)ptr; >>> PetscErrorCode ierr; >>> PetscInt N; >>> PetscInt zero = 0; >>> PetscScalar *d, v; >>> PetscBool assembled; >>> >>> PetscFunctionBegin; >>> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); >>> if (assembled) { ierr = MatZeroEntries(H); >>> CHKERRQ(ierr); } >>> >>> /* Get pointers to vector data */ >>> VecGetSize(X, &N); >>> for (auto i = zero; i != N; ++i) >>> { >>> for (auto j = zero; j != N; ++j) >>> { >>> ierr = >> MatGetValue(data->A, i, j, &v); >>> ierr = MatSetValue(H, >>> i, >> j, v, INSERT_VALUES); >>> } >>> } >>> >>> /* Assemble the matrix */ >>> ierr = MatSetOption(H, MAT_IGNORE_ZERO_ENTRIES, >> PETSC_TRUE); CHKERRQ(ierr); >>> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> PetscFunctionReturn(0); >>> >>> } >>> >>> In this case, after iterating over all the row and column indexes >>> and >> assembling the matrix the function seems to run correctly. >>> >>> >>> But when I try to copy the matrix >>> >>> PetscErrorCode FormJacobian(Tao, Vec X, Mat H, Mat, void *ptr) { >>> >>> PetscFunctionReturn(0); >>> AppCtx *data = (AppCtx *)ptr; >>> PetscErrorCode ierr; >>> ierr = MatAssembled(H, &assembled); CHKERRQ(ierr); if (assembled) { >>> ierr = MatZeroEntries(H); CHKERRQ(ierr); } ierr = >>> MatConvert(data->A, MATSAME, MatReuse::MAT_INITIAL_MATRIX, &H); >>> CHKERRQ(ierr); >> >> You cannot do this because you are creating an entirely new H matrix >> here in the subroutine that TAO won't know about. >> >> You should use MatCopy(data->A, H, SAME_NONZERO_PATTERN); >> >> But since H is constant why have a copy? Just set the values in H >> initially and don't have a data->A matrix at all. Then your >> FormJacobian() doesn't need to do anything since H is already set >> with proper values and H remains the same through all iterations. >> >> >> >> Barry >> >> >>> >>> >>> >>> /* Assemble the matrix */ >>> ierr = MatAssemblyBegin(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> ierr = MatAssemblyEnd(H, MAT_FINAL_ASSEMBLY); >> CHKERRQ(ierr); >>> PetscFunctionReturn(0); } >>> >>> I get the following error messages. >>> >>> Problem size: 259 rows >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for >>> unassembled matrix [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC >>> ERROR: Unknown Name on a arch-mswin-c-debug named LAGRANGE by >>> Christopher Mon Apr 11 19:57:37 2016 [0]PETSC ERROR: Configure >>> options --with-cc="win32fe cl" --with-fc=0 --with-mpi=0 >>> --download-f2cblaslapack=1 --with-debugging=1 [0]PETSC ERROR: #1 >>> MatMult() line 2210 in >>> C:\Apps\Projects\petsc-3.6.3\src\mat\interface\matrix.c >>> [0]PETSC ERROR: #2 MatDFischer() line 284 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\util\tao_util.c >>> [0]PETSC ERROR: #3 Tao_SSLS_FunctionGradient() line 64 in >>> C:\Apps\Projects >>> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssls.c >>> [0]PETSC ERROR: #4 TaoLineSearchComputeObjectiveAndGradient() line >>> 941 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\linesearch\interface\taolinesearch.c >>> [0]PETSC ERROR: #5 TaoSolve_SSILS() line 64 in C:\Apps\Projects >>> \petsc-3.6.3\src\tao\complementarity\impls\ssls\ssils.c >>> [0]PETSC ERROR: #6 TaoSolve() line 204 in C:\Apps\Projects\ >>> petsc-3.6.3\src\tao\interface\taosolver.c >>> [0]PETSC ERROR: #7 run_complementarity() line 394 in >>> c:\apps\projects \complementarity.cpp >>> >>> I am new to PETSC. Can anyone help me? >>> >>> Thanks >>> >>> Chris Athiade >> > From hzhang at mcs.anl.gov Wed Apr 13 13:54:18 2016 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 13 Apr 2016 13:54:18 -0500 Subject: [petsc-users] petsc-users Digest, Vol 88, Issue 41 In-Reply-To: References: Message-ID: Yaoyu: "MatGetSubMatrices() can extract ONLY sequential submatrices (from both sequential and parallel matrices). Use MatGetSubMatrix() to extract a parallel submatrix." Using parallel IS for MatGetSubMatrices() definitely incorrect, unless you only use one process. > > And further. I checked the submatrices I obtained by > MatGetSubMatrices() with both parallel ISs and sequential ISs. The > submatrices are identical. Is it means that I could actually use > parallel IS (created with PETSC_COMM_WORLD) ? > What did you compare with? I do not understand what submatrices would obtain with parallel IS using more than one process. Hong > > > Yaoyu : > > MatGetSubMatrices() returns sequential matrices. > > IS must be sequential, created with PETSC_COMM_SELF. > > See petsc/src/mat/examples/tests/ex42.c > > > > Check your submatrices. > > > > Hong > > > > Hi everyone, > >> > >> I am trying to get values owned by other processors of a parallel > matrix. > >> > >> I tried to create a sub-matrix by using MatGetSubMatrices(), and then > >> MatGetRow() on the sub-matrix. But MatGetRow() give me the following > >> error message: > >> > >> ===== Error message begins ===== > >> > >> No support for this operation for this object type > >> only local rows > >> > >> ===== Error message ends ===== > >> > >> The parallel matrix is a parallel dense matrix. The ISs for > >> MatGetSubMatrices() are created using ISCreateGeneral() and > >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). > >> > >> What did I mistake while using the above functions? Is there a better > >> way to get access to matrix values owned by other processor? > >> > >> Thanks! > >> > >> HU Yaoyu > >> > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160413/6be45909/attachment.html > > > > > > ------------------------------ > > > > _______________________________________________ > > petsc-users mailing list > > petsc-users at mcs.anl.gov > > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > > > > End of petsc-users Digest, Vol 88, Issue 41 > > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From norihiro.w at gmail.com Thu Apr 14 03:05:15 2016 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Thu, 14 Apr 2016 10:05:15 +0200 Subject: [petsc-users] SCALAPACK compilation error In-Reply-To: References: Message-ID: Hi Satish, Many thanks for the tips. From the next time, I'll test with the latest PETSc before asking you. As a little feed back to you, I've tested the latest PETSc and confirmed both "configure" command and compilation work fine. So my problem has already been fixed in the newer version. Best, Nori On Wed, Apr 13, 2016 at 5:14 PM, Satish Balay wrote: > Glad it worked. > > Generally we recommend upgrading to latest release of PETSc - as thats > what we primarily support. And some of the issues such as this one > might be fixed in it [so its best not to revisit such issues with > older versions]. If not already fixed - we would look at fixing the > issue in the latest version. > > Wrt your specific usage - we recommend using --download-fblaslapack [and > not --download-f2cblaslapack=1] when you have a fortran compiler available > [except when using quad precision build] > > It does mean you would have to put some effort updating your code to > use the latest version. We do help with installation issues that you > might encounter. Usually you can grab the 'reconfigure_PETSC_ARCH.py' > script from the previous install and reuse it to build any new > versions. [yes there can be API changes with configure aswell - so > there could be corner cases with this process] > > Satish > > On Wed, 13 Apr 2016, Norihiro Watanabe wrote: > >> Thank you, Satish! LIBS=-lgfortran worked for me. There is no critical >> reason to keep using the old version. I'm just lazy to update it >> because the PETSc interface sometimes changes (didn't check 3.6 yet) >> and I need to change my program as well to support multiple PETSc >> versions (we need to maintain multiple computers and cannot update >> PETSc on every computer at once). >> >> Best, >> Nori >> >> >> On Wed, Apr 13, 2016 at 7:42 AM, Satish Balay wrote: >> > Any perticular reason for using 3.5 instead of currently supported 3.6 releasE? >> > >> > >> > If you have system fortran blas installed - suggest using it >> > [i.e don't specify any blas/lapack option] >> > >> > or use --download-fblaslapack [not --download-f2cblaslapack=1] >> > >> > or use LIBS=-lgfortran >> > >> > Satish >> > >> > On Wed, 13 Apr 2016, Norihiro Watanabe wrote: >> > >> >> Hi, >> >> >> >> I'm trying to compile PETSc v3.5.2 with "--download-scalapack". The >> >> full command is >> >> >> >> ./configure PETSC_ARCH=gcc-release --download-f2cblaslapack=1 >> >> -with-debugging=0 --download-superlu_dist --download-hypre=1 >> >> --download-ml=1 --download-parmetis --download-metis --download-mumps >> >> --download-suitesparse --download-scalapack --with-fc=mpif90 >> >> --with-cc=mpicc --with-cxx=mpicxx >> >> >> >> Unfortunately it fails because linking to gfortran library is missing >> >> (see partial log below). Do you have any idea how to fix it? I found >> >> "-lgfortran" is used in several other places in configure.log but not >> >> for the below particular compilation. conf/scalapack also doesn't have >> >> "-lgfortran". >> >> >> >> >> >> Executing: mpicc -o /tmp/petsc-THHgOH/config.libraries/conftest >> >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> >> -O /tmp/petsc-THHgOH/config.libraries/conftest.o >> >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 >> >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib64 >> >> -lscalapack -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib >> >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2clapack >> >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib >> >> -L/home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib -lf2cblas >> >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib >> >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 >> >> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 >> >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu >> >> -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu >> >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu >> >> -Wl,-rpath,/home/localadmin/tools/petsc/petsc-3.5.2 >> >> -L/home/localadmin/tools/petsc/petsc-3.5.2 >> >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl >> >> -lmpi -lhwloc -lgcc_s -lpthread -ldl >> >> Possible ERROR while running linker: exit code 256 >> >> stderr: >> >> /home/localadmin/tools/petsc/petsc-3.5.2/gcc-release/lib/libscalapack.a(pxerbla.o): >> >> In function `pxerbla_': >> >> pxerbla.f:(.text+0x7d): undefined reference to `_gfortran_st_write' >> >> pxerbla.f:(.text+0x8d): undefined reference to >> >> `_gfortran_transfer_integer_write' >> >> pxerbla.f:(.text+0x9d): undefined reference to >> >> `_gfortran_transfer_integer_write' >> >> pxerbla.f:(.text+0xab): undefined reference to >> >> `_gfortran_transfer_character_write' >> >> pxerbla.f:(.text+0xbb): undefined reference to >> >> `_gfortran_transfer_integer_write' >> >> pxerbla.f:(.text+0xc3): undefined reference to `_gfortran_st_write_done' >> >> collect2: error: ld returned 1 exit status >> >> >> >> >> >> >> >> Thanks, >> >> Nori >> >> >> >> >> > >> >> >> >> > -- Norihiro Watanabe From natacha.bereux at gmail.com Thu Apr 14 10:19:45 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Thu, 14 Apr 2016 17:19:45 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Hi Satish thanks a lot for the answer. Unfortunately, it does not work yet. More precisely : --download-mumps works fine (and every --download-package option works perfectly). I am then able to compile a PETSc library. --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally --with-package-lib=libXXXX.so also works But I would like to use static librairies, preinstalled on my computer ... and this fails. For the moment I gave up compiling with MUMPS, and I am instead trying to compile with Metis 5. I have a preinstalled version in a some directory lets say metis_dir I try -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] --with-metis-include=metis_dir/include this fails (see the attached config.log) -with-metis-dir=metis_dir also fails Is there a problem with static librairies ? Natacha On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay wrote: > On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > Hello, > > I am trying to compile Petsc (3.6.3) with external packages (MUMPS and > its > > prerequisites). > > More precisely I would like PETSc to use a pre-installed version of > MUMPS. > > > > Petsc downloads and compiles the prerequisites (parmetis, scalapack etc) > : > > this works fine. > > What metis/parmetis/slcalapack is this MUMPS installed with? > > What version of MUMPS did you install? > > Why could you not use --download-mumps? > > Using a different metis/parmetis/slcalapack to install MUMPS - and > then specifying --download-metis --download-parmetis > --download-scalapack [i.e different versions/builds of the same > libraries] can result in conflcits. > > > > > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > install, > > but the > > configure step fails with the followiing message: > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > I do not understand what is wrong. > > I have attached the configure.log file. > > > > Any hint would be greatly appreciated ! > > >>>> > Executing: mpicc -show > stdout: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > Defined make macro "MPICC_SHOW" to "gcc > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > <<<< > Ok - so you are using system openmpi with gcc. > > > >>>> > Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest -fPIC > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > /tmp/petsc-0u_4WI/config.libraries/conftest.o > -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps -lzmumps > -lmumps_common -lpord > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lscalapack > -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > -Wl,-rpath,/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -lgfortran -lm -lquadmath -lm -llapack -lblas > -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > -Wl,-rpath,/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -lgfortran -lm -lquadmath -lm > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lparmetis > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lmetis -lm > -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > Possible ERROR while running linker: exit code 256 > stderr: > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > dlr_stats.F:(.text+0x3079): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x30fa): undefined reference to `GOMP_critical_name_end' > dlr_stats.F:(.text+0x310e): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x318f): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > dlr_stats.F:(.text+0x33a9): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x33f9): undefined reference to `GOMP_critical_name_end' > dlr_stats.F:(.text+0x340a): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x345a): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > dlr_stats.F:(.text+0x3576): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x35a7): undefined reference to `GOMP_critical_name_end' > dlr_stats.F:(.text+0x35b8): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x35e9): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > dlr_stats.F:(.text+0x36ac): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x36ce): undefined reference to `GOMP_critical_name_end' > dlr_stats.F:(.text+0x36df): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x3701): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > dlr_stats.F:(.text+0x37c1): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x37e3): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > dlr_stats.F:(.text+0x3839): undefined reference to > `GOMP_critical_name_start' > dlr_stats.F:(.text+0x3856): undefined reference to `GOMP_critical_name_end' > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In function > `__dmumps_ana_lr_MOD_mumps_scotch_kway': > dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > > > > Looks like this MUMPS is built with ptscotch and openmp. > > You can specify -lgomp for openmp. This can be done with the configure > option > LIBS=-lgomp > > Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > figureout > how to specify it. > > Its best to specify all the packages [mumps and its dependencies] you > built manually with the options: > > --with-package-include --with-package-lib > > Satish > > > Best regards, > > > > Natacha > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1800991 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Apr 14 10:41:44 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 14 Apr 2016 10:41:44 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-ssl=0 --with-mpi=1 --with-debugging=1 --PETSC_ARCH=linux-metis --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so --with-blacs-lib=/usr/lib/libblacs-openmpi.so --with-metis-lib="[~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib/libmetis.a,~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib/libGKlib.a]" --with-metis-include=~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include A couple of issues: - there is no more blacs option [as latest scalapack bundles blacs inside scalapack] - configure doesn't understsnad '~' So Try: --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi" --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib -lmetis.a -lGKlib" --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include You would also need the petsc configure option --with-shared-libraries=0 [to be able to build petsc with these static libraries that are not likely compiled with -fPIC option] Satish On Thu, 14 Apr 2016, Natacha BEREUX wrote: > Hi Satish > thanks a lot for the answer. Unfortunately, it does not work yet. > More precisely : > --download-mumps works fine (and every --download-package option works > perfectly). I am then able to compile a PETSc library. > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > --with-package-lib=libXXXX.so also works > > But I would like to use static librairies, preinstalled on my computer ... > and this fails. > > For the moment I gave up compiling with MUMPS, and I am instead trying to > compile with Metis 5. > I have a preinstalled version in a some directory lets say metis_dir > I try > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > --with-metis-include=metis_dir/include > this fails (see the attached config.log) > -with-metis-dir=metis_dir also fails > Is there a problem with static librairies ? > > Natacha > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay wrote: > > > On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > Hello, > > > I am trying to compile Petsc (3.6.3) with external packages (MUMPS and > > its > > > prerequisites). > > > More precisely I would like PETSc to use a pre-installed version of > > MUMPS. > > > > > > Petsc downloads and compiles the prerequisites (parmetis, scalapack etc) > > : > > > this works fine. > > > > What metis/parmetis/slcalapack is this MUMPS installed with? > > > > What version of MUMPS did you install? > > > > Why could you not use --download-mumps? > > > > Using a different metis/parmetis/slcalapack to install MUMPS - and > > then specifying --download-metis --download-parmetis > > --download-scalapack [i.e different versions/builds of the same > > libraries] can result in conflcits. > > > > > > > > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > > install, > > > but the > > > configure step fails with the followiing message: > > > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > > > I do not understand what is wrong. > > > I have attached the configure.log file. > > > > > > Any hint would be greatly appreciated ! > > > > >>>> > > Executing: mpicc -show > > stdout: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > > -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > Defined make macro "MPICC_SHOW" to "gcc > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > > -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > <<<< > > Ok - so you are using system openmpi with gcc. > > > > > > >>>> > > Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest -fPIC > > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > /tmp/petsc-0u_4WI/config.libraries/conftest.o > > -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps -lzmumps > > -lmumps_common -lpord > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lscalapack > > -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > -Wl,-rpath,/usr/lib/openmpi/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > -lgfortran -lm -lquadmath -lm -llapack -lblas > > -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > -Wl,-rpath,/usr/lib/openmpi/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > -lgfortran -lm -lquadmath -lm > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lparmetis > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lmetis -lm > > -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > Possible ERROR while running linker: exit code 256 > > stderr: > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > dlr_stats.F:(.text+0x3079): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x30fa): undefined reference to `GOMP_critical_name_end' > > dlr_stats.F:(.text+0x310e): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x318f): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > dlr_stats.F:(.text+0x33a9): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x33f9): undefined reference to `GOMP_critical_name_end' > > dlr_stats.F:(.text+0x340a): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x345a): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > dlr_stats.F:(.text+0x3576): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x35a7): undefined reference to `GOMP_critical_name_end' > > dlr_stats.F:(.text+0x35b8): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x35e9): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > dlr_stats.F:(.text+0x36ac): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x36ce): undefined reference to `GOMP_critical_name_end' > > dlr_stats.F:(.text+0x36df): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x3701): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > dlr_stats.F:(.text+0x37c1): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x37e3): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > dlr_stats.F:(.text+0x3839): undefined reference to > > `GOMP_critical_name_start' > > dlr_stats.F:(.text+0x3856): undefined reference to `GOMP_critical_name_end' > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In function > > `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > > dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > > dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > > dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > > dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > > > > > > > > Looks like this MUMPS is built with ptscotch and openmp. > > > > You can specify -lgomp for openmp. This can be done with the configure > > option > > LIBS=-lgomp > > > > Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > > figureout > > how to specify it. > > > > Its best to specify all the packages [mumps and its dependencies] you > > built manually with the options: > > > > --with-package-include --with-package-lib > > > > Satish > > > > > Best regards, > > > > > > Natacha > > > > > > From natacha.bereux at gmail.com Thu Apr 14 10:57:16 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Thu, 14 Apr 2016 17:57:16 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Sorry, do not take into account my last email. I made some progress and I am now able to configure PETSc with a pre-installed version of metis. Problems come when I try to configure PETSc with MUMPS My command line is ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-ssl=0 --with-mpi=1 --with-debugging=1 --PETSC_ARCH=linux-metis-mumps --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] --with-metis-include=$METIS_PRE/include --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] --with-mumps-include=$MUMPS_PRE/include where METIS_PRE and MUMPS_PRE are the path to the local installs of metis and mumps) I get (at least) the following error /libdmumps.a(dend_driver.o): undefined reference to symbol 'blacs_gridexit_' /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status Would you have any idea of its meaning ? The configure.log is attached Thanks a lot if you can help me ! Natacha On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX wrote: > Hi Satish > thanks a lot for the answer. Unfortunately, it does not work yet. > More precisely : > --download-mumps works fine (and every --download-package option works > perfectly). I am then able to compile a PETSc library. > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > --with-package-lib=libXXXX.so also works > > But I would like to use static librairies, preinstalled on my computer > ... and this fails. > > For the moment I gave up compiling with MUMPS, and I am instead trying to > compile with Metis 5. > I have a preinstalled version in a some directory lets say metis_dir > I try > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > --with-metis-include=metis_dir/include > this fails (see the attached config.log) > -with-metis-dir=metis_dir also fails > Is there a problem with static librairies ? > > Natacha > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay wrote: > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: >> >> > Hello, >> > I am trying to compile Petsc (3.6.3) with external packages (MUMPS and >> its >> > prerequisites). >> > More precisely I would like PETSc to use a pre-installed version of >> MUMPS. >> > >> > Petsc downloads and compiles the prerequisites (parmetis, scalapack >> etc) : >> > this works fine. >> >> What metis/parmetis/slcalapack is this MUMPS installed with? >> >> What version of MUMPS did you install? >> >> Why could you not use --download-mumps? >> >> Using a different metis/parmetis/slcalapack to install MUMPS - and >> then specifying --download-metis --download-parmetis >> --download-scalapack [i.e different versions/builds of the same >> libraries] can result in conflcits. >> >> > >> > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS >> install, >> > but the >> > configure step fails with the followiing message: >> > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work >> > >> > I do not understand what is wrong. >> > I have attached the configure.log file. >> > >> > Any hint would be greatly appreciated ! >> >> >>>> >> Executing: mpicc -show >> stdout: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc >> Defined make macro "MPICC_SHOW" to "gcc >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" >> <<<< >> Ok - so you are using system openmpi with gcc. >> >> >> >>>> >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O >> /tmp/petsc-0u_4WI/config.libraries/conftest.o >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps -lzmumps >> -lmumps_common -lpord >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lscalapack >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm >> -Wl,-rpath,/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu >> -lgfortran -lm -lquadmath -lm -llapack -lblas >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm >> -Wl,-rpath,/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu >> -lgfortran -lm -lquadmath -lm >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lparmetis >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lmetis -lm >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl >> Possible ERROR while running linker: exit code 256 >> stderr: >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': >> dlr_stats.F:(.text+0x3079): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x30fa): undefined reference to >> `GOMP_critical_name_end' >> dlr_stats.F:(.text+0x310e): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x318f): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': >> dlr_stats.F:(.text+0x33a9): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x33f9): undefined reference to >> `GOMP_critical_name_end' >> dlr_stats.F:(.text+0x340a): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x345a): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': >> dlr_stats.F:(.text+0x3576): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x35a7): undefined reference to >> `GOMP_critical_name_end' >> dlr_stats.F:(.text+0x35b8): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x35e9): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': >> dlr_stats.F:(.text+0x36ac): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x36ce): undefined reference to >> `GOMP_critical_name_end' >> dlr_stats.F:(.text+0x36df): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x3701): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': >> dlr_stats.F:(.text+0x37c1): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x37e3): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': >> dlr_stats.F:(.text+0x3839): undefined reference to >> `GOMP_critical_name_start' >> dlr_stats.F:(.text+0x3856): undefined reference to >> `GOMP_critical_name_end' >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In function >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': >> dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' >> dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' >> dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' >> dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' >> dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' >> >> >> >> Looks like this MUMPS is built with ptscotch and openmp. >> >> You can specify -lgomp for openmp. This can be done with the configure >> option >> LIBS=-lgomp >> >> Wrt and PTSCOTCH stuff depending upon how its installed we'll have to >> figureout >> how to specify it. >> >> Its best to specify all the packages [mumps and its dependencies] you >> built manually with the options: >> >> --with-package-include --with-package-lib >> >> Satish >> >> > Best regards, >> > >> > Natacha >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1924677 bytes Desc: not available URL: From natacha.bereux at gmail.com Thu Apr 14 10:59:23 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Thu, 14 Apr 2016 17:59:23 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Ok I did not see your ultra fast reply, thanks a lot : I immediately correct my configure line ! Natacha On Thu, Apr 14, 2016 at 5:41 PM, Satish Balay wrote: > Configure Options: --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --with-cc=mpicc --with-cxx=mpicxx > --with-fc=mpif90 --with-ssl=0 --with-mpi=1 --with-debugging=1 > --PETSC_ARCH=linux-metis > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > --with-blacs-lib=/usr/lib/libblacs-openmpi.so > --with-metis-lib="[~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib/libmetis.a,~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib/libGKlib.a]" > --with-metis-include=~/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > A couple of issues: > - there is no more blacs option [as latest scalapack bundles blacs inside > scalapack] > - configure doesn't understsnad '~' > > > So Try: > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi" > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > -lmetis.a -lGKlib" > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > You would also need the petsc configure option --with-shared-libraries=0 > [to be able to build petsc with these static libraries that are not likely > compiled with -fPIC option] > > Satish > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > Hi Satish > > thanks a lot for the answer. Unfortunately, it does not work yet. > > More precisely : > > --download-mumps works fine (and every --download-package option works > > perfectly). I am then able to compile a PETSc library. > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > > --with-package-lib=libXXXX.so also works > > > > But I would like to use static librairies, preinstalled on my computer > ... > > and this fails. > > > > For the moment I gave up compiling with MUMPS, and I am instead trying > to > > compile with Metis 5. > > I have a preinstalled version in a some directory lets say metis_dir > > I try > > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > > --with-metis-include=metis_dir/include > > this fails (see the attached config.log) > > -with-metis-dir=metis_dir also fails > > Is there a problem with static librairies ? > > > > Natacha > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay wrote: > > > > > On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > > Hello, > > > > I am trying to compile Petsc (3.6.3) with external packages (MUMPS > and > > > its > > > > prerequisites). > > > > More precisely I would like PETSc to use a pre-installed version of > > > MUMPS. > > > > > > > > Petsc downloads and compiles the prerequisites (parmetis, scalapack > etc) > > > : > > > > this works fine. > > > > > > What metis/parmetis/slcalapack is this MUMPS installed with? > > > > > > What version of MUMPS did you install? > > > > > > Why could you not use --download-mumps? > > > > > > Using a different metis/parmetis/slcalapack to install MUMPS - and > > > then specifying --download-metis --download-parmetis > > > --download-scalapack [i.e different versions/builds of the same > > > libraries] can result in conflcits. > > > > > > > > > > > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > > > install, > > > > but the > > > > configure step fails with the followiing message: > > > > > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > details): > > > > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > > > > > I do not understand what is wrong. > > > > I have attached the configure.log file. > > > > > > > > Any hint would be greatly appreciated ! > > > > > > >>>> > > > Executing: mpicc -show > > > stdout: gcc -I/usr/lib/openmpi/include > -I/usr/lib/openmpi/include/openmpi > > > -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > > Defined make macro "MPICC_SHOW" to "gcc > > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > > > -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > <<<< > > > Ok - so you are using system openmpi with gcc. > > > > > > > > > >>>> > > > Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > -fPIC > > > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > > /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > -lzmumps > > > -lmumps_common -lpord > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lscalapack > > > -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > -Wl,-rpath,/usr/lib/openmpi/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > -lgfortran -lm -lquadmath -lm -llapack -lblas > > > -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > -Wl,-rpath,/usr/lib/openmpi/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > -lgfortran -lm -lquadmath -lm > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lparmetis > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lmetis -lm > > > -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > > Possible ERROR while running linker: exit code 256 > > > stderr: > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > dlr_stats.F:(.text+0x3079): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x30fa): undefined reference to > `GOMP_critical_name_end' > > > dlr_stats.F:(.text+0x310e): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x318f): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > dlr_stats.F:(.text+0x33a9): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x33f9): undefined reference to > `GOMP_critical_name_end' > > > dlr_stats.F:(.text+0x340a): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x345a): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > dlr_stats.F:(.text+0x3576): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x35a7): undefined reference to > `GOMP_critical_name_end' > > > dlr_stats.F:(.text+0x35b8): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x35e9): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > dlr_stats.F:(.text+0x36ac): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x36ce): undefined reference to > `GOMP_critical_name_end' > > > dlr_stats.F:(.text+0x36df): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x3701): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > dlr_stats.F:(.text+0x37c1): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x37e3): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > dlr_stats.F:(.text+0x3839): undefined reference to > > > `GOMP_critical_name_start' > > > dlr_stats.F:(.text+0x3856): undefined reference to > `GOMP_critical_name_end' > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > function > > > `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > > > dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > > > dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > > > dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > > > dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > > > > > > > > > > > > Looks like this MUMPS is built with ptscotch and openmp. > > > > > > You can specify -lgomp for openmp. This can be done with the configure > > > option > > > LIBS=-lgomp > > > > > > Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > > > figureout > > > how to specify it. > > > > > > Its best to specify all the packages [mumps and its dependencies] you > > > built manually with the options: > > > > > > --with-package-include --with-package-lib > > > > > > Satish > > > > > > > Best regards, > > > > > > > > Natacha > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 14 11:07:56 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 14 Apr 2016 11:07:56 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: you'll have to roll-in the --with-blacs-lib option into --with-scalapack-lib option Satish On Thu, 14 Apr 2016, Natacha BEREUX wrote: > Sorry, do not take into account my last email. > I made some progress and I am now able to configure PETSc with a > pre-installed version of metis. > > Problems come when I try to configure PETSc with MUMPS > > My command line is > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-ssl=0 --with-mpi=1 --with-debugging=1 > --PETSC_ARCH=linux-metis-mumps > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > --with-metis-include=$METIS_PRE/include > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > --with-mumps-include=$MUMPS_PRE/include > > where METIS_PRE and MUMPS_PRE are the path to the local installs of metis > and mumps) > > I get (at least) the following error > /libdmumps.a(dend_driver.o): undefined reference to symbol 'blacs_gridexit_' > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing from > command line > collect2: error: ld returned 1 exit status > > > Would you have any idea of its meaning ? > > The configure.log is attached > Thanks a lot if you can help me ! > Natacha > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX > wrote: > > > Hi Satish > > thanks a lot for the answer. Unfortunately, it does not work yet. > > More precisely : > > --download-mumps works fine (and every --download-package option works > > perfectly). I am then able to compile a PETSc library. > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > > --with-package-lib=libXXXX.so also works > > > > But I would like to use static librairies, preinstalled on my computer > > ... and this fails. > > > > For the moment I gave up compiling with MUMPS, and I am instead trying to > > compile with Metis 5. > > I have a preinstalled version in a some directory lets say metis_dir > > I try > > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > > --with-metis-include=metis_dir/include > > this fails (see the attached config.log) > > -with-metis-dir=metis_dir also fails > > Is there a problem with static librairies ? > > > > Natacha > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay wrote: > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > >> > >> > Hello, > >> > I am trying to compile Petsc (3.6.3) with external packages (MUMPS and > >> its > >> > prerequisites). > >> > More precisely I would like PETSc to use a pre-installed version of > >> MUMPS. > >> > > >> > Petsc downloads and compiles the prerequisites (parmetis, scalapack > >> etc) : > >> > this works fine. > >> > >> What metis/parmetis/slcalapack is this MUMPS installed with? > >> > >> What version of MUMPS did you install? > >> > >> Why could you not use --download-mumps? > >> > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > >> then specifying --download-metis --download-parmetis > >> --download-scalapack [i.e different versions/builds of the same > >> libraries] can result in conflcits. > >> > >> > > >> > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > >> install, > >> > but the > >> > configure step fails with the followiing message: > >> > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > >> details): > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > >> > > >> > I do not understand what is wrong. > >> > I have attached the configure.log file. > >> > > >> > Any hint would be greatly appreciated ! > >> > >> >>>> > >> Executing: mpicc -show > >> stdout: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > >> Defined make macro "MPICC_SHOW" to "gcc > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > >> <<<< > >> Ok - so you are using system openmpi with gcc. > >> > >> > >> >>>> > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest -fPIC > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps -lzmumps > >> -lmumps_common -lpord > >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lscalapack > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > >> -Wl,-rpath,/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > >> -Wl,-rpath,/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > >> -lgfortran -lm -lquadmath -lm > >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lparmetis > >> -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib -lmetis -lm > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > >> Possible ERROR while running linker: exit code 256 > >> stderr: > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > >> dlr_stats.F:(.text+0x3079): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x30fa): undefined reference to > >> `GOMP_critical_name_end' > >> dlr_stats.F:(.text+0x310e): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x318f): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > >> dlr_stats.F:(.text+0x33a9): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x33f9): undefined reference to > >> `GOMP_critical_name_end' > >> dlr_stats.F:(.text+0x340a): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x345a): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > >> dlr_stats.F:(.text+0x3576): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x35a7): undefined reference to > >> `GOMP_critical_name_end' > >> dlr_stats.F:(.text+0x35b8): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x35e9): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > >> dlr_stats.F:(.text+0x36ac): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x36ce): undefined reference to > >> `GOMP_critical_name_end' > >> dlr_stats.F:(.text+0x36df): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x3701): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > >> dlr_stats.F:(.text+0x37c1): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x37e3): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > >> dlr_stats.F:(.text+0x3839): undefined reference to > >> `GOMP_critical_name_start' > >> dlr_stats.F:(.text+0x3856): undefined reference to > >> `GOMP_critical_name_end' > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In function > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > >> dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > >> dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > >> dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > >> dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > >> dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > >> > >> > >> > >> Looks like this MUMPS is built with ptscotch and openmp. > >> > >> You can specify -lgomp for openmp. This can be done with the configure > >> option > >> LIBS=-lgomp > >> > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > >> figureout > >> how to specify it. > >> > >> Its best to specify all the packages [mumps and its dependencies] you > >> built manually with the options: > >> > >> --with-package-include --with-package-lib > >> > >> Satish > >> > >> > Best regards, > >> > > >> > Natacha > >> > > >> > > > > > From huyaoyu1986 at gmail.com Thu Apr 14 12:00:44 2016 From: huyaoyu1986 at gmail.com (Yaoyu Hu) Date: Fri, 15 Apr 2016 01:00:44 +0800 Subject: [petsc-users] Problem when getting matrix values owned by other processor Message-ID: Hi Hong, I think what you said about the ISs for MatGetSubMatrices() are right. Today I tried to run my program to get submatrix by MatGetSubMatrices() twice. One time with ISs created with PETSC_COMM_WORLD and the other time with PETSC_COMM_SELF. I was creating a 6x3 submatrix on each processor from a 570x3 parallel dense matrix. There are only 2 processors in total. The results are as follows: ======== submatrix with ISs created with PETSC_COMM_WORLD ========= Processor 0 Mat Object: 1 MPI processes type: seqdense 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 Processor 1 Mat Object: 1 MPI processes type: seqdense 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 ==== end of submatrix with ISs created with PETSC_COMM_WORLD ====== ======== submatrix with ISs created with PETSC_COMM_SELF ========= Processor 0 Mat Object: 1 MPI processes type: seqdense 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 Processor 1 Mat Object: 1 MPI processes type: seqdense 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 ==== end of submatrix with ISs created with PETSC_COMM_SELF ====== The results are identical. I really agree with you that only sequential ISs should be used when creating submatrix by MatGetSubMatrices(). The above results may be just a coincidence. I take your suggestions and I am using only sequential ISs with MatGetSubMatrices() now. Yet another question: Can I use IS which have duplicate entries in it? The documentation of MatGetSubMatrices() says that "The index sets may not have duplicate entries", so I think no duplicated entries are allowed in the IS. But again, I just tried IS which have duplicated entries. And the resultant submatrices for each processor seemd to be correct. In fact, in the above sample submatrices I showed, the submatrix owned by Processor 0 has its row IS specified as [x, x, a, b, c, d]. Where 'x' means the duplicated entries in the IS. Then I got two identical rows in the submatrix owned by Processor 0. But I think I was doing this incorrectly. Thanks. HU Yaoyu > > Yaoyu: > > "MatGetSubMatrices() can extract ONLY sequential submatrices > (from both sequential and parallel matrices). Use MatGetSubMatrix() > to extract a parallel submatrix." > > Using parallel IS for MatGetSubMatrices() definitely incorrect, unless you > only use one process. > >> >> And further. I checked the submatrices I obtained by >> MatGetSubMatrices() with both parallel ISs and sequential ISs. The >> submatrices are identical. Is it means that I could actually use >> parallel IS (created with PETSC_COMM_WORLD) ? >> > > What did you compare with? I do not understand what submatrices would > obtain with parallel IS using more than one process. > > Hong > >> >> > Yaoyu : >> > MatGetSubMatrices() returns sequential matrices. >> > IS must be sequential, created with PETSC_COMM_SELF. >> > See petsc/src/mat/examples/tests/ex42.c >> > >> > Check your submatrices. >> > >> > Hong >> > >> > Hi everyone, >> >> >> >> I am trying to get values owned by other processors of a parallel >> matrix. >> >> >> >> I tried to create a sub-matrix by using MatGetSubMatrices(), and then >> >> MatGetRow() on the sub-matrix. But MatGetRow() give me the following >> >> error message: >> >> >> >> ===== Error message begins ===== >> >> >> >> No support for this operation for this object type >> >> only local rows >> >> >> >> ===== Error message ends ===== >> >> >> >> The parallel matrix is a parallel dense matrix. The ISs for >> >> MatGetSubMatrices() are created using ISCreateGeneral() and >> >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). >> >> >> >> What did I mistake while using the above functions? Is there a better >> >> way to get access to matrix values owned by other processor? >> >> >> >> Thanks! >> >> >> >> HU Yaoyu >> >> >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: < >> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160413/6be45909/attachment.html >> > >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > petsc-users mailing list >> > petsc-users at mcs.anl.gov >> > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users >> > >> > >> > End of petsc-users Digest, Vol 88, Issue 41 >> > ******************************************* >> From hzhang at mcs.anl.gov Thu Apr 14 13:01:48 2016 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 14 Apr 2016 13:01:48 -0500 Subject: [petsc-users] Problem when getting matrix values owned by other processor In-Reply-To: References: Message-ID: Yaoyu : Can you send me the code for calling MatGetSubMatrices()? I want to check how did you create mpi ISs. Checking MatGetSubMatrices() for mpiaij matrix format, it seems duplicate IS indices would work. I guess duplicate rows might be ok, but duplicate columns might run into trouble from our implementation. By definition, a submatrix of A should not have duplicate row/column of A, isn't it? Hong Hi Hong, > > > I think what you said about the ISs for MatGetSubMatrices() are right. > > Today I tried to run my program to get submatrix by > MatGetSubMatrices() twice. One time with ISs created with > PETSC_COMM_WORLD and the other time with PETSC_COMM_SELF. I was > creating a 6x3 submatrix on each processor from a 570x3 parallel dense > matrix. There are only 2 processors in total. The results are as > follows: > > ======== submatrix with ISs created with PETSC_COMM_WORLD ========= > Processor 0 > Mat Object: 1 MPI processes > type: seqdense > 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 > 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 > 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 > 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 > > Processor 1 > Mat Object: 1 MPI processes > type: seqdense > 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 > 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 > 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 > 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 > 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 > 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 > > ==== end of submatrix with ISs created with PETSC_COMM_WORLD ====== > > ======== submatrix with ISs created with PETSC_COMM_SELF ========= > Processor 0 > Mat Object: 1 MPI processes > type: seqdense > 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 > 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 > 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 > 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 > > Processor 1 > Mat Object: 1 MPI processes > type: seqdense > 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 > 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 > 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 > 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 > 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 > 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 > > ==== end of submatrix with ISs created with PETSC_COMM_SELF ====== > > The results are identical. I really agree with you that only > sequential ISs should be used when creating submatrix by > MatGetSubMatrices(). The above results may be just a coincidence. > > I take your suggestions and I am using only sequential ISs with > MatGetSubMatrices() now. > > Yet another question: Can I use IS which have duplicate entries in it? > The documentation of MatGetSubMatrices() says that "The index sets may > not have duplicate entries", so I think no duplicated entries are > allowed in the IS. But again, I just tried IS which have duplicated > entries. And the resultant submatrices for each processor seemd to be > correct. In fact, in the above sample submatrices I showed, the > submatrix owned by Processor 0 has its row IS specified as [x, x, a, > b, c, d]. Where 'x' means the duplicated entries in the IS. Then I got > two identical rows in the submatrix owned by Processor 0. > > But I think I was doing this incorrectly. > > Thanks. > > HU Yaoyu > > > > > Yaoyu: > > > > "MatGetSubMatrices() can extract ONLY sequential submatrices > > (from both sequential and parallel matrices). Use MatGetSubMatrix() > > to extract a parallel submatrix." > > > > Using parallel IS for MatGetSubMatrices() definitely incorrect, unless > you > > only use one process. > > > >> > >> And further. I checked the submatrices I obtained by > >> MatGetSubMatrices() with both parallel ISs and sequential ISs. The > >> submatrices are identical. Is it means that I could actually use > >> parallel IS (created with PETSC_COMM_WORLD) ? > >> > > > > What did you compare with? I do not understand what submatrices would > > obtain with parallel IS using more than one process. > > > > Hong > > > >> > >> > Yaoyu : > >> > MatGetSubMatrices() returns sequential matrices. > >> > IS must be sequential, created with PETSC_COMM_SELF. > >> > See petsc/src/mat/examples/tests/ex42.c > >> > > >> > Check your submatrices. > >> > > >> > Hong > >> > > >> > Hi everyone, > >> >> > >> >> I am trying to get values owned by other processors of a parallel > >> matrix. > >> >> > >> >> I tried to create a sub-matrix by using MatGetSubMatrices(), and then > >> >> MatGetRow() on the sub-matrix. But MatGetRow() give me the following > >> >> error message: > >> >> > >> >> ===== Error message begins ===== > >> >> > >> >> No support for this operation for this object type > >> >> only local rows > >> >> > >> >> ===== Error message ends ===== > >> >> > >> >> The parallel matrix is a parallel dense matrix. The ISs for > >> >> MatGetSubMatrices() are created using ISCreateGeneral() and > >> >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). > >> >> > >> >> What did I mistake while using the above functions? Is there a better > >> >> way to get access to matrix values owned by other processor? > >> >> > >> >> Thanks! > >> >> > >> >> HU Yaoyu > >> >> > >> > -------------- next part -------------- > >> > An HTML attachment was scrubbed... > >> > URL: < > >> > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160413/6be45909/attachment.html > >> > > >> > > >> > ------------------------------ > >> > > >> > _______________________________________________ > >> > petsc-users mailing list > >> > petsc-users at mcs.anl.gov > >> > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > >> > > >> > > >> > End of petsc-users Digest, Vol 88, Issue 41 > >> > ******************************************* > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huyaoyu1986 at gmail.com Fri Apr 15 01:21:04 2016 From: huyaoyu1986 at gmail.com (Yaoyu Hu) Date: Fri, 15 Apr 2016 14:21:04 +0800 Subject: [petsc-users] Problem when getting matrix values owned by other processor Message-ID: Hi Hong, I do not know if it is OK to put an attachment in the email sending to the mailing list. So I just send the code, the input file (for you to re-load the parallel matrix, binary file), and the result file directly and only to you. I tested the sample code on my computer. The content of the result file is what I got on the terminal. There is one particular thing that, I use COMPLEX number. I thought you may be using PETSc compiled using REAL number. So I attach the ASCII version of the input file. You can compile and run the code if you wish. To change the communicator for creating ISs, just change the macro defined at the beginning of the source file. And also, you are definitely right about the definition of a submatrix that no duplicated rows should be in a submatrix. The situation for me is that, I am trying to write a piece of code to do some simple CFD simulations using finite volume method. The row IS I passed to MatGetSubMatrices() is the neighbor element indices for the current element being processed. The reason for the duplicated indices in the row IS is that I just simply put a ZERO index at the place where the actual neighbor element is the boundary of the fluid domain. You see, it is all because I?m too lazy to deal with the differences between the boundary elements and the inner elements while I?m gathering information from the parallel matrix. I think I should stop doing this. And I am working on refining the code. Do you have any suggestions for me to handle the boundary elements? Thanks. HU Yaoyu 2016-04-15 2:01 GMT+08:00 Hong : > Yaoyu : > Can you send me the code for calling MatGetSubMatrices()? > I want to check how did you create mpi ISs. > > Checking MatGetSubMatrices() for mpiaij matrix format, it seems duplicate IS > indices would work. I guess duplicate rows might be ok, but duplicate > columns might run into trouble from our implementation. > > By definition, a submatrix of A should not have duplicate row/column of A, > isn't it? > > Hong > > Hi Hong, >> >> >> I think what you said about the ISs for MatGetSubMatrices() are right. >> >> Today I tried to run my program to get submatrix by >> MatGetSubMatrices() twice. One time with ISs created with >> PETSC_COMM_WORLD and the other time with PETSC_COMM_SELF. I was >> creating a 6x3 submatrix on each processor from a 570x3 parallel dense >> matrix. There are only 2 processors in total. The results are as >> follows: >> >> ======== submatrix with ISs created with PETSC_COMM_WORLD ========= >> Processor 0 >> Mat Object: 1 MPI processes >> type: seqdense >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 >> 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 >> 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 >> 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 >> 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 >> >> Processor 1 >> Mat Object: 1 MPI processes >> type: seqdense >> 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 >> 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 >> 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 >> 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 >> 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 >> 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 >> >> ==== end of submatrix with ISs created with PETSC_COMM_WORLD ====== >> >> ======== submatrix with ISs created with PETSC_COMM_SELF ========= >> Processor 0 >> Mat Object: 1 MPI processes >> type: seqdense >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 >> 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 >> 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 >> 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 >> 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 >> >> Processor 1 >> Mat Object: 1 MPI processes >> type: seqdense >> 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 >> 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 >> 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 >> 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 >> 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 >> 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 >> >> ==== end of submatrix with ISs created with PETSC_COMM_SELF ====== >> >> The results are identical. I really agree with you that only >> sequential ISs should be used when creating submatrix by >> MatGetSubMatrices(). The above results may be just a coincidence. >> >> I take your suggestions and I am using only sequential ISs with >> MatGetSubMatrices() now. >> >> Yet another question: Can I use IS which have duplicate entries in it? >> The documentation of MatGetSubMatrices() says that "The index sets may >> not have duplicate entries", so I think no duplicated entries are >> allowed in the IS. But again, I just tried IS which have duplicated >> entries. And the resultant submatrices for each processor seemd to be >> correct. In fact, in the above sample submatrices I showed, the >> submatrix owned by Processor 0 has its row IS specified as [x, x, a, >> b, c, d]. Where 'x' means the duplicated entries in the IS. Then I got >> two identical rows in the submatrix owned by Processor 0. >> >> But I think I was doing this incorrectly. >> >> Thanks. >> >> HU Yaoyu >> >> > >> > Yaoyu: >> > >> > "MatGetSubMatrices() can extract ONLY sequential submatrices >> > (from both sequential and parallel matrices). Use MatGetSubMatrix() >> > to extract a parallel submatrix." >> > >> > Using parallel IS for MatGetSubMatrices() definitely incorrect, unless >> > you >> > only use one process. >> > >> >> >> >> And further. I checked the submatrices I obtained by >> >> MatGetSubMatrices() with both parallel ISs and sequential ISs. The >> >> submatrices are identical. Is it means that I could actually use >> >> parallel IS (created with PETSC_COMM_WORLD) ? >> >> >> > >> > What did you compare with? I do not understand what submatrices would >> > obtain with parallel IS using more than one process. >> > >> > Hong >> > >> >> >> >> > Yaoyu : >> >> > MatGetSubMatrices() returns sequential matrices. >> >> > IS must be sequential, created with PETSC_COMM_SELF. >> >> > See petsc/src/mat/examples/tests/ex42.c >> >> > >> >> > Check your submatrices. >> >> > >> >> > Hong >> >> > >> >> > Hi everyone, >> >> >> >> >> >> I am trying to get values owned by other processors of a parallel >> >> matrix. >> >> >> >> >> >> I tried to create a sub-matrix by using MatGetSubMatrices(), and >> >> >> then >> >> >> MatGetRow() on the sub-matrix. But MatGetRow() give me the following >> >> >> error message: >> >> >> >> >> >> ===== Error message begins ===== >> >> >> >> >> >> No support for this operation for this object type >> >> >> only local rows >> >> >> >> >> >> ===== Error message ends ===== >> >> >> >> >> >> The parallel matrix is a parallel dense matrix. The ISs for >> >> >> MatGetSubMatrices() are created using ISCreateGeneral() and >> >> >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). >> >> >> >> >> >> What did I mistake while using the above functions? Is there a >> >> >> better >> >> >> way to get access to matrix values owned by other processor? >> >> >> >> >> >> Thanks! >> >> >> >> >> >> HU Yaoyu >> >> >> From marco.mavari at gmail.com Fri Apr 15 02:58:27 2016 From: marco.mavari at gmail.com (marco mavari) Date: Fri, 15 Apr 2016 09:58:27 +0200 Subject: [petsc-users] Form Local function from global Message-ID: Hello, I am using external library (MOOSE) to provide function/jacobian evaluation routines. It communicates with Petsc by SNES. However, I would like to do nonlinear domain decomposition ( similar to NASM/ASPIN). Is it possible to extract LocalEvaluation routines from GlobalFunctions? best, marco -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Apr 15 03:05:54 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 15 Apr 2016 09:05:54 +0100 Subject: [petsc-users] Form Local function from global In-Reply-To: References: Message-ID: <87d1prfg9p.fsf@jedbrown.org> marco mavari writes: > However, I would like to do nonlinear domain decomposition ( similar to > NASM/ASPIN). > Is it possible to extract LocalEvaluation routines from GlobalFunctions? No, the global function is global, so you can't call "just the local part". Moreover, those nonlinear domain decomposition techniques require overlap, which also isn't provided automatically. As a practical matter, someone (maybe you) would need to add support for nonlinear DD to the discretization libraries. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Fri Apr 15 10:50:45 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 15 Apr 2016 16:50:45 +0100 Subject: [petsc-users] Ordering with MatCreateNest and MPI In-Reply-To: References: <87shyxzh2c.fsf@jedbrown.org> <000401d190c8$4f1ee9e0$ed5cbda0$@polito.it> <871t6hz46p.fsf@jedbrown.org> <000601d1916f$caeb7140$60c253c0$@polito.it> <878u0owk5r.fsf@jedbrown.org> Message-ID: <87a8kug9be.fsf@jedbrown.org> FRANCAVILLA MATTEO ALESSANDRO writes: > Dear Jed, > > thanks for the reply. Despite my efforts, I'm still unable to use the > information carried by the index set. I'm able to extract the IS through > MatNestGetISs but I don't really know how to interprete it and to exploit it > to pursue my goal. Try VecGetSubVector(y,isrows[0],&y0) and VecView y0. > I don't seem to find a tutorial explaining how to do the following: > > "MatNestGetISs give you the ISs to extract or set the portions of x and y > corresponding to the first and second block. The index sets are not > globally contiguous and can't be if you want sane parallelism." > > Can you please give me some references for the above operation? If you had two processes and all of the first block was on the first process, then there could be no load balancing. Load balancing requires that the index sets not be globally contiguous. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hzhang at mcs.anl.gov Fri Apr 15 12:05:07 2016 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 15 Apr 2016 12:05:07 -0500 Subject: [petsc-users] Problem when getting matrix values owned by other processor In-Reply-To: References: Message-ID: Yaoyu : Thanks for sending me your code and matrices. I tested it and found the answer. When ISs are created with #define MY_COMMUNICATOR (PETSC_COMM_WORLD) the parallel subMatrixIS_Row is stored as (ISView() shows it) IS Object: 2 MPI processes type: general [0] Number of indices in set 6 [0] 0 0 [0] 1 190 [0] 2 0 [0] 3 10 [0] 4 9 [0] 5 1 [1] Number of indices in set 6 [1] 0 95 [1] 1 475 [1] 2 275 [1] 3 295 [1] 4 284 [1] 5 286 i.e., your local ISs are concatenated as a parallel IS. In MatGetSubMatrices_MPIAIJ() implementation, petsc only uses local size and local indices of input IS, this explains why you got same answer. > > The situation for me is that, I am trying to write a piece of code to > do some simple CFD simulations using finite volume method. The row IS > I passed to MatGetSubMatrices() is the neighbor element indices for > the current element being processed. The reason for the duplicated > indices in the row IS is that I just simply put a ZERO index at the > place where the actual neighbor element is the boundary of the fluid > domain. You see, it is all because I?m too lazy to deal with the > differences between the boundary elements and the inner elements while > I?m gathering information from the parallel matrix. I think I should > stop doing this. And I am working on refining the code. Do you have > any suggestions for me to handle the boundary elements? > Sorry, I'm not an expert on finite element. Others may help you on this issue. Intuitively, matrix is assembled from elements. You should have more efficient way to retrieve elements without using matrix. Hong > > 2016-04-15 2:01 GMT+08:00 Hong : > > Yaoyu : > > Can you send me the code for calling MatGetSubMatrices()? > > I want to check how did you create mpi ISs. > > > > Checking MatGetSubMatrices() for mpiaij matrix format, it seems > duplicate IS > > indices would work. I guess duplicate rows might be ok, but duplicate > > columns might run into trouble from our implementation. > > > > By definition, a submatrix of A should not have duplicate row/column of > A, > > isn't it? > > > > Hong > > > > Hi Hong, > >> > >> > >> I think what you said about the ISs for MatGetSubMatrices() are right. > >> > >> Today I tried to run my program to get submatrix by > >> MatGetSubMatrices() twice. One time with ISs created with > >> PETSC_COMM_WORLD and the other time with PETSC_COMM_SELF. I was > >> creating a 6x3 submatrix on each processor from a 570x3 parallel dense > >> matrix. There are only 2 processors in total. The results are as > >> follows: > >> > >> ======== submatrix with ISs created with PETSC_COMM_WORLD ========= > >> Processor 0 > >> Mat Object: 1 MPI processes > >> type: seqdense > >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > >> 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 > >> 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 > >> 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 > >> 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 > >> > >> Processor 1 > >> Mat Object: 1 MPI processes > >> type: seqdense > >> 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 > >> 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 > >> 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 > >> 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 > >> 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 > >> 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 > >> > >> ==== end of submatrix with ISs created with PETSC_COMM_WORLD ====== > >> > >> ======== submatrix with ISs created with PETSC_COMM_SELF ========= > >> Processor 0 > >> Mat Object: 1 MPI processes > >> type: seqdense > >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > >> 2.7074996615625420e+10 6.8399452804377966e+11 -1.2730861976538324e+08 > >> 2.7359996580000423e+10 8.2079343365253589e+11 6.3654303672968522e+06 > >> 2.9639996295000458e+10 1.9151846785225828e+12 -1.0184686034752984e+08 > >> 2.9924996259375465e+10 1.1947308948674746e+12 -3.8181422587255287e+08 > >> 5.4149993231250839e+10 2.6675786593707410e+13 -4.9650360873402004e+09 > >> > >> Processor 1 > >> Mat Object: 1 MPI processes > >> type: seqdense > >> 5.4149993231250839e+10 1.9183199555758315e+12 5.9880080526417112e+08 > >> 7.8374990203126236e+10 5.7441876402563477e+12 1.7367770705944958e+09 > >> 8.0939989882501266e+10 5.7347670250898613e+12 1.7900992494213123e+09 > >> 8.1509989811251282e+10 5.7751527083651416e+12 1.8027055821637158e+09 > >> 8.4074989490626328e+10 6.1543675577970762e+12 1.8557943339637902e+09 > >> 1.0829998646250171e+11 9.5915997778791719e+12 2.9940040263208551e+09 > >> > >> ==== end of submatrix with ISs created with PETSC_COMM_SELF ====== > >> > >> The results are identical. I really agree with you that only > >> sequential ISs should be used when creating submatrix by > >> MatGetSubMatrices(). The above results may be just a coincidence. > >> > >> I take your suggestions and I am using only sequential ISs with > >> MatGetSubMatrices() now. > >> > >> Yet another question: Can I use IS which have duplicate entries in it? > >> The documentation of MatGetSubMatrices() says that "The index sets may > >> not have duplicate entries", so I think no duplicated entries are > >> allowed in the IS. But again, I just tried IS which have duplicated > >> entries. And the resultant submatrices for each processor seemd to be > >> correct. In fact, in the above sample submatrices I showed, the > >> submatrix owned by Processor 0 has its row IS specified as [x, x, a, > >> b, c, d]. Where 'x' means the duplicated entries in the IS. Then I got > >> two identical rows in the submatrix owned by Processor 0. > >> > >> But I think I was doing this incorrectly. > >> > >> Thanks. > >> > >> HU Yaoyu > >> > >> > > >> > Yaoyu: > >> > > >> > "MatGetSubMatrices() can extract ONLY sequential submatrices > >> > (from both sequential and parallel matrices). Use MatGetSubMatrix() > >> > to extract a parallel submatrix." > >> > > >> > Using parallel IS for MatGetSubMatrices() definitely incorrect, unless > >> > you > >> > only use one process. > >> > > >> >> > >> >> And further. I checked the submatrices I obtained by > >> >> MatGetSubMatrices() with both parallel ISs and sequential ISs. The > >> >> submatrices are identical. Is it means that I could actually use > >> >> parallel IS (created with PETSC_COMM_WORLD) ? > >> >> > >> > > >> > What did you compare with? I do not understand what submatrices would > >> > obtain with parallel IS using more than one process. > >> > > >> > Hong > >> > > >> >> > >> >> > Yaoyu : > >> >> > MatGetSubMatrices() returns sequential matrices. > >> >> > IS must be sequential, created with PETSC_COMM_SELF. > >> >> > See petsc/src/mat/examples/tests/ex42.c > >> >> > > >> >> > Check your submatrices. > >> >> > > >> >> > Hong > >> >> > > >> >> > Hi everyone, > >> >> >> > >> >> >> I am trying to get values owned by other processors of a parallel > >> >> matrix. > >> >> >> > >> >> >> I tried to create a sub-matrix by using MatGetSubMatrices(), and > >> >> >> then > >> >> >> MatGetRow() on the sub-matrix. But MatGetRow() give me the > following > >> >> >> error message: > >> >> >> > >> >> >> ===== Error message begins ===== > >> >> >> > >> >> >> No support for this operation for this object type > >> >> >> only local rows > >> >> >> > >> >> >> ===== Error message ends ===== > >> >> >> > >> >> >> The parallel matrix is a parallel dense matrix. The ISs for > >> >> >> MatGetSubMatrices() are created using ISCreateGeneral() and > >> >> >> PETSC_COMM_WORLD. The row IS is sorted by ISSort(). > >> >> >> > >> >> >> What did I mistake while using the above functions? Is there a > >> >> >> better > >> >> >> way to get access to matrix values owned by other processor? > >> >> >> > >> >> >> Thanks! > >> >> >> > >> >> >> HU Yaoyu > >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benfogelson at gmail.com Fri Apr 15 12:17:37 2016 From: benfogelson at gmail.com (Ben Fogelson) Date: Fri, 15 Apr 2016 17:17:37 +0000 Subject: [petsc-users] Petsc4py DMREDUNDANT or equivalent functionality Message-ID: Hi everyone, I'm trying to use a timestepper to solve a nonlinear system with Petsc4py. This system includes a grid variables which I can manage using a DMDA, but it also includes a small number of globally coupled scalar values. This seems like what DMREDUNDANT was designed for: make DMDA and DMREDUNDANT objects, stick them into a composite, and use the global vector generated by that composite DM as the state variable in my nonlinear solver. As best I can tell, Petsc4py doesn't have a method like DM().createRedundant()--perhaps it this DM just isn't implemented? Can anyone recommend an alternate approach to getting the same functionality? Thanks, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Apr 15 13:48:41 2016 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 15 Apr 2016 21:48:41 +0300 Subject: [petsc-users] Petsc4py DMREDUNDANT or equivalent functionality In-Reply-To: References: Message-ID: On 15 April 2016 at 20:17, Ben Fogelson wrote: > As best I can tell, Petsc4py doesn't have a method like > DM().createRedundant()--perhaps it this DM just isn't implemented? Yes, that is the case. > Can > anyone recommend an alternate approach to getting the same functionality? I guess the best way to go would be to implement the missing DM. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From mkury at berkeley.edu Fri Apr 15 18:23:20 2016 From: mkury at berkeley.edu (Matthew Kury) Date: Fri, 15 Apr 2016 16:23:20 -0700 Subject: [petsc-users] DMPlexCreateLocalVec Within a function Message-ID: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> Hello, I have done some reading and I have not been able to find out what the proper way to create and use vectors within a function. If I have a function that passes in dm and V then: +++ DMPlexGetLocalVec(dm, &VLocal) ; DMGlobalToLocalBegin(dm,V,INSERT_VALUES,VLocal); DMGlobalToLocalEnd(dm,V,INSERT_VALUES,VLocal) ; VecGetArray(VLoc,VLocalArray); ?Stuff with VLocalArray? VecRestoreArray(VLoc,VLocalArray); DMLocalToGlobalBegin(dm,VLocal,ADD_VALUES,V); DMLocalToGlobalEnd(dm,VLocal,ADD_VALUES,V); DMRestoreLocalVector(dm,&VLocal); +++ The second time I call the function I get errors with the VecGetArray() function. I would appreciate any advice as to what is going wrong. Thank you, Matthew From jed at jedbrown.org Sat Apr 16 03:21:41 2016 From: jed at jedbrown.org (Jed Brown) Date: Sat, 16 Apr 2016 09:21:41 +0100 Subject: [petsc-users] DMPlexCreateLocalVec Within a function In-Reply-To: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> References: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> Message-ID: <87shymdkve.fsf@jedbrown.org> Matthew Kury writes: > Hello, > > I have done some reading and I have not been able to find out what the proper way to create and use vectors within a function. > > If I have a function that passes in dm and V then: > +++ > DMPlexGetLocalVec(dm, &VLocal) ; This function doesn't exist, and neither does DMPlexCreateLocalVec (from the subject of your email), but when I gaze into my crystal ball, it says you're probably spelling it DMGetLocalVector. Please send the full error message. > DMGlobalToLocalBegin(dm,V,INSERT_VALUES,VLocal); > DMGlobalToLocalEnd(dm,V,INSERT_VALUES,VLocal) ; > > VecGetArray(VLoc,VLocalArray); > > ?Stuff with VLocalArray? > > VecRestoreArray(VLoc,VLocalArray); > > DMLocalToGlobalBegin(dm,VLocal,ADD_VALUES,V); > DMLocalToGlobalEnd(dm,VLocal,ADD_VALUES,V); > > DMRestoreLocalVector(dm,&VLocal); > +++ > > The second time I call the function I get errors with the VecGetArray() function. I would appreciate any advice as to what is going wrong. > > Thank you, > > Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mkury at berkeley.edu Sat Apr 16 03:38:29 2016 From: mkury at berkeley.edu (Matthew Kury) Date: Sat, 16 Apr 2016 01:38:29 -0700 Subject: [petsc-users] DMPlexCreateLocalVec Within a function In-Reply-To: <87shymdkve.fsf@jedbrown.org> References: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> <87shymdkve.fsf@jedbrown.org> Message-ID: Yes you are correct I am sorry about that. I meant DMGetLocalVector, and the ilk. here is a dummy function I am working on to try to figure out what is going wrong: ///////////////////////////////////////////// // Get the information to operate on // Make Local Versions Vec XLoc; DM Xdm; PetscSection XSec; PetscReal *XPoints=NULL; PetscErrorCode ierr; ierr = DMGetLocalVector(msh.dm,&XLoc);chk; ierr = DMGetCoordinateDM(msh.dm,&Xdm);chk; ierr = DMGetCoordinatesLocal(msh.dm, &XLoc);chk; VecView(XLoc,pview); ierr = DMDAVecGetArrayRead(Xdm,XLoc,&XPoints);chk; ierr = DMDAVecRestoreArrayRead(Xdm,XLoc,&XPoints);chk; ierr = DMRestoreLocalVector(Xdm,&XLoc);chk; //////////// end of function the error I get that seems to appear after the 2nd call or so is: 0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Wrong type of object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 [0]PETSC ERROR: ./net.out on a arch-darwin-c-debug named Matthews-MacBook-Pro-2.local by mkury Sat Apr 16 01:33:34 2016 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-petsc4py --download-triangle --download-chaco --download-metis --download-parmetis --download-ctetgen [0]PETSC ERROR: #2294 DMDAVecGetArrayRead() line 299 in /usr/local/petsc-3.6.1/src/dm/impls/da/dagetarray.c [0]PETSC ERROR: #2295 Element_Investigation_TEST() line 248 in /Users/mkury/Documents/Code/petFEM/src/elementFunctions.c thanks again, Matt On Apr 16, 2016, at 1:21 AM, Jed Brown wrote: > Matthew Kury writes: > >> Hello, >> >> I have done some reading and I have not been able to find out what the proper way to create and use vectors within a function. >> >> If I have a function that passes in dm and V then: >> +++ >> DMPlexGetLocalVec(dm, &VLocal) ; > > This function doesn't exist, and neither does DMPlexCreateLocalVec (from > the subject of your email), but when I gaze into my crystal ball, it > says you're probably spelling it DMGetLocalVector. Please send the full > error message. > >> DMGlobalToLocalBegin(dm,V,INSERT_VALUES,VLocal); >> DMGlobalToLocalEnd(dm,V,INSERT_VALUES,VLocal) ; >> >> VecGetArray(VLoc,VLocalArray); >> >> ?Stuff with VLocalArray? >> >> VecRestoreArray(VLoc,VLocalArray); >> >> DMLocalToGlobalBegin(dm,VLocal,ADD_VALUES,V); >> DMLocalToGlobalEnd(dm,VLocal,ADD_VALUES,V); >> >> DMRestoreLocalVector(dm,&VLocal); >> +++ >> >> The second time I call the function I get errors with the VecGetArray() function. I would appreciate any advice as to what is going wrong. >> >> Thank you, >> >> Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Apr 16 03:51:12 2016 From: jed at jedbrown.org (Jed Brown) Date: Sat, 16 Apr 2016 09:51:12 +0100 Subject: [petsc-users] DMPlexCreateLocalVec Within a function In-Reply-To: References: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> <87shymdkve.fsf@jedbrown.org> Message-ID: <87potqdji7.fsf@jedbrown.org> Matthew Kury writes: > Yes you are correct I am sorry about that. I meant DMGetLocalVector, and the ilk. > > here is a dummy function I am working on to try to figure out what is going wrong: > ///////////////////////////////////////////// > // Get the information to operate on > // Make Local Versions > Vec XLoc; > DM Xdm; > PetscSection XSec; > PetscReal *XPoints=NULL; > > PetscErrorCode ierr; > > > ierr = DMGetLocalVector(msh.dm,&XLoc);chk; Here, you get XLoc from msh.dm. > ierr = DMGetCoordinateDM(msh.dm,&Xdm);chk; > ierr = DMGetCoordinatesLocal(msh.dm, &XLoc);chk; > VecView(XLoc,pview); > > ierr = DMDAVecGetArrayRead(Xdm,XLoc,&XPoints);chk; > ierr = DMDAVecRestoreArrayRead(Xdm,XLoc,&XPoints);chk; > > ierr = DMRestoreLocalVector(Xdm,&XLoc);chk; Here, you return it to Xdm. > //////////// end of function > > the error I get that seems to appear after the 2nd call or so is: > 0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Wrong type of object: Parameter # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./net.out on a arch-darwin-c-debug named Matthews-MacBook-Pro-2.local by mkury Sat Apr 16 01:33:34 2016 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-petsc4py --download-triangle --download-chaco --download-metis --download-parmetis --download-ctetgen > [0]PETSC ERROR: #2294 DMDAVecGetArrayRead() line 299 in /usr/local/petsc-3.6.1/src/dm/impls/da/dagetarray.c This #2294 is nowhere near the top of your stack. Please check error codes. Running in a debugger would help you understand what is wrong. > [0]PETSC ERROR: #2295 Element_Investigation_TEST() line 248 in /Users/mkury/Documents/Code/petFEM/src/elementFunctions.c > > > > > > > thanks again, > > Matt > On Apr 16, 2016, at 1:21 AM, Jed Brown wrote: > >> Matthew Kury writes: >> >>> Hello, >>> >>> I have done some reading and I have not been able to find out what the proper way to create and use vectors within a function. >>> >>> If I have a function that passes in dm and V then: >>> +++ >>> DMPlexGetLocalVec(dm, &VLocal) ; >> >> This function doesn't exist, and neither does DMPlexCreateLocalVec (from >> the subject of your email), but when I gaze into my crystal ball, it >> says you're probably spelling it DMGetLocalVector. Please send the full >> error message. >> >>> DMGlobalToLocalBegin(dm,V,INSERT_VALUES,VLocal); >>> DMGlobalToLocalEnd(dm,V,INSERT_VALUES,VLocal) ; >>> >>> VecGetArray(VLoc,VLocalArray); >>> >>> ?Stuff with VLocalArray? >>> >>> VecRestoreArray(VLoc,VLocalArray); >>> >>> DMLocalToGlobalBegin(dm,VLocal,ADD_VALUES,V); >>> DMLocalToGlobalEnd(dm,VLocal,ADD_VALUES,V); >>> >>> DMRestoreLocalVector(dm,&VLocal); >>> +++ >>> >>> The second time I call the function I get errors with the VecGetArray() function. I would appreciate any advice as to what is going wrong. >>> >>> Thank you, >>> >>> Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Sat Apr 16 09:19:36 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 16 Apr 2016 10:19:36 -0400 Subject: [petsc-users] DMPlexCreateLocalVec Within a function In-Reply-To: <87potqdji7.fsf@jedbrown.org> References: <44D6FE80-7AF3-406F-8D7D-F94BD93B3A39@berkeley.edu> <87shymdkve.fsf@jedbrown.org> <87potqdji7.fsf@jedbrown.org> Message-ID: On Sat, Apr 16, 2016 at 4:51 AM, Jed Brown wrote: > Matthew Kury writes: > > > Yes you are correct I am sorry about that. I meant DMGetLocalVector, and > the ilk. > > > > here is a dummy function I am working on to try to figure out what is > going wrong: > > ///////////////////////////////////////////// > > // Get the information to operate on > > // Make Local Versions > > Vec XLoc; > > DM Xdm; > > PetscSection XSec; > > PetscReal *XPoints=NULL; > > > > PetscErrorCode ierr; > > > > > > ierr = DMGetLocalVector(msh.dm,&XLoc);chk; > > Here, you get XLoc from msh.dm. > > > ierr = DMGetCoordinateDM(msh.dm,&Xdm);chk; > > ierr = DMGetCoordinatesLocal(msh.dm, &XLoc);chk; > > VecView(XLoc,pview); > > > > ierr = DMDAVecGetArrayRead(Xdm,XLoc,&XPoints);chk; > Also here Xdm is supposed to be a DMDA. Is it? Matt > > ierr = DMDAVecRestoreArrayRead(Xdm,XLoc,&XPoints);chk; > > > > ierr = DMRestoreLocalVector(Xdm,&XLoc);chk; > > Here, you return it to Xdm. > > > //////////// end of function > > > > the error I get that seems to appear after the 2nd call or so is: > > 0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Wrong type of object: Parameter # 2 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > > [0]PETSC ERROR: ./net.out on a arch-darwin-c-debug named > Matthews-MacBook-Pro-2.local by mkury Sat Apr 16 01:33:34 2016 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > --download-petsc4py --download-triangle --download-chaco --download-metis > --download-parmetis --download-ctetgen > > [0]PETSC ERROR: #2294 DMDAVecGetArrayRead() line 299 in > /usr/local/petsc-3.6.1/src/dm/impls/da/dagetarray.c > > This #2294 is nowhere near the top of your stack. Please check error > codes. Running in a debugger would help you understand what is wrong. > > > [0]PETSC ERROR: #2295 Element_Investigation_TEST() line 248 in > /Users/mkury/Documents/Code/petFEM/src/elementFunctions.c > > > > > > > > > > > > > > thanks again, > > > > Matt > > On Apr 16, 2016, at 1:21 AM, Jed Brown wrote: > > > >> Matthew Kury writes: > >> > >>> Hello, > >>> > >>> I have done some reading and I have not been able to find out what > the proper way to create and use vectors within a function. > >>> > >>> If I have a function that passes in dm and V then: > >>> +++ > >>> DMPlexGetLocalVec(dm, &VLocal) ; > >> > >> This function doesn't exist, and neither does DMPlexCreateLocalVec (from > >> the subject of your email), but when I gaze into my crystal ball, it > >> says you're probably spelling it DMGetLocalVector. Please send the full > >> error message. > >> > >>> DMGlobalToLocalBegin(dm,V,INSERT_VALUES,VLocal); > >>> DMGlobalToLocalEnd(dm,V,INSERT_VALUES,VLocal) ; > >>> > >>> VecGetArray(VLoc,VLocalArray); > >>> > >>> ?Stuff with VLocalArray? > >>> > >>> VecRestoreArray(VLoc,VLocalArray); > >>> > >>> DMLocalToGlobalBegin(dm,VLocal,ADD_VALUES,V); > >>> DMLocalToGlobalEnd(dm,VLocal,ADD_VALUES,V); > >>> > >>> DMRestoreLocalVector(dm,&VLocal); > >>> +++ > >>> > >>> The second time I call the function I get errors with the > VecGetArray() function. I would appreciate any advice as to what is going > wrong. > >>> > >>> Thank you, > >>> > >>> Matthew > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgbk2008 at gmail.com Sat Apr 16 17:54:10 2016 From: hgbk2008 at gmail.com (Hoang Giang Bui) Date: Sun, 17 Apr 2016 00:54:10 +0200 Subject: [petsc-users] fieldsplit preconditioner for indefinite matrix Message-ID: Hello I'm solving an indefinite problem arising from mesh tying/contact using Lagrange multiplier, the matrix has the form K = [A P^T P 0] I used the FIELDSPLIT preconditioner with one field is the main variable (displacement) and the other field for dual variable (Lagrange multiplier). The block size for each field is 3. According to the manual, I first chose the preconditioner based on Schur complement to treat this problem. The parameters used for the solve is -ksp_type gmres -ksp_max_it 300 -ksp_gmres_restart 300 -ksp_gmres_modifiedgramschmidt -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type diag -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_detect_saddle_point -fieldsplit_u_pc_type hypre -fieldsplit_u_pc_hypre_type boomeramg -fieldsplit_u_pc_hypre_boomeramg_coarsen_type PMIS -fieldsplit_lu_pc_type hypre -fieldsplit_lu_pc_hypre_type boomeramg -fieldsplit_lu_pc_hypre_boomeramg_coarsen_type PMIS For the test case, a small problem is solved on 2 processes. Due to the decomposition, the contact only happens in 1 proc, so the size of Lagrange multiplier dofs on proc 0 is 0. 0: mIndexU.size(): 80490 0: mIndexLU.size(): 0 1: mIndexU.size(): 103836 1: mIndexLU.size(): 2583 However, with this setup the solver takes very long at KSPSolve before going to iteration, and the first iteration seems forever so I have to stop the calculation. I guessed that the solver takes time to compute the Schur complement, but according to the manual only the diagonal of A is used to approximate the Schur complement, so it should not take long to compute this. Note that I ran the same problem with direct solver (MUMPS) and it's able to produce the valid results. The parameter for the solve is pretty standard -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps Hence the matrix/rhs must not have any problem here. Do you have any idea or suggestion for this case? Giang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 16 18:15:08 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 16 Apr 2016 19:15:08 -0400 Subject: [petsc-users] fieldsplit preconditioner for indefinite matrix In-Reply-To: References: Message-ID: On Sat, Apr 16, 2016 at 6:54 PM, Hoang Giang Bui wrote: > Hello > > I'm solving an indefinite problem arising from mesh tying/contact using > Lagrange multiplier, the matrix has the form > > K = [A P^T > P 0] > > I used the FIELDSPLIT preconditioner with one field is the main variable > (displacement) and the other field for dual variable (Lagrange multiplier). > The block size for each field is 3. According to the manual, I first chose > the preconditioner based on Schur complement to treat this problem. > For any solver question, please send us the output of -ksp_view -ksp_monitor_true_residual -ksp_converged_reason However, I will comment below > The parameters used for the solve is > -ksp_type gmres > You need 'fgmres' here with the options you have below. > -ksp_max_it 300 > -ksp_gmres_restart 300 > -ksp_gmres_modifiedgramschmidt > -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type diag > -pc_fieldsplit_schur_precondition selfp > It could be taking time in the MatMatMult() here if that matrix is dense. Is there any reason to believe that is a good preconditioner for your problem? > -pc_fieldsplit_detect_saddle_point > -fieldsplit_u_pc_type hypre > I would just use MUMPS here to start, especially if it works on the whole problem. Same with the one below. Matt > -fieldsplit_u_pc_hypre_type boomeramg > -fieldsplit_u_pc_hypre_boomeramg_coarsen_type PMIS > -fieldsplit_lu_pc_type hypre > -fieldsplit_lu_pc_hypre_type boomeramg > -fieldsplit_lu_pc_hypre_boomeramg_coarsen_type PMIS > > For the test case, a small problem is solved on 2 processes. Due to the > decomposition, the contact only happens in 1 proc, so the size of Lagrange > multiplier dofs on proc 0 is 0. > > 0: mIndexU.size(): 80490 > 0: mIndexLU.size(): 0 > 1: mIndexU.size(): 103836 > 1: mIndexLU.size(): 2583 > > However, with this setup the solver takes very long at KSPSolve before > going to iteration, and the first iteration seems forever so I have to stop > the calculation. I guessed that the solver takes time to compute the Schur > complement, but according to the manual only the diagonal of A is used to > approximate the Schur complement, so it should not take long to compute > this. > > Note that I ran the same problem with direct solver (MUMPS) and it's able > to produce the valid results. The parameter for the solve is pretty standard > -ksp_type preonly > -pc_type lu > -pc_factor_mat_solver_package mumps > > Hence the matrix/rhs must not have any problem here. Do you have any idea > or suggestion for this case? > > > Giang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgbk2008 at gmail.com Sun Apr 17 03:25:08 2016 From: hgbk2008 at gmail.com (Hoang Giang Bui) Date: Sun, 17 Apr 2016 10:25:08 +0200 Subject: [petsc-users] fieldsplit preconditioner for indefinite matrix In-Reply-To: References: Message-ID: > It could be taking time in the MatMatMult() here if that matrix is dense. > Is there any reason to > believe that is a good preconditioner for your problem? > This is the first approach to the problem, so I chose the most simple setting. Do you have any other recommendation? For any solver question, please send us the output of > > -ksp_view -ksp_monitor_true_residual -ksp_converged_reason > > I sent here the full output (after changed to fgmres), again it takes long at the first iteration but after that, it does not converge -ksp_type fgmres -ksp_max_it 300 -ksp_gmres_restart 300 -ksp_gmres_modifiedgramschmidt -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type diag -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_detect_saddle_point -fieldsplit_u_ksp_type preonly -fieldsplit_u_pc_type lu -fieldsplit_u_pc_factor_mat_solver_package mumps -fieldsplit_lu_ksp_type preonly -fieldsplit_lu_pc_type lu -fieldsplit_lu_pc_factor_mat_solver_package mumps 0 KSP unpreconditioned resid norm 3.037772453815e+06 true resid norm 3.037772453815e+06 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 3.024368791893e+06 true resid norm 3.024368791296e+06 ||r(i)||/||b|| 9.955876673705e-01 2 KSP unpreconditioned resid norm 3.008534454663e+06 true resid norm 3.008534454904e+06 ||r(i)||/||b|| 9.903751846607e-01 3 KSP unpreconditioned resid norm 4.633282412600e+02 true resid norm 4.607539866185e+02 ||r(i)||/||b|| 1.516749505184e-04 4 KSP unpreconditioned resid norm 4.630592911836e+02 true resid norm 4.605625897903e+02 ||r(i)||/||b|| 1.516119448683e-04 5 KSP unpreconditioned resid norm 2.145735509629e+02 true resid norm 2.111697416683e+02 ||r(i)||/||b|| 6.951466736857e-05 6 KSP unpreconditioned resid norm 2.145734219762e+02 true resid norm 2.112001242378e+02 ||r(i)||/||b|| 6.952466896346e-05 7 KSP unpreconditioned resid norm 1.892914067411e+02 true resid norm 1.831020928502e+02 ||r(i)||/||b|| 6.027511791420e-05 8 KSP unpreconditioned resid norm 1.892906351597e+02 true resid norm 1.831422357767e+02 ||r(i)||/||b|| 6.028833250718e-05 9 KSP unpreconditioned resid norm 1.891426729822e+02 true resid norm 1.835600473014e+02 ||r(i)||/||b|| 6.042587128964e-05 10 KSP unpreconditioned resid norm 1.891425181679e+02 true resid norm 1.855772578041e+02 ||r(i)||/||b|| 6.108991395027e-05 11 KSP unpreconditioned resid norm 1.891417382057e+02 true resid norm 1.833302669042e+02 ||r(i)||/||b|| 6.035023020699e-05 12 KSP unpreconditioned resid norm 1.891414749001e+02 true resid norm 1.827923591605e+02 ||r(i)||/||b|| 6.017315712076e-05 13 KSP unpreconditioned resid norm 1.891414702834e+02 true resid norm 1.849895606391e+02 ||r(i)||/||b|| 6.089645075515e-05 14 KSP unpreconditioned resid norm 1.891414687385e+02 true resid norm 1.852700958573e+02 ||r(i)||/||b|| 6.098879974523e-05 15 KSP unpreconditioned resid norm 1.891399614701e+02 true resid norm 1.817034334576e+02 ||r(i)||/||b|| 5.981469521503e-05 16 KSP unpreconditioned resid norm 1.891393964580e+02 true resid norm 1.823173574739e+02 ||r(i)||/||b|| 6.001679199012e-05 17 KSP unpreconditioned resid norm 1.890868604964e+02 true resid norm 1.834754811775e+02 ||r(i)||/||b|| 6.039803308740e-05 18 KSP unpreconditioned resid norm 1.888442703508e+02 true resid norm 1.852079421560e+02 ||r(i)||/||b|| 6.096833945658e-05 19 KSP unpreconditioned resid norm 1.888131521870e+02 true resid norm 1.810111295757e+02 ||r(i)||/||b|| 5.958679668335e-05 20 KSP unpreconditioned resid norm 1.888038471618e+02 true resid norm 1.814080717355e+02 ||r(i)||/||b|| 5.971746550920e-05 21 KSP unpreconditioned resid norm 1.885794485272e+02 true resid norm 1.843223565278e+02 ||r(i)||/||b|| 6.067681478129e-05 22 KSP unpreconditioned resid norm 1.884898771362e+02 true resid norm 1.842766260526e+02 ||r(i)||/||b|| 6.066176083110e-05 23 KSP unpreconditioned resid norm 1.884840498049e+02 true resid norm 1.813011285152e+02 ||r(i)||/||b|| 5.968226102238e-05 24 KSP unpreconditioned resid norm 1.884105698955e+02 true resid norm 1.811513025118e+02 ||r(i)||/||b|| 5.963294001309e-05 25 KSP unpreconditioned resid norm 1.881392557375e+02 true resid norm 1.835706567649e+02 ||r(i)||/||b|| 6.042936380386e-05 26 KSP unpreconditioned resid norm 1.881234481250e+02 true resid norm 1.843633799886e+02 ||r(i)||/||b|| 6.069031923609e-05 27 KSP unpreconditioned resid norm 1.852572648925e+02 true resid norm 1.791532195358e+02 ||r(i)||/||b|| 5.897519391579e-05 28 KSP unpreconditioned resid norm 1.852177694782e+02 true resid norm 1.800935543889e+02 ||r(i)||/||b|| 5.928474141066e-05 29 KSP unpreconditioned resid norm 1.844720976468e+02 true resid norm 1.806835899755e+02 ||r(i)||/||b|| 5.947897438749e-05 30 KSP unpreconditioned resid norm 1.843525447108e+02 true resid norm 1.811351238391e+02 ||r(i)||/||b|| 5.962761417881e-05 31 KSP unpreconditioned resid norm 1.834262885149e+02 true resid norm 1.778584233423e+02 ||r(i)||/||b|| 5.854896179565e-05 32 KSP unpreconditioned resid norm 1.833523213017e+02 true resid norm 1.773290649733e+02 ||r(i)||/||b|| 5.837470306591e-05 33 KSP unpreconditioned resid norm 1.821645929344e+02 true resid norm 1.781151248933e+02 ||r(i)||/||b|| 5.863346501467e-05 34 KSP unpreconditioned resid norm 1.820831279534e+02 true resid norm 1.789778939067e+02 ||r(i)||/||b|| 5.891747872094e-05 35 KSP unpreconditioned resid norm 1.814860919375e+02 true resid norm 1.757339506869e+02 ||r(i)||/||b|| 5.784960965928e-05 36 KSP unpreconditioned resid norm 1.812512010159e+02 true resid norm 1.764086437459e+02 ||r(i)||/||b|| 5.807171090922e-05 37 KSP unpreconditioned resid norm 1.804298150360e+02 true resid norm 1.780147196442e+02 ||r(i)||/||b|| 5.860041275333e-05 38 KSP unpreconditioned resid norm 1.799675012847e+02 true resid norm 1.780554543786e+02 ||r(i)||/||b|| 5.861382216269e-05 39 KSP unpreconditioned resid norm 1.793156052097e+02 true resid norm 1.747985717965e+02 ||r(i)||/||b|| 5.754169361071e-05 40 KSP unpreconditioned resid norm 1.789109248325e+02 true resid norm 1.734086984879e+02 ||r(i)||/||b|| 5.708416319009e-05 41 KSP unpreconditioned resid norm 1.788931581371e+02 true resid norm 1.766103879126e+02 ||r(i)||/||b|| 5.813812278494e-05 42 KSP unpreconditioned resid norm 1.785522436483e+02 true resid norm 1.762597032909e+02 ||r(i)||/||b|| 5.802268141233e-05 43 KSP unpreconditioned resid norm 1.783317950582e+02 true resid norm 1.752774080448e+02 ||r(i)||/||b|| 5.769932103530e-05 44 KSP unpreconditioned resid norm 1.782832982797e+02 true resid norm 1.741667594885e+02 ||r(i)||/||b|| 5.733370821430e-05 45 KSP unpreconditioned resid norm 1.781302427969e+02 true resid norm 1.760315735899e+02 ||r(i)||/||b|| 5.794758372005e-05 46 KSP unpreconditioned resid norm 1.780557458973e+02 true resid norm 1.757279911034e+02 ||r(i)||/||b|| 5.784764783244e-05 47 KSP unpreconditioned resid norm 1.774691940686e+02 true resid norm 1.729436852773e+02 ||r(i)||/||b|| 5.693108615167e-05 48 KSP unpreconditioned resid norm 1.771436357084e+02 true resid norm 1.734001323688e+02 ||r(i)||/||b|| 5.708134332148e-05 49 KSP unpreconditioned resid norm 1.756105727417e+02 true resid norm 1.740222172981e+02 ||r(i)||/||b|| 5.728612657594e-05 50 KSP unpreconditioned resid norm 1.756011794480e+02 true resid norm 1.736979026533e+02 ||r(i)||/||b|| 5.717936589858e-05 51 KSP unpreconditioned resid norm 1.751096154950e+02 true resid norm 1.713154407940e+02 ||r(i)||/||b|| 5.639508666256e-05 52 KSP unpreconditioned resid norm 1.712639990486e+02 true resid norm 1.684444278579e+02 ||r(i)||/||b|| 5.544998199137e-05 53 KSP unpreconditioned resid norm 1.710183053728e+02 true resid norm 1.692712952670e+02 ||r(i)||/||b|| 5.572217729951e-05 54 KSP unpreconditioned resid norm 1.655470115849e+02 true resid norm 1.631767858448e+02 ||r(i)||/||b|| 5.371593439788e-05 55 KSP unpreconditioned resid norm 1.648313805392e+02 true resid norm 1.617509396670e+02 ||r(i)||/||b|| 5.324656211951e-05 56 KSP unpreconditioned resid norm 1.643417766012e+02 true resid norm 1.614766932468e+02 ||r(i)||/||b|| 5.315628332992e-05 57 KSP unpreconditioned resid norm 1.643165564782e+02 true resid norm 1.611660297521e+02 ||r(i)||/||b|| 5.305401645527e-05 58 KSP unpreconditioned resid norm 1.639561245303e+02 true resid norm 1.616105878219e+02 ||r(i)||/||b|| 5.320035989496e-05 59 KSP unpreconditioned resid norm 1.636859175366e+02 true resid norm 1.601704798933e+02 ||r(i)||/||b|| 5.272629281109e-05 60 KSP unpreconditioned resid norm 1.633269681891e+02 true resid norm 1.603249334191e+02 ||r(i)||/||b|| 5.277713714789e-05 61 KSP unpreconditioned resid norm 1.633257086864e+02 true resid norm 1.602922744638e+02 ||r(i)||/||b|| 5.276638619280e-05 62 KSP unpreconditioned resid norm 1.629449737049e+02 true resid norm 1.605812790996e+02 ||r(i)||/||b|| 5.286152321842e-05 63 KSP unpreconditioned resid norm 1.629422151091e+02 true resid norm 1.589656479615e+02 ||r(i)||/||b|| 5.232967589850e-05 64 KSP unpreconditioned resid norm 1.624767340901e+02 true resid norm 1.601925152173e+02 ||r(i)||/||b|| 5.273354658809e-05 65 KSP unpreconditioned resid norm 1.614000473427e+02 true resid norm 1.600055285874e+02 ||r(i)||/||b|| 5.267199272497e-05 66 KSP unpreconditioned resid norm 1.599192711038e+02 true resid norm 1.602225820054e+02 ||r(i)||/||b|| 5.274344423136e-05 67 KSP unpreconditioned resid norm 1.562002802473e+02 true resid norm 1.582069452329e+02 ||r(i)||/||b|| 5.207991962471e-05 68 KSP unpreconditioned resid norm 1.552436010567e+02 true resid norm 1.584249134588e+02 ||r(i)||/||b|| 5.215167227548e-05 69 KSP unpreconditioned resid norm 1.507627069906e+02 true resid norm 1.530713322210e+02 ||r(i)||/||b|| 5.038933447066e-05 70 KSP unpreconditioned resid norm 1.503802419288e+02 true resid norm 1.526772130725e+02 ||r(i)||/||b|| 5.025959494786e-05 71 KSP unpreconditioned resid norm 1.483645684459e+02 true resid norm 1.509599328686e+02 ||r(i)||/||b|| 4.969428591633e-05 72 KSP unpreconditioned resid norm 1.481979533059e+02 true resid norm 1.535340885300e+02 ||r(i)||/||b|| 5.054166856281e-05 73 KSP unpreconditioned resid norm 1.481400704979e+02 true resid norm 1.509082933863e+02 ||r(i)||/||b|| 4.967728678847e-05 74 KSP unpreconditioned resid norm 1.481132272449e+02 true resid norm 1.513298398754e+02 ||r(i)||/||b|| 4.981605507858e-05 75 KSP unpreconditioned resid norm 1.481101708026e+02 true resid norm 1.502466334943e+02 ||r(i)||/||b|| 4.945947590828e-05 76 KSP unpreconditioned resid norm 1.481010335860e+02 true resid norm 1.533384206564e+02 ||r(i)||/||b|| 5.047725693339e-05 77 KSP unpreconditioned resid norm 1.480865328511e+02 true resid norm 1.508354096349e+02 ||r(i)||/||b|| 4.965329428986e-05 78 KSP unpreconditioned resid norm 1.480582653674e+02 true resid norm 1.493335938981e+02 ||r(i)||/||b|| 4.915891370027e-05 79 KSP unpreconditioned resid norm 1.480031554288e+02 true resid norm 1.505131104808e+02 ||r(i)||/||b|| 4.954719708903e-05 80 KSP unpreconditioned resid norm 1.479574822714e+02 true resid norm 1.540226621640e+02 ||r(i)||/||b|| 5.070250142355e-05 81 KSP unpreconditioned resid norm 1.479574535946e+02 true resid norm 1.498368142318e+02 ||r(i)||/||b|| 4.932456808727e-05 82 KSP unpreconditioned resid norm 1.479436001532e+02 true resid norm 1.512355315895e+02 ||r(i)||/||b|| 4.978500986785e-05 83 KSP unpreconditioned resid norm 1.479410419985e+02 true resid norm 1.513924042216e+02 ||r(i)||/||b|| 4.983665054686e-05 84 KSP unpreconditioned resid norm 1.477087197314e+02 true resid norm 1.519847216835e+02 ||r(i)||/||b|| 5.003163469095e-05 85 KSP unpreconditioned resid norm 1.477081559094e+02 true resid norm 1.507153721984e+02 ||r(i)||/||b|| 4.961377933660e-05 86 KSP unpreconditioned resid norm 1.476420890986e+02 true resid norm 1.512147907360e+02 ||r(i)||/||b|| 4.977818221576e-05 87 KSP unpreconditioned resid norm 1.476086929880e+02 true resid norm 1.508513380647e+02 ||r(i)||/||b|| 4.965853774704e-05 88 KSP unpreconditioned resid norm 1.475729830724e+02 true resid norm 1.521640656963e+02 ||r(i)||/||b|| 5.009067269183e-05 89 KSP unpreconditioned resid norm 1.472338605465e+02 true resid norm 1.506094588356e+02 ||r(i)||/||b|| 4.957891386713e-05 90 KSP unpreconditioned resid norm 1.472079944867e+02 true resid norm 1.504582871439e+02 ||r(i)||/||b|| 4.952914987262e-05 91 KSP unpreconditioned resid norm 1.469363056078e+02 true resid norm 1.506425446156e+02 ||r(i)||/||b|| 4.958980532804e-05 92 KSP unpreconditioned resid norm 1.469110799022e+02 true resid norm 1.509842019134e+02 ||r(i)||/||b|| 4.970227500870e-05 93 KSP unpreconditioned resid norm 1.468779696240e+02 true resid norm 1.501105195969e+02 ||r(i)||/||b|| 4.941466876770e-05 94 KSP unpreconditioned resid norm 1.468777757710e+02 true resid norm 1.491460779150e+02 ||r(i)||/||b|| 4.909718558007e-05 95 KSP unpreconditioned resid norm 1.468774588833e+02 true resid norm 1.519041612996e+02 ||r(i)||/||b|| 5.000511513258e-05 96 KSP unpreconditioned resid norm 1.468771672305e+02 true resid norm 1.508986277767e+02 ||r(i)||/||b|| 4.967410498018e-05 97 KSP unpreconditioned resid norm 1.468771086724e+02 true resid norm 1.500987040931e+02 ||r(i)||/||b|| 4.941077923878e-05 98 KSP unpreconditioned resid norm 1.468769529855e+02 true resid norm 1.509749203169e+02 ||r(i)||/||b|| 4.969921961314e-05 99 KSP unpreconditioned resid norm 1.468539019917e+02 true resid norm 1.505087391266e+02 ||r(i)||/||b|| 4.954575808916e-05 100 KSP unpreconditioned resid norm 1.468527260351e+02 true resid norm 1.519470484364e+02 ||r(i)||/||b|| 5.001923308823e-05 101 KSP unpreconditioned resid norm 1.468342327062e+02 true resid norm 1.489814197970e+02 ||r(i)||/||b|| 4.904298200804e-05 102 KSP unpreconditioned resid norm 1.468333201903e+02 true resid norm 1.491479405434e+02 ||r(i)||/||b|| 4.909779873608e-05 103 KSP unpreconditioned resid norm 1.468287736823e+02 true resid norm 1.496401088908e+02 ||r(i)||/||b|| 4.925981493540e-05 104 KSP unpreconditioned resid norm 1.468269778777e+02 true resid norm 1.509676608058e+02 ||r(i)||/||b|| 4.969682986500e-05 105 KSP unpreconditioned resid norm 1.468214752527e+02 true resid norm 1.500441644659e+02 ||r(i)||/||b|| 4.939282541636e-05 106 KSP unpreconditioned resid norm 1.468208033546e+02 true resid norm 1.510964155942e+02 ||r(i)||/||b|| 4.973921447094e-05 107 KSP unpreconditioned resid norm 1.467590018852e+02 true resid norm 1.512302088409e+02 ||r(i)||/||b|| 4.978325767980e-05 108 KSP unpreconditioned resid norm 1.467588908565e+02 true resid norm 1.501053278370e+02 ||r(i)||/||b|| 4.941295969963e-05 109 KSP unpreconditioned resid norm 1.467570731153e+02 true resid norm 1.485494378220e+02 ||r(i)||/||b|| 4.890077847519e-05 110 KSP unpreconditioned resid norm 1.467399860352e+02 true resid norm 1.504418099302e+02 ||r(i)||/||b|| 4.952372576205e-05 111 KSP unpreconditioned resid norm 1.467095654863e+02 true resid norm 1.507288583410e+02 ||r(i)||/||b|| 4.961821882075e-05 112 KSP unpreconditioned resid norm 1.467065865602e+02 true resid norm 1.517786399520e+02 ||r(i)||/||b|| 4.996379493842e-05 113 KSP unpreconditioned resid norm 1.466898232510e+02 true resid norm 1.491434236258e+02 ||r(i)||/||b|| 4.909631181838e-05 114 KSP unpreconditioned resid norm 1.466897921426e+02 true resid norm 1.505605420512e+02 ||r(i)||/||b|| 4.956281102033e-05 115 KSP unpreconditioned resid norm 1.466593121787e+02 true resid norm 1.500608650677e+02 ||r(i)||/||b|| 4.939832306376e-05 116 KSP unpreconditioned resid norm 1.466590894710e+02 true resid norm 1.503102560128e+02 ||r(i)||/||b|| 4.948041971478e-05 117 KSP unpreconditioned resid norm 1.465338856917e+02 true resid norm 1.501331730933e+02 ||r(i)||/||b|| 4.942212604002e-05 118 KSP unpreconditioned resid norm 1.464192893188e+02 true resid norm 1.505131429801e+02 ||r(i)||/||b|| 4.954720778744e-05 119 KSP unpreconditioned resid norm 1.463859793112e+02 true resid norm 1.504355712014e+02 ||r(i)||/||b|| 4.952167204377e-05 120 KSP unpreconditioned resid norm 1.459254939182e+02 true resid norm 1.526513923221e+02 ||r(i)||/||b|| 5.025109505170e-05 121 KSP unpreconditioned resid norm 1.456973020864e+02 true resid norm 1.496897691500e+02 ||r(i)||/||b|| 4.927616252562e-05 122 KSP unpreconditioned resid norm 1.456904663212e+02 true resid norm 1.488752755634e+02 ||r(i)||/||b|| 4.900804053853e-05 123 KSP unpreconditioned resid norm 1.449254956591e+02 true resid norm 1.494048196254e+02 ||r(i)||/||b|| 4.918236039628e-05 124 KSP unpreconditioned resid norm 1.448408616171e+02 true resid norm 1.507801939332e+02 ||r(i)||/||b|| 4.963511791142e-05 125 KSP unpreconditioned resid norm 1.447662934870e+02 true resid norm 1.495157701445e+02 ||r(i)||/||b|| 4.921888404010e-05 126 KSP unpreconditioned resid norm 1.446934748257e+02 true resid norm 1.511098625097e+02 ||r(i)||/||b|| 4.974364104196e-05 127 KSP unpreconditioned resid norm 1.446892504333e+02 true resid norm 1.493367018275e+02 ||r(i)||/||b|| 4.915993679512e-05 128 KSP unpreconditioned resid norm 1.446838883996e+02 true resid norm 1.510097796622e+02 ||r(i)||/||b|| 4.971069491153e-05 129 KSP unpreconditioned resid norm 1.446696373784e+02 true resid norm 1.463776964101e+02 ||r(i)||/||b|| 4.818586600396e-05 130 KSP unpreconditioned resid norm 1.446690766798e+02 true resid norm 1.495018999638e+02 ||r(i)||/||b|| 4.921431813499e-05 131 KSP unpreconditioned resid norm 1.446480744133e+02 true resid norm 1.499605592408e+02 ||r(i)||/||b|| 4.936530353102e-05 132 KSP unpreconditioned resid norm 1.446220543422e+02 true resid norm 1.498225445439e+02 ||r(i)||/||b|| 4.931987066895e-05 133 KSP unpreconditioned resid norm 1.446156526760e+02 true resid norm 1.481441673781e+02 ||r(i)||/||b|| 4.876736807329e-05 134 KSP unpreconditioned resid norm 1.446152477418e+02 true resid norm 1.501616466283e+02 ||r(i)||/||b|| 4.943149920257e-05 135 KSP unpreconditioned resid norm 1.445744489044e+02 true resid norm 1.505958339620e+02 ||r(i)||/||b|| 4.957442871432e-05 136 KSP unpreconditioned resid norm 1.445307936181e+02 true resid norm 1.502091787932e+02 ||r(i)||/||b|| 4.944714624841e-05 137 KSP unpreconditioned resid norm 1.444543817248e+02 true resid norm 1.491871661616e+02 ||r(i)||/||b|| 4.911071136162e-05 138 KSP unpreconditioned resid norm 1.444176915911e+02 true resid norm 1.478091693367e+02 ||r(i)||/||b|| 4.865709054379e-05 139 KSP unpreconditioned resid norm 1.444173719058e+02 true resid norm 1.495962731374e+02 ||r(i)||/||b|| 4.924538470600e-05 140 KSP unpreconditioned resid norm 1.444075340820e+02 true resid norm 1.515103203654e+02 ||r(i)||/||b|| 4.987546719477e-05 141 KSP unpreconditioned resid norm 1.444050342939e+02 true resid norm 1.498145746307e+02 ||r(i)||/||b|| 4.931724706454e-05 142 KSP unpreconditioned resid norm 1.443757787691e+02 true resid norm 1.492291154146e+02 ||r(i)||/||b|| 4.912452057664e-05 143 KSP unpreconditioned resid norm 1.440588930707e+02 true resid norm 1.485032724987e+02 ||r(i)||/||b|| 4.888558137795e-05 144 KSP unpreconditioned resid norm 1.438299468441e+02 true resid norm 1.506129385276e+02 ||r(i)||/||b|| 4.958005934200e-05 145 KSP unpreconditioned resid norm 1.434543079403e+02 true resid norm 1.471733741230e+02 ||r(i)||/||b|| 4.844779402032e-05 146 KSP unpreconditioned resid norm 1.433157223870e+02 true resid norm 1.481025707968e+02 ||r(i)||/||b|| 4.875367495378e-05 147 KSP unpreconditioned resid norm 1.430111913458e+02 true resid norm 1.485000481919e+02 ||r(i)||/||b|| 4.888451997299e-05 148 KSP unpreconditioned resid norm 1.430056153071e+02 true resid norm 1.496425172884e+02 ||r(i)||/||b|| 4.926060775239e-05 149 KSP unpreconditioned resid norm 1.429327762233e+02 true resid norm 1.467613264791e+02 ||r(i)||/||b|| 4.831215264157e-05 150 KSP unpreconditioned resid norm 1.424230217603e+02 true resid norm 1.460277537447e+02 ||r(i)||/||b|| 4.807066887493e-05 151 KSP unpreconditioned resid norm 1.421912821676e+02 true resid norm 1.470486188164e+02 ||r(i)||/||b|| 4.840672599809e-05 152 KSP unpreconditioned resid norm 1.420344275315e+02 true resid norm 1.481536901943e+02 ||r(i)||/||b|| 4.877050287565e-05 153 KSP unpreconditioned resid norm 1.420071178597e+02 true resid norm 1.450813684108e+02 ||r(i)||/||b|| 4.775912963085e-05 154 KSP unpreconditioned resid norm 1.419367456470e+02 true resid norm 1.472052819440e+02 ||r(i)||/||b|| 4.845829771059e-05 155 KSP unpreconditioned resid norm 1.419032748919e+02 true resid norm 1.479193155584e+02 ||r(i)||/||b|| 4.869334942209e-05 156 KSP unpreconditioned resid norm 1.418899781440e+02 true resid norm 1.478677351572e+02 ||r(i)||/||b|| 4.867636974307e-05 157 KSP unpreconditioned resid norm 1.418895621075e+02 true resid norm 1.455168237674e+02 ||r(i)||/||b|| 4.790247656128e-05 158 KSP unpreconditioned resid norm 1.418061469023e+02 true resid norm 1.467147028974e+02 ||r(i)||/||b|| 4.829680469093e-05 159 KSP unpreconditioned resid norm 1.417948698213e+02 true resid norm 1.478376854834e+02 ||r(i)||/||b|| 4.866647773362e-05 160 KSP unpreconditioned resid norm 1.415166832324e+02 true resid norm 1.475436433192e+02 ||r(i)||/||b|| 4.856968241116e-05 161 KSP unpreconditioned resid norm 1.414939087573e+02 true resid norm 1.468361945080e+02 ||r(i)||/||b|| 4.833679834170e-05 162 KSP unpreconditioned resid norm 1.414544622036e+02 true resid norm 1.475730757600e+02 ||r(i)||/||b|| 4.857937123456e-05 163 KSP unpreconditioned resid norm 1.413780373982e+02 true resid norm 1.463891808066e+02 ||r(i)||/||b|| 4.818964653614e-05 164 KSP unpreconditioned resid norm 1.413741853943e+02 true resid norm 1.481999741168e+02 ||r(i)||/||b|| 4.878573901436e-05 165 KSP unpreconditioned resid norm 1.413725682642e+02 true resid norm 1.458413423932e+02 ||r(i)||/||b|| 4.800930438685e-05 166 KSP unpreconditioned resid norm 1.412970845566e+02 true resid norm 1.481492296610e+02 ||r(i)||/||b|| 4.876903451901e-05 167 KSP unpreconditioned resid norm 1.410100899597e+02 true resid norm 1.468338434340e+02 ||r(i)||/||b|| 4.833602439497e-05 168 KSP unpreconditioned resid norm 1.409983320599e+02 true resid norm 1.485378957202e+02 ||r(i)||/||b|| 4.889697894709e-05 169 KSP unpreconditioned resid norm 1.407688141293e+02 true resid norm 1.461003623074e+02 ||r(i)||/||b|| 4.809457078458e-05 170 KSP unpreconditioned resid norm 1.407072771004e+02 true resid norm 1.463217409181e+02 ||r(i)||/||b|| 4.816744609502e-05 171 KSP unpreconditioned resid norm 1.407069670790e+02 true resid norm 1.464695099700e+02 ||r(i)||/||b|| 4.821608997937e-05 172 KSP unpreconditioned resid norm 1.402361094414e+02 true resid norm 1.493786053835e+02 ||r(i)||/||b|| 4.917373096721e-05 173 KSP unpreconditioned resid norm 1.400618325859e+02 true resid norm 1.465475533254e+02 ||r(i)||/||b|| 4.824178096070e-05 174 KSP unpreconditioned resid norm 1.400573078320e+02 true resid norm 1.471993735980e+02 ||r(i)||/||b|| 4.845635275056e-05 175 KSP unpreconditioned resid norm 1.400258865388e+02 true resid norm 1.479779387468e+02 ||r(i)||/||b|| 4.871264750624e-05 176 KSP unpreconditioned resid norm 1.396589283831e+02 true resid norm 1.476626943974e+02 ||r(i)||/||b|| 4.860887266654e-05 177 KSP unpreconditioned resid norm 1.395796112440e+02 true resid norm 1.443093901655e+02 ||r(i)||/||b|| 4.750500320860e-05 178 KSP unpreconditioned resid norm 1.394749154493e+02 true resid norm 1.447914005206e+02 ||r(i)||/||b|| 4.766367551289e-05 179 KSP unpreconditioned resid norm 1.394476969416e+02 true resid norm 1.455635964329e+02 ||r(i)||/||b|| 4.791787358864e-05 180 KSP unpreconditioned resid norm 1.391990722790e+02 true resid norm 1.457511594620e+02 ||r(i)||/||b|| 4.797961719582e-05 181 KSP unpreconditioned resid norm 1.391686315799e+02 true resid norm 1.460567495143e+02 ||r(i)||/||b|| 4.808021395114e-05 182 KSP unpreconditioned resid norm 1.387654475794e+02 true resid norm 1.468215388414e+02 ||r(i)||/||b|| 4.833197386362e-05 183 KSP unpreconditioned resid norm 1.384925240232e+02 true resid norm 1.456091052791e+02 ||r(i)||/||b|| 4.793285458106e-05 184 KSP unpreconditioned resid norm 1.378003249970e+02 true resid norm 1.453421051371e+02 ||r(i)||/||b|| 4.784496118351e-05 185 KSP unpreconditioned resid norm 1.377904214978e+02 true resid norm 1.441752187090e+02 ||r(i)||/||b|| 4.746083549740e-05 186 KSP unpreconditioned resid norm 1.376670282479e+02 true resid norm 1.441674745344e+02 ||r(i)||/||b|| 4.745828620353e-05 187 KSP unpreconditioned resid norm 1.376636051755e+02 true resid norm 1.463118783906e+02 ||r(i)||/||b|| 4.816419946362e-05 188 KSP unpreconditioned resid norm 1.363148994276e+02 true resid norm 1.432997756128e+02 ||r(i)||/||b|| 4.717264962781e-05 189 KSP unpreconditioned resid norm 1.363051099558e+02 true resid norm 1.451009062639e+02 ||r(i)||/||b|| 4.776556126897e-05 190 KSP unpreconditioned resid norm 1.362538398564e+02 true resid norm 1.438957985476e+02 ||r(i)||/||b|| 4.736885357127e-05 191 KSP unpreconditioned resid norm 1.358335705250e+02 true resid norm 1.436616069458e+02 ||r(i)||/||b|| 4.729176037047e-05 192 KSP unpreconditioned resid norm 1.337424103882e+02 true resid norm 1.432816138672e+02 ||r(i)||/||b|| 4.716667098856e-05 193 KSP unpreconditioned resid norm 1.337419543121e+02 true resid norm 1.405274691954e+02 ||r(i)||/||b|| 4.626003801533e-05 194 KSP unpreconditioned resid norm 1.322568117657e+02 true resid norm 1.417123189671e+02 ||r(i)||/||b|| 4.665007702902e-05 195 KSP unpreconditioned resid norm 1.320880115122e+02 true resid norm 1.413658215058e+02 ||r(i)||/||b|| 4.653601402181e-05 196 KSP unpreconditioned resid norm 1.312526182172e+02 true resid norm 1.420574070412e+02 ||r(i)||/||b|| 4.676367608204e-05 197 KSP unpreconditioned resid norm 1.311651332692e+02 true resid norm 1.398984125128e+02 ||r(i)||/||b|| 4.605295973934e-05 198 KSP unpreconditioned resid norm 1.294482397720e+02 true resid norm 1.380390703259e+02 ||r(i)||/||b|| 4.544088552537e-05 199 KSP unpreconditioned resid norm 1.293598434732e+02 true resid norm 1.373830689903e+02 ||r(i)||/||b|| 4.522493737731e-05 200 KSP unpreconditioned resid norm 1.265165992897e+02 true resid norm 1.375015523244e+02 ||r(i)||/||b|| 4.526394073779e-05 201 KSP unpreconditioned resid norm 1.263813235463e+02 true resid norm 1.356820166419e+02 ||r(i)||/||b|| 4.466497037047e-05 202 KSP unpreconditioned resid norm 1.243190164198e+02 true resid norm 1.366420975402e+02 ||r(i)||/||b|| 4.498101803792e-05 203 KSP unpreconditioned resid norm 1.230747513665e+02 true resid norm 1.348856851681e+02 ||r(i)||/||b|| 4.440282714351e-05 204 KSP unpreconditioned resid norm 1.198014010398e+02 true resid norm 1.325188356617e+02 ||r(i)||/||b|| 4.362368731578e-05 205 KSP unpreconditioned resid norm 1.195977240348e+02 true resid norm 1.299721846860e+02 ||r(i)||/||b|| 4.278535889769e-05 206 KSP unpreconditioned resid norm 1.130620928393e+02 true resid norm 1.266961052950e+02 ||r(i)||/||b|| 4.170691097546e-05 207 KSP unpreconditioned resid norm 1.123992882530e+02 true resid norm 1.270907813369e+02 ||r(i)||/||b|| 4.183683382120e-05 208 KSP unpreconditioned resid norm 1.063236317163e+02 true resid norm 1.182163029843e+02 ||r(i)||/||b|| 3.891545689533e-05 209 KSP unpreconditioned resid norm 1.059802897214e+02 true resid norm 1.187516613498e+02 ||r(i)||/||b|| 3.909169075539e-05 210 KSP unpreconditioned resid norm 9.878733567790e+01 true resid norm 1.124812677115e+02 ||r(i)||/||b|| 3.702754877846e-05 211 KSP unpreconditioned resid norm 9.861048081032e+01 true resid norm 1.117192174341e+02 ||r(i)||/||b|| 3.677669052986e-05 212 KSP unpreconditioned resid norm 9.169383217455e+01 true resid norm 1.102172324977e+02 ||r(i)||/||b|| 3.628225424167e-05 213 KSP unpreconditioned resid norm 9.146164223196e+01 true resid norm 1.121134424773e+02 ||r(i)||/||b|| 3.690646491198e-05 214 KSP unpreconditioned resid norm 8.692213412954e+01 true resid norm 1.056264039532e+02 ||r(i)||/||b|| 3.477100591276e-05 215 KSP unpreconditioned resid norm 8.685846611574e+01 true resid norm 1.029018845366e+02 ||r(i)||/||b|| 3.387412523521e-05 216 KSP unpreconditioned resid norm 7.808516472373e+01 true resid norm 9.749023000535e+01 ||r(i)||/||b|| 3.209267036539e-05 217 KSP unpreconditioned resid norm 7.786400257086e+01 true resid norm 1.004515546585e+02 ||r(i)||/||b|| 3.306750462244e-05 218 KSP unpreconditioned resid norm 6.646475864029e+01 true resid norm 9.429020541969e+01 ||r(i)||/||b|| 3.103925881653e-05 219 KSP unpreconditioned resid norm 6.643821996375e+01 true resid norm 8.864525788550e+01 ||r(i)||/||b|| 2.918100655438e-05 220 KSP unpreconditioned resid norm 5.625046780791e+01 true resid norm 8.410041684883e+01 ||r(i)||/||b|| 2.768489678784e-05 221 KSP unpreconditioned resid norm 5.623343238032e+01 true resid norm 8.815552919640e+01 ||r(i)||/||b|| 2.901979346270e-05 222 KSP unpreconditioned resid norm 4.491016868776e+01 true resid norm 8.557052117768e+01 ||r(i)||/||b|| 2.816883834410e-05 223 KSP unpreconditioned resid norm 4.461976108543e+01 true resid norm 7.867894425332e+01 ||r(i)||/||b|| 2.590020992340e-05 224 KSP unpreconditioned resid norm 3.535718264955e+01 true resid norm 7.609346753983e+01 ||r(i)||/||b|| 2.504910051583e-05 225 KSP unpreconditioned resid norm 3.525592897743e+01 true resid norm 7.926812413349e+01 ||r(i)||/||b|| 2.609416121143e-05 226 KSP unpreconditioned resid norm 2.633469451114e+01 true resid norm 7.883483297310e+01 ||r(i)||/||b|| 2.595152670968e-05 227 KSP unpreconditioned resid norm 2.614440577316e+01 true resid norm 7.398963634249e+01 ||r(i)||/||b|| 2.435654331172e-05 228 KSP unpreconditioned resid norm 1.988460252721e+01 true resid norm 7.147825835126e+01 ||r(i)||/||b|| 2.352982635730e-05 229 KSP unpreconditioned resid norm 1.975927240058e+01 true resid norm 7.488507147714e+01 ||r(i)||/||b|| 2.465131033205e-05 230 KSP unpreconditioned resid norm 1.505732242656e+01 true resid norm 7.888901529160e+01 ||r(i)||/||b|| 2.596936291016e-05 231 KSP unpreconditioned resid norm 1.504120870628e+01 true resid norm 7.126366562975e+01 ||r(i)||/||b|| 2.345918488406e-05 232 KSP unpreconditioned resid norm 1.163470506257e+01 true resid norm 7.142763663542e+01 ||r(i)||/||b|| 2.351316226655e-05 233 KSP unpreconditioned resid norm 1.157114340949e+01 true resid norm 7.464790352976e+01 ||r(i)||/||b|| 2.457323735226e-05 234 KSP unpreconditioned resid norm 8.702850618357e+00 true resid norm 7.798031063059e+01 ||r(i)||/||b|| 2.567022771329e-05 235 KSP unpreconditioned resid norm 8.702017371082e+00 true resid norm 7.032943782131e+01 ||r(i)||/||b|| 2.315164775854e-05 236 KSP unpreconditioned resid norm 6.422855779486e+00 true resid norm 6.800345168870e+01 ||r(i)||/||b|| 2.238595968678e-05 237 KSP unpreconditioned resid norm 6.413921210094e+00 true resid norm 7.408432731879e+01 ||r(i)||/||b|| 2.438771449973e-05 238 KSP unpreconditioned resid norm 4.949111361190e+00 true resid norm 7.744087979524e+01 ||r(i)||/||b|| 2.549265324267e-05 239 KSP unpreconditioned resid norm 4.947369357666e+00 true resid norm 7.104259266677e+01 ||r(i)||/||b|| 2.338641018933e-05 240 KSP unpreconditioned resid norm 3.873645232239e+00 true resid norm 6.908028336929e+01 ||r(i)||/||b|| 2.274044037845e-05 241 KSP unpreconditioned resid norm 3.841473653930e+00 true resid norm 7.431718972562e+01 ||r(i)||/||b|| 2.446437014474e-05 242 KSP unpreconditioned resid norm 3.057267436362e+00 true resid norm 7.685939322732e+01 ||r(i)||/||b|| 2.530123450517e-05 243 KSP unpreconditioned resid norm 2.980906717815e+00 true resid norm 6.975661521135e+01 ||r(i)||/||b|| 2.296308109705e-05 244 KSP unpreconditioned resid norm 2.415633545154e+00 true resid norm 6.989644258184e+01 ||r(i)||/||b|| 2.300911067057e-05 245 KSP unpreconditioned resid norm 2.363923146996e+00 true resid norm 7.486631867276e+01 ||r(i)||/||b|| 2.464513712301e-05 246 KSP unpreconditioned resid norm 1.947823635306e+00 true resid norm 7.671103669547e+01 ||r(i)||/||b|| 2.525239722914e-05 247 KSP unpreconditioned resid norm 1.942156637334e+00 true resid norm 6.835715877902e+01 ||r(i)||/||b|| 2.250239602152e-05 248 KSP unpreconditioned resid norm 1.675749569790e+00 true resid norm 7.111781390782e+01 ||r(i)||/||b|| 2.341117216285e-05 249 KSP unpreconditioned resid norm 1.673819729570e+00 true resid norm 7.552508026111e+01 ||r(i)||/||b|| 2.486199391474e-05 250 KSP unpreconditioned resid norm 1.453311843294e+00 true resid norm 7.639099426865e+01 ||r(i)||/||b|| 2.514704291716e-05 251 KSP unpreconditioned resid norm 1.452846325098e+00 true resid norm 6.951401359923e+01 ||r(i)||/||b|| 2.288321941689e-05 252 KSP unpreconditioned resid norm 1.335008887441e+00 true resid norm 6.912230871414e+01 ||r(i)||/||b|| 2.275427464204e-05 253 KSP unpreconditioned resid norm 1.334477013356e+00 true resid norm 7.412281497148e+01 ||r(i)||/||b|| 2.440038419546e-05 254 KSP unpreconditioned resid norm 1.248507835050e+00 true resid norm 7.801932499175e+01 ||r(i)||/||b|| 2.568307079543e-05 255 KSP unpreconditioned resid norm 1.248246596771e+00 true resid norm 7.094899926215e+01 ||r(i)||/||b|| 2.335560030938e-05 256 KSP unpreconditioned resid norm 1.208952722414e+00 true resid norm 7.101235824005e+01 ||r(i)||/||b|| 2.337645736134e-05 257 KSP unpreconditioned resid norm 1.208780664971e+00 true resid norm 7.562936418444e+01 ||r(i)||/||b|| 2.489632299136e-05 258 KSP unpreconditioned resid norm 1.179956701653e+00 true resid norm 7.812300941072e+01 ||r(i)||/||b|| 2.571720252207e-05 259 KSP unpreconditioned resid norm 1.179219541297e+00 true resid norm 7.131201918549e+01 ||r(i)||/||b|| 2.347510232240e-05 260 KSP unpreconditioned resid norm 1.160215487467e+00 true resid norm 7.222079766175e+01 ||r(i)||/||b|| 2.377426181841e-05 261 KSP unpreconditioned resid norm 1.159115040554e+00 true resid norm 7.481372509179e+01 ||r(i)||/||b|| 2.462782391678e-05 262 KSP unpreconditioned resid norm 1.151973184765e+00 true resid norm 7.709040836137e+01 ||r(i)||/||b|| 2.537728204907e-05 263 KSP unpreconditioned resid norm 1.150882463576e+00 true resid norm 7.032588895526e+01 ||r(i)||/||b|| 2.315047951236e-05 264 KSP unpreconditioned resid norm 1.137617003277e+00 true resid norm 7.004055871264e+01 ||r(i)||/||b|| 2.305655205500e-05 265 KSP unpreconditioned resid norm 1.137134003401e+00 true resid norm 7.610459827221e+01 ||r(i)||/||b|| 2.505276462582e-05 266 KSP unpreconditioned resid norm 1.131425778253e+00 true resid norm 7.852741072990e+01 ||r(i)||/||b|| 2.585032681802e-05 267 KSP unpreconditioned resid norm 1.131176695314e+00 true resid norm 7.064571495865e+01 ||r(i)||/||b|| 2.325576258022e-05 268 KSP unpreconditioned resid norm 1.125420065063e+00 true resid norm 7.138837220124e+01 ||r(i)||/||b|| 2.350023686323e-05 269 KSP unpreconditioned resid norm 1.124779989266e+00 true resid norm 7.585594020759e+01 ||r(i)||/||b|| 2.497090923065e-05 270 KSP unpreconditioned resid norm 1.119805446125e+00 true resid norm 7.703631305135e+01 ||r(i)||/||b|| 2.535947449079e-05 271 KSP unpreconditioned resid norm 1.119024433863e+00 true resid norm 7.081439585094e+01 ||r(i)||/||b|| 2.331129040360e-05 272 KSP unpreconditioned resid norm 1.115694452861e+00 true resid norm 7.134872343512e+01 ||r(i)||/||b|| 2.348718494222e-05 273 KSP unpreconditioned resid norm 1.113572716158e+00 true resid norm 7.600475566242e+01 ||r(i)||/||b|| 2.501989757889e-05 274 KSP unpreconditioned resid norm 1.108711406381e+00 true resid norm 7.738835220359e+01 ||r(i)||/||b|| 2.547536175937e-05 275 KSP unpreconditioned resid norm 1.107890435549e+00 true resid norm 7.093429729336e+01 ||r(i)||/||b|| 2.335076058915e-05 276 KSP unpreconditioned resid norm 1.103340227961e+00 true resid norm 7.145267197866e+01 ||r(i)||/||b|| 2.352140361564e-05 277 KSP unpreconditioned resid norm 1.102897652964e+00 true resid norm 7.448617654625e+01 ||r(i)||/||b|| 2.451999867624e-05 278 KSP unpreconditioned resid norm 1.102576754158e+00 true resid norm 7.707165090465e+01 ||r(i)||/||b|| 2.537110730854e-05 279 KSP unpreconditioned resid norm 1.102564028537e+00 true resid norm 7.009637628868e+01 ||r(i)||/||b|| 2.307492656359e-05 280 KSP unpreconditioned resid norm 1.100828424712e+00 true resid norm 7.059832880916e+01 ||r(i)||/||b|| 2.324016360096e-05 281 KSP unpreconditioned resid norm 1.100686341559e+00 true resid norm 7.460867988528e+01 ||r(i)||/||b|| 2.456032537644e-05 282 KSP unpreconditioned resid norm 1.099417185996e+00 true resid norm 7.763784632467e+01 ||r(i)||/||b|| 2.555749237477e-05 283 KSP unpreconditioned resid norm 1.099379061087e+00 true resid norm 7.017139420999e+01 ||r(i)||/||b|| 2.309962160657e-05 284 KSP unpreconditioned resid norm 1.097928047676e+00 true resid norm 6.983706716123e+01 ||r(i)||/||b|| 2.298956496018e-05 285 KSP unpreconditioned resid norm 1.096490152934e+00 true resid norm 7.414445779601e+01 ||r(i)||/||b|| 2.440750876614e-05 286 KSP unpreconditioned resid norm 1.094691490227e+00 true resid norm 7.634526287231e+01 ||r(i)||/||b|| 2.513198866374e-05 287 KSP unpreconditioned resid norm 1.093560358328e+00 true resid norm 7.003716824146e+01 ||r(i)||/||b|| 2.305543595061e-05 288 KSP unpreconditioned resid norm 1.093357856424e+00 true resid norm 6.964715939684e+01 ||r(i)||/||b|| 2.292704949292e-05 289 KSP unpreconditioned resid norm 1.091881434739e+00 true resid norm 7.429955169250e+01 ||r(i)||/||b|| 2.445856390566e-05 290 KSP unpreconditioned resid norm 1.091817808496e+00 true resid norm 7.607892786798e+01 ||r(i)||/||b|| 2.504431422190e-05 291 KSP unpreconditioned resid norm 1.090295101202e+00 true resid norm 6.942248339413e+01 ||r(i)||/||b|| 2.285308871866e-05 292 KSP unpreconditioned resid norm 1.089995012773e+00 true resid norm 6.995557798353e+01 ||r(i)||/||b|| 2.302857736947e-05 293 KSP unpreconditioned resid norm 1.089975910578e+00 true resid norm 7.453210925277e+01 ||r(i)||/||b|| 2.453511919866e-05 294 KSP unpreconditioned resid norm 1.085570944646e+00 true resid norm 7.629598425927e+01 ||r(i)||/||b|| 2.511576670710e-05 295 KSP unpreconditioned resid norm 1.085363565621e+00 true resid norm 7.025539955712e+01 ||r(i)||/||b|| 2.312727520749e-05 296 KSP unpreconditioned resid norm 1.083348574106e+00 true resid norm 7.003219621882e+01 ||r(i)||/||b|| 2.305379921754e-05 297 KSP unpreconditioned resid norm 1.082180374430e+00 true resid norm 7.473048827106e+01 ||r(i)||/||b|| 2.460042330597e-05 298 KSP unpreconditioned resid norm 1.081326671068e+00 true resid norm 7.660142838935e+01 ||r(i)||/||b|| 2.521631542651e-05 299 KSP unpreconditioned resid norm 1.078679751898e+00 true resid norm 7.077868424247e+01 ||r(i)||/||b|| 2.329953454992e-05 300 KSP unpreconditioned resid norm 1.078656949888e+00 true resid norm 7.074960394994e+01 ||r(i)||/||b|| 2.328996164972e-05 Linear solve did not converge due to DIVERGED_ITS iterations 300 KSP Object: 2 MPI processes type: fgmres GMRES: restart=300, using Modified Gram-Schmidt Orthogonalization GMRES: happy breakdown tolerance 1e-30 maximum iterations=300, initial guess is zero tolerances: relative=1e-09, absolute=1e-20, divergence=10000 right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 2 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization DIAG Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses (lumped, if requested) A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_u_) 2 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_u_) 2 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Mat Object: 2 MPI processes type: mpiaij rows=184326, cols=184326 package used to perform factorization: mumps total: nonzeros=4.03041e+08, allocated nonzeros=4.03041e+08 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0 CNTL(4) (value of static pivoting): -1 CNTL(5) (fixation for null pivots): 0 RINFO(1) (local estimated flops for the elimination after analysis): [0] 5.59214e+11 [1] 5.35237e+11 RINFO(2) (local estimated flops for the assembly after factorization): [0] 4.2839e+08 [1] 3.799e+08 RINFO(3) (local estimated flops for the elimination after factorization): [0] 5.59214e+11 [1] 5.35237e+11 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 2621 [1] 2649 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 2621 [1] 2649 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 90423 [1] 93903 RINFOG(1) (global estimated flops for the elimination after analysis): 1.09445e+12 RINFOG(2) (global estimated flops for the assembly after factorization): 8.0829e+08 RINFOG(3) (global estimated flops for the elimination after factorization): 1.09445e+12 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 403041366 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 2265748 INFOG(5) (estimated maximum front size in the complete tree): 6663 INFOG(6) (number of nodes in the complete tree): 2812 INFOG(7) (ordering option effectively use after analysis): 5 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 403041366 INFOG(10) (total integer space store the matrix factors after factorization): 2265766 INFOG(11) (order of largest frontal matrix after factorization): 6663 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 2649 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 5270 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 2649 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 5270 INFOG(20) (estimated number of entries in the factors): 403041366 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 2121 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 4174 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 403041366 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 2467, 4922 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix = precond matrix: Mat Object: (fieldsplit_u_) 2 MPI processes type: mpiaij rows=184326, cols=184326, bs=3 total: nonzeros=3.32649e+07, allocated nonzeros=3.32649e+07 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 26829 nodes, limit used is 5 KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_lu_) 2 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_lu_) 2 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Mat Object: 2 MPI processes type: mpiaij rows=2583, cols=2583 package used to perform factorization: mumps total: nonzeros=2.17621e+06, allocated nonzeros=2.17621e+06 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0 CNTL(4) (value of static pivoting): -1 CNTL(5) (fixation for null pivots): 0 RINFO(1) (local estimated flops for the elimination after analysis): [0] 5.12794e+08 [1] 5.02142e+08 RINFO(2) (local estimated flops for the assembly after factorization): [0] 815031 [1] 745263 RINFO(3) (local estimated flops for the elimination after factorization): [0] 5.12794e+08 [1] 5.02142e+08 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 34 [1] 34 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 34 [1] 34 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 1158 [1] 1425 RINFOG(1) (global estimated flops for the elimination after analysis): 1.01494e+09 RINFOG(2) (global estimated flops for the assembly after factorization): 1.56029e+06 RINFOG(3) (global estimated flops for the elimination after factorization): 1.01494e+09 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 2176209 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 14427 INFOG(5) (estimated maximum front size in the complete tree): 699 INFOG(6) (number of nodes in the complete tree): 15 INFOG(7) (ordering option effectively use after analysis): 2 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 2176209 INFOG(10) (total integer space store the matrix factors after factorization): 14427 INFOG(11) (order of largest frontal matrix after factorization): 699 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 34 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 68 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 34 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 68 INFOG(20) (estimated number of entries in the factors): 2176209 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 30 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 59 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 2176209 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 16, 32 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_lu_) 2 MPI processes type: schurcomplement rows=2583, cols=2583 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_lu_) 2 MPI processes type: mpiaij rows=2583, cols=2583, bs=3 total: nonzeros=117369, allocated nonzeros=117369 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines A10 Mat Object: 2 MPI processes type: mpiaij rows=2583, cols=184326, rbs=3, cbs = 1 total: nonzeros=292770, allocated nonzeros=292770 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines KSP of A00 KSP Object: (fieldsplit_u_) 2 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_u_) 2 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Mat Object: 2 MPI processes type: mpiaij rows=184326, cols=184326 package used to perform factorization: mumps total: nonzeros=4.03041e+08, allocated nonzeros=4.03041e+08 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0 CNTL(4) (value of static pivoting): -1 CNTL(5) (fixation for null pivots): 0 RINFO(1) (local estimated flops for the elimination after analysis): [0] 5.59214e+11 [1] 5.35237e+11 RINFO(2) (local estimated flops for the assembly after factorization): [0] 4.2839e+08 [1] 3.799e+08 RINFO(3) (local estimated flops for the elimination after factorization): [0] 5.59214e+11 [1] 5.35237e+11 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 2621 [1] 2649 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 2621 [1] 2649 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 90423 [1] 93903 RINFOG(1) (global estimated flops for the elimination after analysis): 1.09445e+12 RINFOG(2) (global estimated flops for the assembly after factorization): 8.0829e+08 RINFOG(3) (global estimated flops for the elimination after factorization): 1.09445e+12 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 403041366 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 2265748 INFOG(5) (estimated maximum front size in the complete tree): 6663 INFOG(6) (number of nodes in the complete tree): 2812 INFOG(7) (ordering option effectively use after analysis): 5 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 403041366 INFOG(10) (total integer space store the matrix factors after factorization): 2265766 INFOG(11) (order of largest frontal matrix after factorization): 6663 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 2649 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 5270 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 2649 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 5270 INFOG(20) (estimated number of entries in the factors): 403041366 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 2121 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 4174 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 403041366 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 2467, 4922 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix = precond matrix: Mat Object: (fieldsplit_u_) 2 MPI processes type: mpiaij rows=184326, cols=184326, bs=3 total: nonzeros=3.32649e+07, allocated nonzeros=3.32649e+07 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 26829 nodes, limit used is 5 A01 Mat Object: 2 MPI processes type: mpiaij rows=184326, cols=2583, rbs=3, cbs = 1 total: nonzeros=292770, allocated nonzeros=292770 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 16098 nodes, limit used is 5 Mat Object: 2 MPI processes type: mpiaij rows=2583, cols=2583, rbs=3, cbs = 1 total: nonzeros=1.25158e+06, allocated nonzeros=1.25158e+06 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpiaij rows=186909, cols=186909 total: nonzeros=3.39678e+07, allocated nonzeros=3.39678e+07 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 26829 nodes, limit used is 5 KSPSolve completed Giang On Sun, Apr 17, 2016 at 1:15 AM, Matthew Knepley wrote: > On Sat, Apr 16, 2016 at 6:54 PM, Hoang Giang Bui > wrote: > >> Hello >> >> I'm solving an indefinite problem arising from mesh tying/contact using >> Lagrange multiplier, the matrix has the form >> >> K = [A P^T >> P 0] >> >> I used the FIELDSPLIT preconditioner with one field is the main variable >> (displacement) and the other field for dual variable (Lagrange multiplier). >> The block size for each field is 3. According to the manual, I first chose >> the preconditioner based on Schur complement to treat this problem. >> > > > For any solver question, please send us the output of > > -ksp_view -ksp_monitor_true_residual -ksp_converged_reason > > > However, I will comment below > > >> The parameters used for the solve is >> -ksp_type gmres >> > > You need 'fgmres' here with the options you have below. > > >> -ksp_max_it 300 >> -ksp_gmres_restart 300 >> -ksp_gmres_modifiedgramschmidt >> -pc_fieldsplit_type schur >> -pc_fieldsplit_schur_fact_type diag >> -pc_fieldsplit_schur_precondition selfp >> > > > It could be taking time in the MatMatMult() here if that matrix is dense. > Is there any reason to > believe that is a good preconditioner for your problem? > > > >> -pc_fieldsplit_detect_saddle_point >> -fieldsplit_u_pc_type hypre >> > > I would just use MUMPS here to start, especially if it works on the whole > problem. Same with the one below. > > Matt > > >> -fieldsplit_u_pc_hypre_type boomeramg >> -fieldsplit_u_pc_hypre_boomeramg_coarsen_type PMIS >> -fieldsplit_lu_pc_type hypre >> -fieldsplit_lu_pc_hypre_type boomeramg >> -fieldsplit_lu_pc_hypre_boomeramg_coarsen_type PMIS >> >> For the test case, a small problem is solved on 2 processes. Due to the >> decomposition, the contact only happens in 1 proc, so the size of Lagrange >> multiplier dofs on proc 0 is 0. >> >> 0: mIndexU.size(): 80490 >> 0: mIndexLU.size(): 0 >> 1: mIndexU.size(): 103836 >> 1: mIndexLU.size(): 2583 >> >> However, with this setup the solver takes very long at KSPSolve before >> going to iteration, and the first iteration seems forever so I have to stop >> the calculation. I guessed that the solver takes time to compute the Schur >> complement, but according to the manual only the diagonal of A is used to >> approximate the Schur complement, so it should not take long to compute >> this. >> >> Note that I ran the same problem with direct solver (MUMPS) and it's able >> to produce the valid results. The parameter for the solve is pretty standard >> -ksp_type preonly >> -pc_type lu >> -pc_factor_mat_solver_package mumps >> >> Hence the matrix/rhs must not have any problem here. Do you have any idea >> or suggestion for this case? >> >> >> Giang >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 17 08:16:07 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 17 Apr 2016 09:16:07 -0400 Subject: [petsc-users] fieldsplit preconditioner for indefinite matrix In-Reply-To: References: Message-ID: On Sun, Apr 17, 2016 at 4:25 AM, Hoang Giang Bui wrote: > > >> It could be taking time in the MatMatMult() here if that matrix is dense. >> Is there any reason to >> believe that is a good preconditioner for your problem? >> > > This is the first approach to the problem, so I chose the most simple > setting. Do you have any other recommendation? > This is in no way the simplest PC. We need to make it simpler first. 1) Run on only 1 proc 2) Use -pc_fieldsplit_schur_fact_type full 3) Use -fieldsplit_lu_ksp_type gmres -fieldsplit_lu_ksp_monitor_true_residual This should converge in 1 outer iteration, but we will see how good your Schur complement preconditioner is for this problem. You need to start out from something you understand and then start making approximations. Matt > For any solver question, please send us the output of >> >> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason >> >> > I sent here the full output (after changed to fgmres), again it takes long > at the first iteration but after that, it does not converge > > -ksp_type fgmres > -ksp_max_it 300 > -ksp_gmres_restart 300 > -ksp_gmres_modifiedgramschmidt > -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type diag > -pc_fieldsplit_schur_precondition selfp > -pc_fieldsplit_detect_saddle_point > -fieldsplit_u_ksp_type preonly > -fieldsplit_u_pc_type lu > -fieldsplit_u_pc_factor_mat_solver_package mumps > -fieldsplit_lu_ksp_type preonly > -fieldsplit_lu_pc_type lu > -fieldsplit_lu_pc_factor_mat_solver_package mumps > > 0 KSP unpreconditioned resid norm 3.037772453815e+06 true resid norm > 3.037772453815e+06 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 3.024368791893e+06 true resid norm > 3.024368791296e+06 ||r(i)||/||b|| 9.955876673705e-01 > 2 KSP unpreconditioned resid norm 3.008534454663e+06 true resid norm > 3.008534454904e+06 ||r(i)||/||b|| 9.903751846607e-01 > 3 KSP unpreconditioned resid norm 4.633282412600e+02 true resid norm > 4.607539866185e+02 ||r(i)||/||b|| 1.516749505184e-04 > 4 KSP unpreconditioned resid norm 4.630592911836e+02 true resid norm > 4.605625897903e+02 ||r(i)||/||b|| 1.516119448683e-04 > 5 KSP unpreconditioned resid norm 2.145735509629e+02 true resid norm > 2.111697416683e+02 ||r(i)||/||b|| 6.951466736857e-05 > 6 KSP unpreconditioned resid norm 2.145734219762e+02 true resid norm > 2.112001242378e+02 ||r(i)||/||b|| 6.952466896346e-05 > 7 KSP unpreconditioned resid norm 1.892914067411e+02 true resid norm > 1.831020928502e+02 ||r(i)||/||b|| 6.027511791420e-05 > 8 KSP unpreconditioned resid norm 1.892906351597e+02 true resid norm > 1.831422357767e+02 ||r(i)||/||b|| 6.028833250718e-05 > 9 KSP unpreconditioned resid norm 1.891426729822e+02 true resid norm > 1.835600473014e+02 ||r(i)||/||b|| 6.042587128964e-05 > 10 KSP unpreconditioned resid norm 1.891425181679e+02 true resid norm > 1.855772578041e+02 ||r(i)||/||b|| 6.108991395027e-05 > 11 KSP unpreconditioned resid norm 1.891417382057e+02 true resid norm > 1.833302669042e+02 ||r(i)||/||b|| 6.035023020699e-05 > 12 KSP unpreconditioned resid norm 1.891414749001e+02 true resid norm > 1.827923591605e+02 ||r(i)||/||b|| 6.017315712076e-05 > 13 KSP unpreconditioned resid norm 1.891414702834e+02 true resid norm > 1.849895606391e+02 ||r(i)||/||b|| 6.089645075515e-05 > 14 KSP unpreconditioned resid norm 1.891414687385e+02 true resid norm > 1.852700958573e+02 ||r(i)||/||b|| 6.098879974523e-05 > 15 KSP unpreconditioned resid norm 1.891399614701e+02 true resid norm > 1.817034334576e+02 ||r(i)||/||b|| 5.981469521503e-05 > 16 KSP unpreconditioned resid norm 1.891393964580e+02 true resid norm > 1.823173574739e+02 ||r(i)||/||b|| 6.001679199012e-05 > 17 KSP unpreconditioned resid norm 1.890868604964e+02 true resid norm > 1.834754811775e+02 ||r(i)||/||b|| 6.039803308740e-05 > 18 KSP unpreconditioned resid norm 1.888442703508e+02 true resid norm > 1.852079421560e+02 ||r(i)||/||b|| 6.096833945658e-05 > 19 KSP unpreconditioned resid norm 1.888131521870e+02 true resid norm > 1.810111295757e+02 ||r(i)||/||b|| 5.958679668335e-05 > 20 KSP unpreconditioned resid norm 1.888038471618e+02 true resid norm > 1.814080717355e+02 ||r(i)||/||b|| 5.971746550920e-05 > 21 KSP unpreconditioned resid norm 1.885794485272e+02 true resid norm > 1.843223565278e+02 ||r(i)||/||b|| 6.067681478129e-05 > 22 KSP unpreconditioned resid norm 1.884898771362e+02 true resid norm > 1.842766260526e+02 ||r(i)||/||b|| 6.066176083110e-05 > 23 KSP unpreconditioned resid norm 1.884840498049e+02 true resid norm > 1.813011285152e+02 ||r(i)||/||b|| 5.968226102238e-05 > 24 KSP unpreconditioned resid norm 1.884105698955e+02 true resid norm > 1.811513025118e+02 ||r(i)||/||b|| 5.963294001309e-05 > 25 KSP unpreconditioned resid norm 1.881392557375e+02 true resid norm > 1.835706567649e+02 ||r(i)||/||b|| 6.042936380386e-05 > 26 KSP unpreconditioned resid norm 1.881234481250e+02 true resid norm > 1.843633799886e+02 ||r(i)||/||b|| 6.069031923609e-05 > 27 KSP unpreconditioned resid norm 1.852572648925e+02 true resid norm > 1.791532195358e+02 ||r(i)||/||b|| 5.897519391579e-05 > 28 KSP unpreconditioned resid norm 1.852177694782e+02 true resid norm > 1.800935543889e+02 ||r(i)||/||b|| 5.928474141066e-05 > 29 KSP unpreconditioned resid norm 1.844720976468e+02 true resid norm > 1.806835899755e+02 ||r(i)||/||b|| 5.947897438749e-05 > 30 KSP unpreconditioned resid norm 1.843525447108e+02 true resid norm > 1.811351238391e+02 ||r(i)||/||b|| 5.962761417881e-05 > 31 KSP unpreconditioned resid norm 1.834262885149e+02 true resid norm > 1.778584233423e+02 ||r(i)||/||b|| 5.854896179565e-05 > 32 KSP unpreconditioned resid norm 1.833523213017e+02 true resid norm > 1.773290649733e+02 ||r(i)||/||b|| 5.837470306591e-05 > 33 KSP unpreconditioned resid norm 1.821645929344e+02 true resid norm > 1.781151248933e+02 ||r(i)||/||b|| 5.863346501467e-05 > 34 KSP unpreconditioned resid norm 1.820831279534e+02 true resid norm > 1.789778939067e+02 ||r(i)||/||b|| 5.891747872094e-05 > 35 KSP unpreconditioned resid norm 1.814860919375e+02 true resid norm > 1.757339506869e+02 ||r(i)||/||b|| 5.784960965928e-05 > 36 KSP unpreconditioned resid norm 1.812512010159e+02 true resid norm > 1.764086437459e+02 ||r(i)||/||b|| 5.807171090922e-05 > 37 KSP unpreconditioned resid norm 1.804298150360e+02 true resid norm > 1.780147196442e+02 ||r(i)||/||b|| 5.860041275333e-05 > 38 KSP unpreconditioned resid norm 1.799675012847e+02 true resid norm > 1.780554543786e+02 ||r(i)||/||b|| 5.861382216269e-05 > 39 KSP unpreconditioned resid norm 1.793156052097e+02 true resid norm > 1.747985717965e+02 ||r(i)||/||b|| 5.754169361071e-05 > 40 KSP unpreconditioned resid norm 1.789109248325e+02 true resid norm > 1.734086984879e+02 ||r(i)||/||b|| 5.708416319009e-05 > 41 KSP unpreconditioned resid norm 1.788931581371e+02 true resid norm > 1.766103879126e+02 ||r(i)||/||b|| 5.813812278494e-05 > 42 KSP unpreconditioned resid norm 1.785522436483e+02 true resid norm > 1.762597032909e+02 ||r(i)||/||b|| 5.802268141233e-05 > 43 KSP unpreconditioned resid norm 1.783317950582e+02 true resid norm > 1.752774080448e+02 ||r(i)||/||b|| 5.769932103530e-05 > 44 KSP unpreconditioned resid norm 1.782832982797e+02 true resid norm > 1.741667594885e+02 ||r(i)||/||b|| 5.733370821430e-05 > 45 KSP unpreconditioned resid norm 1.781302427969e+02 true resid norm > 1.760315735899e+02 ||r(i)||/||b|| 5.794758372005e-05 > 46 KSP unpreconditioned resid norm 1.780557458973e+02 true resid norm > 1.757279911034e+02 ||r(i)||/||b|| 5.784764783244e-05 > 47 KSP unpreconditioned resid norm 1.774691940686e+02 true resid norm > 1.729436852773e+02 ||r(i)||/||b|| 5.693108615167e-05 > 48 KSP unpreconditioned resid norm 1.771436357084e+02 true resid norm > 1.734001323688e+02 ||r(i)||/||b|| 5.708134332148e-05 > 49 KSP unpreconditioned resid norm 1.756105727417e+02 true resid norm > 1.740222172981e+02 ||r(i)||/||b|| 5.728612657594e-05 > 50 KSP unpreconditioned resid norm 1.756011794480e+02 true resid norm > 1.736979026533e+02 ||r(i)||/||b|| 5.717936589858e-05 > 51 KSP unpreconditioned resid norm 1.751096154950e+02 true resid norm > 1.713154407940e+02 ||r(i)||/||b|| 5.639508666256e-05 > 52 KSP unpreconditioned resid norm 1.712639990486e+02 true resid norm > 1.684444278579e+02 ||r(i)||/||b|| 5.544998199137e-05 > 53 KSP unpreconditioned resid norm 1.710183053728e+02 true resid norm > 1.692712952670e+02 ||r(i)||/||b|| 5.572217729951e-05 > 54 KSP unpreconditioned resid norm 1.655470115849e+02 true resid norm > 1.631767858448e+02 ||r(i)||/||b|| 5.371593439788e-05 > 55 KSP unpreconditioned resid norm 1.648313805392e+02 true resid norm > 1.617509396670e+02 ||r(i)||/||b|| 5.324656211951e-05 > 56 KSP unpreconditioned resid norm 1.643417766012e+02 true resid norm > 1.614766932468e+02 ||r(i)||/||b|| 5.315628332992e-05 > 57 KSP unpreconditioned resid norm 1.643165564782e+02 true resid norm > 1.611660297521e+02 ||r(i)||/||b|| 5.305401645527e-05 > 58 KSP unpreconditioned resid norm 1.639561245303e+02 true resid norm > 1.616105878219e+02 ||r(i)||/||b|| 5.320035989496e-05 > 59 KSP unpreconditioned resid norm 1.636859175366e+02 true resid norm > 1.601704798933e+02 ||r(i)||/||b|| 5.272629281109e-05 > 60 KSP unpreconditioned resid norm 1.633269681891e+02 true resid norm > 1.603249334191e+02 ||r(i)||/||b|| 5.277713714789e-05 > 61 KSP unpreconditioned resid norm 1.633257086864e+02 true resid norm > 1.602922744638e+02 ||r(i)||/||b|| 5.276638619280e-05 > 62 KSP unpreconditioned resid norm 1.629449737049e+02 true resid norm > 1.605812790996e+02 ||r(i)||/||b|| 5.286152321842e-05 > 63 KSP unpreconditioned resid norm 1.629422151091e+02 true resid norm > 1.589656479615e+02 ||r(i)||/||b|| 5.232967589850e-05 > 64 KSP unpreconditioned resid norm 1.624767340901e+02 true resid norm > 1.601925152173e+02 ||r(i)||/||b|| 5.273354658809e-05 > 65 KSP unpreconditioned resid norm 1.614000473427e+02 true resid norm > 1.600055285874e+02 ||r(i)||/||b|| 5.267199272497e-05 > 66 KSP unpreconditioned resid norm 1.599192711038e+02 true resid norm > 1.602225820054e+02 ||r(i)||/||b|| 5.274344423136e-05 > 67 KSP unpreconditioned resid norm 1.562002802473e+02 true resid norm > 1.582069452329e+02 ||r(i)||/||b|| 5.207991962471e-05 > 68 KSP unpreconditioned resid norm 1.552436010567e+02 true resid norm > 1.584249134588e+02 ||r(i)||/||b|| 5.215167227548e-05 > 69 KSP unpreconditioned resid norm 1.507627069906e+02 true resid norm > 1.530713322210e+02 ||r(i)||/||b|| 5.038933447066e-05 > 70 KSP unpreconditioned resid norm 1.503802419288e+02 true resid norm > 1.526772130725e+02 ||r(i)||/||b|| 5.025959494786e-05 > 71 KSP unpreconditioned resid norm 1.483645684459e+02 true resid norm > 1.509599328686e+02 ||r(i)||/||b|| 4.969428591633e-05 > 72 KSP unpreconditioned resid norm 1.481979533059e+02 true resid norm > 1.535340885300e+02 ||r(i)||/||b|| 5.054166856281e-05 > 73 KSP unpreconditioned resid norm 1.481400704979e+02 true resid norm > 1.509082933863e+02 ||r(i)||/||b|| 4.967728678847e-05 > 74 KSP unpreconditioned resid norm 1.481132272449e+02 true resid norm > 1.513298398754e+02 ||r(i)||/||b|| 4.981605507858e-05 > 75 KSP unpreconditioned resid norm 1.481101708026e+02 true resid norm > 1.502466334943e+02 ||r(i)||/||b|| 4.945947590828e-05 > 76 KSP unpreconditioned resid norm 1.481010335860e+02 true resid norm > 1.533384206564e+02 ||r(i)||/||b|| 5.047725693339e-05 > 77 KSP unpreconditioned resid norm 1.480865328511e+02 true resid norm > 1.508354096349e+02 ||r(i)||/||b|| 4.965329428986e-05 > 78 KSP unpreconditioned resid norm 1.480582653674e+02 true resid norm > 1.493335938981e+02 ||r(i)||/||b|| 4.915891370027e-05 > 79 KSP unpreconditioned resid norm 1.480031554288e+02 true resid norm > 1.505131104808e+02 ||r(i)||/||b|| 4.954719708903e-05 > 80 KSP unpreconditioned resid norm 1.479574822714e+02 true resid norm > 1.540226621640e+02 ||r(i)||/||b|| 5.070250142355e-05 > 81 KSP unpreconditioned resid norm 1.479574535946e+02 true resid norm > 1.498368142318e+02 ||r(i)||/||b|| 4.932456808727e-05 > 82 KSP unpreconditioned resid norm 1.479436001532e+02 true resid norm > 1.512355315895e+02 ||r(i)||/||b|| 4.978500986785e-05 > 83 KSP unpreconditioned resid norm 1.479410419985e+02 true resid norm > 1.513924042216e+02 ||r(i)||/||b|| 4.983665054686e-05 > 84 KSP unpreconditioned resid norm 1.477087197314e+02 true resid norm > 1.519847216835e+02 ||r(i)||/||b|| 5.003163469095e-05 > 85 KSP unpreconditioned resid norm 1.477081559094e+02 true resid norm > 1.507153721984e+02 ||r(i)||/||b|| 4.961377933660e-05 > 86 KSP unpreconditioned resid norm 1.476420890986e+02 true resid norm > 1.512147907360e+02 ||r(i)||/||b|| 4.977818221576e-05 > 87 KSP unpreconditioned resid norm 1.476086929880e+02 true resid norm > 1.508513380647e+02 ||r(i)||/||b|| 4.965853774704e-05 > 88 KSP unpreconditioned resid norm 1.475729830724e+02 true resid norm > 1.521640656963e+02 ||r(i)||/||b|| 5.009067269183e-05 > 89 KSP unpreconditioned resid norm 1.472338605465e+02 true resid norm > 1.506094588356e+02 ||r(i)||/||b|| 4.957891386713e-05 > 90 KSP unpreconditioned resid norm 1.472079944867e+02 true resid norm > 1.504582871439e+02 ||r(i)||/||b|| 4.952914987262e-05 > 91 KSP unpreconditioned resid norm 1.469363056078e+02 true resid norm > 1.506425446156e+02 ||r(i)||/||b|| 4.958980532804e-05 > 92 KSP unpreconditioned resid norm 1.469110799022e+02 true resid norm > 1.509842019134e+02 ||r(i)||/||b|| 4.970227500870e-05 > 93 KSP unpreconditioned resid norm 1.468779696240e+02 true resid norm > 1.501105195969e+02 ||r(i)||/||b|| 4.941466876770e-05 > 94 KSP unpreconditioned resid norm 1.468777757710e+02 true resid norm > 1.491460779150e+02 ||r(i)||/||b|| 4.909718558007e-05 > 95 KSP unpreconditioned resid norm 1.468774588833e+02 true resid norm > 1.519041612996e+02 ||r(i)||/||b|| 5.000511513258e-05 > 96 KSP unpreconditioned resid norm 1.468771672305e+02 true resid norm > 1.508986277767e+02 ||r(i)||/||b|| 4.967410498018e-05 > 97 KSP unpreconditioned resid norm 1.468771086724e+02 true resid norm > 1.500987040931e+02 ||r(i)||/||b|| 4.941077923878e-05 > 98 KSP unpreconditioned resid norm 1.468769529855e+02 true resid norm > 1.509749203169e+02 ||r(i)||/||b|| 4.969921961314e-05 > 99 KSP unpreconditioned resid norm 1.468539019917e+02 true resid norm > 1.505087391266e+02 ||r(i)||/||b|| 4.954575808916e-05 > 100 KSP unpreconditioned resid norm 1.468527260351e+02 true resid norm > 1.519470484364e+02 ||r(i)||/||b|| 5.001923308823e-05 > 101 KSP unpreconditioned resid norm 1.468342327062e+02 true resid norm > 1.489814197970e+02 ||r(i)||/||b|| 4.904298200804e-05 > 102 KSP unpreconditioned resid norm 1.468333201903e+02 true resid norm > 1.491479405434e+02 ||r(i)||/||b|| 4.909779873608e-05 > 103 KSP unpreconditioned resid norm 1.468287736823e+02 true resid norm > 1.496401088908e+02 ||r(i)||/||b|| 4.925981493540e-05 > 104 KSP unpreconditioned resid norm 1.468269778777e+02 true resid norm > 1.509676608058e+02 ||r(i)||/||b|| 4.969682986500e-05 > 105 KSP unpreconditioned resid norm 1.468214752527e+02 true resid norm > 1.500441644659e+02 ||r(i)||/||b|| 4.939282541636e-05 > 106 KSP unpreconditioned resid norm 1.468208033546e+02 true resid norm > 1.510964155942e+02 ||r(i)||/||b|| 4.973921447094e-05 > 107 KSP unpreconditioned resid norm 1.467590018852e+02 true resid norm > 1.512302088409e+02 ||r(i)||/||b|| 4.978325767980e-05 > 108 KSP unpreconditioned resid norm 1.467588908565e+02 true resid norm > 1.501053278370e+02 ||r(i)||/||b|| 4.941295969963e-05 > 109 KSP unpreconditioned resid norm 1.467570731153e+02 true resid norm > 1.485494378220e+02 ||r(i)||/||b|| 4.890077847519e-05 > 110 KSP unpreconditioned resid norm 1.467399860352e+02 true resid norm > 1.504418099302e+02 ||r(i)||/||b|| 4.952372576205e-05 > 111 KSP unpreconditioned resid norm 1.467095654863e+02 true resid norm > 1.507288583410e+02 ||r(i)||/||b|| 4.961821882075e-05 > 112 KSP unpreconditioned resid norm 1.467065865602e+02 true resid norm > 1.517786399520e+02 ||r(i)||/||b|| 4.996379493842e-05 > 113 KSP unpreconditioned resid norm 1.466898232510e+02 true resid norm > 1.491434236258e+02 ||r(i)||/||b|| 4.909631181838e-05 > 114 KSP unpreconditioned resid norm 1.466897921426e+02 true resid norm > 1.505605420512e+02 ||r(i)||/||b|| 4.956281102033e-05 > 115 KSP unpreconditioned resid norm 1.466593121787e+02 true resid norm > 1.500608650677e+02 ||r(i)||/||b|| 4.939832306376e-05 > 116 KSP unpreconditioned resid norm 1.466590894710e+02 true resid norm > 1.503102560128e+02 ||r(i)||/||b|| 4.948041971478e-05 > 117 KSP unpreconditioned resid norm 1.465338856917e+02 true resid norm > 1.501331730933e+02 ||r(i)||/||b|| 4.942212604002e-05 > 118 KSP unpreconditioned resid norm 1.464192893188e+02 true resid norm > 1.505131429801e+02 ||r(i)||/||b|| 4.954720778744e-05 > 119 KSP unpreconditioned resid norm 1.463859793112e+02 true resid norm > 1.504355712014e+02 ||r(i)||/||b|| 4.952167204377e-05 > 120 KSP unpreconditioned resid norm 1.459254939182e+02 true resid norm > 1.526513923221e+02 ||r(i)||/||b|| 5.025109505170e-05 > 121 KSP unpreconditioned resid norm 1.456973020864e+02 true resid norm > 1.496897691500e+02 ||r(i)||/||b|| 4.927616252562e-05 > 122 KSP unpreconditioned resid norm 1.456904663212e+02 true resid norm > 1.488752755634e+02 ||r(i)||/||b|| 4.900804053853e-05 > 123 KSP unpreconditioned resid norm 1.449254956591e+02 true resid norm > 1.494048196254e+02 ||r(i)||/||b|| 4.918236039628e-05 > 124 KSP unpreconditioned resid norm 1.448408616171e+02 true resid norm > 1.507801939332e+02 ||r(i)||/||b|| 4.963511791142e-05 > 125 KSP unpreconditioned resid norm 1.447662934870e+02 true resid norm > 1.495157701445e+02 ||r(i)||/||b|| 4.921888404010e-05 > 126 KSP unpreconditioned resid norm 1.446934748257e+02 true resid norm > 1.511098625097e+02 ||r(i)||/||b|| 4.974364104196e-05 > 127 KSP unpreconditioned resid norm 1.446892504333e+02 true resid norm > 1.493367018275e+02 ||r(i)||/||b|| 4.915993679512e-05 > 128 KSP unpreconditioned resid norm 1.446838883996e+02 true resid norm > 1.510097796622e+02 ||r(i)||/||b|| 4.971069491153e-05 > 129 KSP unpreconditioned resid norm 1.446696373784e+02 true resid norm > 1.463776964101e+02 ||r(i)||/||b|| 4.818586600396e-05 > 130 KSP unpreconditioned resid norm 1.446690766798e+02 true resid norm > 1.495018999638e+02 ||r(i)||/||b|| 4.921431813499e-05 > 131 KSP unpreconditioned resid norm 1.446480744133e+02 true resid norm > 1.499605592408e+02 ||r(i)||/||b|| 4.936530353102e-05 > 132 KSP unpreconditioned resid norm 1.446220543422e+02 true resid norm > 1.498225445439e+02 ||r(i)||/||b|| 4.931987066895e-05 > 133 KSP unpreconditioned resid norm 1.446156526760e+02 true resid norm > 1.481441673781e+02 ||r(i)||/||b|| 4.876736807329e-05 > 134 KSP unpreconditioned resid norm 1.446152477418e+02 true resid norm > 1.501616466283e+02 ||r(i)||/||b|| 4.943149920257e-05 > 135 KSP unpreconditioned resid norm 1.445744489044e+02 true resid norm > 1.505958339620e+02 ||r(i)||/||b|| 4.957442871432e-05 > 136 KSP unpreconditioned resid norm 1.445307936181e+02 true resid norm > 1.502091787932e+02 ||r(i)||/||b|| 4.944714624841e-05 > 137 KSP unpreconditioned resid norm 1.444543817248e+02 true resid norm > 1.491871661616e+02 ||r(i)||/||b|| 4.911071136162e-05 > 138 KSP unpreconditioned resid norm 1.444176915911e+02 true resid norm > 1.478091693367e+02 ||r(i)||/||b|| 4.865709054379e-05 > 139 KSP unpreconditioned resid norm 1.444173719058e+02 true resid norm > 1.495962731374e+02 ||r(i)||/||b|| 4.924538470600e-05 > 140 KSP unpreconditioned resid norm 1.444075340820e+02 true resid norm > 1.515103203654e+02 ||r(i)||/||b|| 4.987546719477e-05 > 141 KSP unpreconditioned resid norm 1.444050342939e+02 true resid norm > 1.498145746307e+02 ||r(i)||/||b|| 4.931724706454e-05 > 142 KSP unpreconditioned resid norm 1.443757787691e+02 true resid norm > 1.492291154146e+02 ||r(i)||/||b|| 4.912452057664e-05 > 143 KSP unpreconditioned resid norm 1.440588930707e+02 true resid norm > 1.485032724987e+02 ||r(i)||/||b|| 4.888558137795e-05 > 144 KSP unpreconditioned resid norm 1.438299468441e+02 true resid norm > 1.506129385276e+02 ||r(i)||/||b|| 4.958005934200e-05 > 145 KSP unpreconditioned resid norm 1.434543079403e+02 true resid norm > 1.471733741230e+02 ||r(i)||/||b|| 4.844779402032e-05 > 146 KSP unpreconditioned resid norm 1.433157223870e+02 true resid norm > 1.481025707968e+02 ||r(i)||/||b|| 4.875367495378e-05 > 147 KSP unpreconditioned resid norm 1.430111913458e+02 true resid norm > 1.485000481919e+02 ||r(i)||/||b|| 4.888451997299e-05 > 148 KSP unpreconditioned resid norm 1.430056153071e+02 true resid norm > 1.496425172884e+02 ||r(i)||/||b|| 4.926060775239e-05 > 149 KSP unpreconditioned resid norm 1.429327762233e+02 true resid norm > 1.467613264791e+02 ||r(i)||/||b|| 4.831215264157e-05 > 150 KSP unpreconditioned resid norm 1.424230217603e+02 true resid norm > 1.460277537447e+02 ||r(i)||/||b|| 4.807066887493e-05 > 151 KSP unpreconditioned resid norm 1.421912821676e+02 true resid norm > 1.470486188164e+02 ||r(i)||/||b|| 4.840672599809e-05 > 152 KSP unpreconditioned resid norm 1.420344275315e+02 true resid norm > 1.481536901943e+02 ||r(i)||/||b|| 4.877050287565e-05 > 153 KSP unpreconditioned resid norm 1.420071178597e+02 true resid norm > 1.450813684108e+02 ||r(i)||/||b|| 4.775912963085e-05 > 154 KSP unpreconditioned resid norm 1.419367456470e+02 true resid norm > 1.472052819440e+02 ||r(i)||/||b|| 4.845829771059e-05 > 155 KSP unpreconditioned resid norm 1.419032748919e+02 true resid norm > 1.479193155584e+02 ||r(i)||/||b|| 4.869334942209e-05 > 156 KSP unpreconditioned resid norm 1.418899781440e+02 true resid norm > 1.478677351572e+02 ||r(i)||/||b|| 4.867636974307e-05 > 157 KSP unpreconditioned resid norm 1.418895621075e+02 true resid norm > 1.455168237674e+02 ||r(i)||/||b|| 4.790247656128e-05 > 158 KSP unpreconditioned resid norm 1.418061469023e+02 true resid norm > 1.467147028974e+02 ||r(i)||/||b|| 4.829680469093e-05 > 159 KSP unpreconditioned resid norm 1.417948698213e+02 true resid norm > 1.478376854834e+02 ||r(i)||/||b|| 4.866647773362e-05 > 160 KSP unpreconditioned resid norm 1.415166832324e+02 true resid norm > 1.475436433192e+02 ||r(i)||/||b|| 4.856968241116e-05 > 161 KSP unpreconditioned resid norm 1.414939087573e+02 true resid norm > 1.468361945080e+02 ||r(i)||/||b|| 4.833679834170e-05 > 162 KSP unpreconditioned resid norm 1.414544622036e+02 true resid norm > 1.475730757600e+02 ||r(i)||/||b|| 4.857937123456e-05 > 163 KSP unpreconditioned resid norm 1.413780373982e+02 true resid norm > 1.463891808066e+02 ||r(i)||/||b|| 4.818964653614e-05 > 164 KSP unpreconditioned resid norm 1.413741853943e+02 true resid norm > 1.481999741168e+02 ||r(i)||/||b|| 4.878573901436e-05 > 165 KSP unpreconditioned resid norm 1.413725682642e+02 true resid norm > 1.458413423932e+02 ||r(i)||/||b|| 4.800930438685e-05 > 166 KSP unpreconditioned resid norm 1.412970845566e+02 true resid norm > 1.481492296610e+02 ||r(i)||/||b|| 4.876903451901e-05 > 167 KSP unpreconditioned resid norm 1.410100899597e+02 true resid norm > 1.468338434340e+02 ||r(i)||/||b|| 4.833602439497e-05 > 168 KSP unpreconditioned resid norm 1.409983320599e+02 true resid norm > 1.485378957202e+02 ||r(i)||/||b|| 4.889697894709e-05 > 169 KSP unpreconditioned resid norm 1.407688141293e+02 true resid norm > 1.461003623074e+02 ||r(i)||/||b|| 4.809457078458e-05 > 170 KSP unpreconditioned resid norm 1.407072771004e+02 true resid norm > 1.463217409181e+02 ||r(i)||/||b|| 4.816744609502e-05 > 171 KSP unpreconditioned resid norm 1.407069670790e+02 true resid norm > 1.464695099700e+02 ||r(i)||/||b|| 4.821608997937e-05 > 172 KSP unpreconditioned resid norm 1.402361094414e+02 true resid norm > 1.493786053835e+02 ||r(i)||/||b|| 4.917373096721e-05 > 173 KSP unpreconditioned resid norm 1.400618325859e+02 true resid norm > 1.465475533254e+02 ||r(i)||/||b|| 4.824178096070e-05 > 174 KSP unpreconditioned resid norm 1.400573078320e+02 true resid norm > 1.471993735980e+02 ||r(i)||/||b|| 4.845635275056e-05 > 175 KSP unpreconditioned resid norm 1.400258865388e+02 true resid norm > 1.479779387468e+02 ||r(i)||/||b|| 4.871264750624e-05 > 176 KSP unpreconditioned resid norm 1.396589283831e+02 true resid norm > 1.476626943974e+02 ||r(i)||/||b|| 4.860887266654e-05 > 177 KSP unpreconditioned resid norm 1.395796112440e+02 true resid norm > 1.443093901655e+02 ||r(i)||/||b|| 4.750500320860e-05 > 178 KSP unpreconditioned resid norm 1.394749154493e+02 true resid norm > 1.447914005206e+02 ||r(i)||/||b|| 4.766367551289e-05 > 179 KSP unpreconditioned resid norm 1.394476969416e+02 true resid norm > 1.455635964329e+02 ||r(i)||/||b|| 4.791787358864e-05 > 180 KSP unpreconditioned resid norm 1.391990722790e+02 true resid norm > 1.457511594620e+02 ||r(i)||/||b|| 4.797961719582e-05 > 181 KSP unpreconditioned resid norm 1.391686315799e+02 true resid norm > 1.460567495143e+02 ||r(i)||/||b|| 4.808021395114e-05 > 182 KSP unpreconditioned resid norm 1.387654475794e+02 true resid norm > 1.468215388414e+02 ||r(i)||/||b|| 4.833197386362e-05 > 183 KSP unpreconditioned resid norm 1.384925240232e+02 true resid norm > 1.456091052791e+02 ||r(i)||/||b|| 4.793285458106e-05 > 184 KSP unpreconditioned resid norm 1.378003249970e+02 true resid norm > 1.453421051371e+02 ||r(i)||/||b|| 4.784496118351e-05 > 185 KSP unpreconditioned resid norm 1.377904214978e+02 true resid norm > 1.441752187090e+02 ||r(i)||/||b|| 4.746083549740e-05 > 186 KSP unpreconditioned resid norm 1.376670282479e+02 true resid norm > 1.441674745344e+02 ||r(i)||/||b|| 4.745828620353e-05 > 187 KSP unpreconditioned resid norm 1.376636051755e+02 true resid norm > 1.463118783906e+02 ||r(i)||/||b|| 4.816419946362e-05 > 188 KSP unpreconditioned resid norm 1.363148994276e+02 true resid norm > 1.432997756128e+02 ||r(i)||/||b|| 4.717264962781e-05 > 189 KSP unpreconditioned resid norm 1.363051099558e+02 true resid norm > 1.451009062639e+02 ||r(i)||/||b|| 4.776556126897e-05 > 190 KSP unpreconditioned resid norm 1.362538398564e+02 true resid norm > 1.438957985476e+02 ||r(i)||/||b|| 4.736885357127e-05 > 191 KSP unpreconditioned resid norm 1.358335705250e+02 true resid norm > 1.436616069458e+02 ||r(i)||/||b|| 4.729176037047e-05 > 192 KSP unpreconditioned resid norm 1.337424103882e+02 true resid norm > 1.432816138672e+02 ||r(i)||/||b|| 4.716667098856e-05 > 193 KSP unpreconditioned resid norm 1.337419543121e+02 true resid norm > 1.405274691954e+02 ||r(i)||/||b|| 4.626003801533e-05 > 194 KSP unpreconditioned resid norm 1.322568117657e+02 true resid norm > 1.417123189671e+02 ||r(i)||/||b|| 4.665007702902e-05 > 195 KSP unpreconditioned resid norm 1.320880115122e+02 true resid norm > 1.413658215058e+02 ||r(i)||/||b|| 4.653601402181e-05 > 196 KSP unpreconditioned resid norm 1.312526182172e+02 true resid norm > 1.420574070412e+02 ||r(i)||/||b|| 4.676367608204e-05 > 197 KSP unpreconditioned resid norm 1.311651332692e+02 true resid norm > 1.398984125128e+02 ||r(i)||/||b|| 4.605295973934e-05 > 198 KSP unpreconditioned resid norm 1.294482397720e+02 true resid norm > 1.380390703259e+02 ||r(i)||/||b|| 4.544088552537e-05 > 199 KSP unpreconditioned resid norm 1.293598434732e+02 true resid norm > 1.373830689903e+02 ||r(i)||/||b|| 4.522493737731e-05 > 200 KSP unpreconditioned resid norm 1.265165992897e+02 true resid norm > 1.375015523244e+02 ||r(i)||/||b|| 4.526394073779e-05 > 201 KSP unpreconditioned resid norm 1.263813235463e+02 true resid norm > 1.356820166419e+02 ||r(i)||/||b|| 4.466497037047e-05 > 202 KSP unpreconditioned resid norm 1.243190164198e+02 true resid norm > 1.366420975402e+02 ||r(i)||/||b|| 4.498101803792e-05 > 203 KSP unpreconditioned resid norm 1.230747513665e+02 true resid norm > 1.348856851681e+02 ||r(i)||/||b|| 4.440282714351e-05 > 204 KSP unpreconditioned resid norm 1.198014010398e+02 true resid norm > 1.325188356617e+02 ||r(i)||/||b|| 4.362368731578e-05 > 205 KSP unpreconditioned resid norm 1.195977240348e+02 true resid norm > 1.299721846860e+02 ||r(i)||/||b|| 4.278535889769e-05 > 206 KSP unpreconditioned resid norm 1.130620928393e+02 true resid norm > 1.266961052950e+02 ||r(i)||/||b|| 4.170691097546e-05 > 207 KSP unpreconditioned resid norm 1.123992882530e+02 true resid norm > 1.270907813369e+02 ||r(i)||/||b|| 4.183683382120e-05 > 208 KSP unpreconditioned resid norm 1.063236317163e+02 true resid norm > 1.182163029843e+02 ||r(i)||/||b|| 3.891545689533e-05 > 209 KSP unpreconditioned resid norm 1.059802897214e+02 true resid norm > 1.187516613498e+02 ||r(i)||/||b|| 3.909169075539e-05 > 210 KSP unpreconditioned resid norm 9.878733567790e+01 true resid norm > 1.124812677115e+02 ||r(i)||/||b|| 3.702754877846e-05 > 211 KSP unpreconditioned resid norm 9.861048081032e+01 true resid norm > 1.117192174341e+02 ||r(i)||/||b|| 3.677669052986e-05 > 212 KSP unpreconditioned resid norm 9.169383217455e+01 true resid norm > 1.102172324977e+02 ||r(i)||/||b|| 3.628225424167e-05 > 213 KSP unpreconditioned resid norm 9.146164223196e+01 true resid norm > 1.121134424773e+02 ||r(i)||/||b|| 3.690646491198e-05 > 214 KSP unpreconditioned resid norm 8.692213412954e+01 true resid norm > 1.056264039532e+02 ||r(i)||/||b|| 3.477100591276e-05 > 215 KSP unpreconditioned resid norm 8.685846611574e+01 true resid norm > 1.029018845366e+02 ||r(i)||/||b|| 3.387412523521e-05 > 216 KSP unpreconditioned resid norm 7.808516472373e+01 true resid norm > 9.749023000535e+01 ||r(i)||/||b|| 3.209267036539e-05 > 217 KSP unpreconditioned resid norm 7.786400257086e+01 true resid norm > 1.004515546585e+02 ||r(i)||/||b|| 3.306750462244e-05 > 218 KSP unpreconditioned resid norm 6.646475864029e+01 true resid norm > 9.429020541969e+01 ||r(i)||/||b|| 3.103925881653e-05 > 219 KSP unpreconditioned resid norm 6.643821996375e+01 true resid norm > 8.864525788550e+01 ||r(i)||/||b|| 2.918100655438e-05 > 220 KSP unpreconditioned resid norm 5.625046780791e+01 true resid norm > 8.410041684883e+01 ||r(i)||/||b|| 2.768489678784e-05 > 221 KSP unpreconditioned resid norm 5.623343238032e+01 true resid norm > 8.815552919640e+01 ||r(i)||/||b|| 2.901979346270e-05 > 222 KSP unpreconditioned resid norm 4.491016868776e+01 true resid norm > 8.557052117768e+01 ||r(i)||/||b|| 2.816883834410e-05 > 223 KSP unpreconditioned resid norm 4.461976108543e+01 true resid norm > 7.867894425332e+01 ||r(i)||/||b|| 2.590020992340e-05 > 224 KSP unpreconditioned resid norm 3.535718264955e+01 true resid norm > 7.609346753983e+01 ||r(i)||/||b|| 2.504910051583e-05 > 225 KSP unpreconditioned resid norm 3.525592897743e+01 true resid norm > 7.926812413349e+01 ||r(i)||/||b|| 2.609416121143e-05 > 226 KSP unpreconditioned resid norm 2.633469451114e+01 true resid norm > 7.883483297310e+01 ||r(i)||/||b|| 2.595152670968e-05 > 227 KSP unpreconditioned resid norm 2.614440577316e+01 true resid norm > 7.398963634249e+01 ||r(i)||/||b|| 2.435654331172e-05 > 228 KSP unpreconditioned resid norm 1.988460252721e+01 true resid norm > 7.147825835126e+01 ||r(i)||/||b|| 2.352982635730e-05 > 229 KSP unpreconditioned resid norm 1.975927240058e+01 true resid norm > 7.488507147714e+01 ||r(i)||/||b|| 2.465131033205e-05 > 230 KSP unpreconditioned resid norm 1.505732242656e+01 true resid norm > 7.888901529160e+01 ||r(i)||/||b|| 2.596936291016e-05 > 231 KSP unpreconditioned resid norm 1.504120870628e+01 true resid norm > 7.126366562975e+01 ||r(i)||/||b|| 2.345918488406e-05 > 232 KSP unpreconditioned resid norm 1.163470506257e+01 true resid norm > 7.142763663542e+01 ||r(i)||/||b|| 2.351316226655e-05 > 233 KSP unpreconditioned resid norm 1.157114340949e+01 true resid norm > 7.464790352976e+01 ||r(i)||/||b|| 2.457323735226e-05 > 234 KSP unpreconditioned resid norm 8.702850618357e+00 true resid norm > 7.798031063059e+01 ||r(i)||/||b|| 2.567022771329e-05 > 235 KSP unpreconditioned resid norm 8.702017371082e+00 true resid norm > 7.032943782131e+01 ||r(i)||/||b|| 2.315164775854e-05 > 236 KSP unpreconditioned resid norm 6.422855779486e+00 true resid norm > 6.800345168870e+01 ||r(i)||/||b|| 2.238595968678e-05 > 237 KSP unpreconditioned resid norm 6.413921210094e+00 true resid norm > 7.408432731879e+01 ||r(i)||/||b|| 2.438771449973e-05 > 238 KSP unpreconditioned resid norm 4.949111361190e+00 true resid norm > 7.744087979524e+01 ||r(i)||/||b|| 2.549265324267e-05 > 239 KSP unpreconditioned resid norm 4.947369357666e+00 true resid norm > 7.104259266677e+01 ||r(i)||/||b|| 2.338641018933e-05 > 240 KSP unpreconditioned resid norm 3.873645232239e+00 true resid norm > 6.908028336929e+01 ||r(i)||/||b|| 2.274044037845e-05 > 241 KSP unpreconditioned resid norm 3.841473653930e+00 true resid norm > 7.431718972562e+01 ||r(i)||/||b|| 2.446437014474e-05 > 242 KSP unpreconditioned resid norm 3.057267436362e+00 true resid norm > 7.685939322732e+01 ||r(i)||/||b|| 2.530123450517e-05 > 243 KSP unpreconditioned resid norm 2.980906717815e+00 true resid norm > 6.975661521135e+01 ||r(i)||/||b|| 2.296308109705e-05 > 244 KSP unpreconditioned resid norm 2.415633545154e+00 true resid norm > 6.989644258184e+01 ||r(i)||/||b|| 2.300911067057e-05 > 245 KSP unpreconditioned resid norm 2.363923146996e+00 true resid norm > 7.486631867276e+01 ||r(i)||/||b|| 2.464513712301e-05 > 246 KSP unpreconditioned resid norm 1.947823635306e+00 true resid norm > 7.671103669547e+01 ||r(i)||/||b|| 2.525239722914e-05 > 247 KSP unpreconditioned resid norm 1.942156637334e+00 true resid norm > 6.835715877902e+01 ||r(i)||/||b|| 2.250239602152e-05 > 248 KSP unpreconditioned resid norm 1.675749569790e+00 true resid norm > 7.111781390782e+01 ||r(i)||/||b|| 2.341117216285e-05 > 249 KSP unpreconditioned resid norm 1.673819729570e+00 true resid norm > 7.552508026111e+01 ||r(i)||/||b|| 2.486199391474e-05 > 250 KSP unpreconditioned resid norm 1.453311843294e+00 true resid norm > 7.639099426865e+01 ||r(i)||/||b|| 2.514704291716e-05 > 251 KSP unpreconditioned resid norm 1.452846325098e+00 true resid norm > 6.951401359923e+01 ||r(i)||/||b|| 2.288321941689e-05 > 252 KSP unpreconditioned resid norm 1.335008887441e+00 true resid norm > 6.912230871414e+01 ||r(i)||/||b|| 2.275427464204e-05 > 253 KSP unpreconditioned resid norm 1.334477013356e+00 true resid norm > 7.412281497148e+01 ||r(i)||/||b|| 2.440038419546e-05 > 254 KSP unpreconditioned resid norm 1.248507835050e+00 true resid norm > 7.801932499175e+01 ||r(i)||/||b|| 2.568307079543e-05 > 255 KSP unpreconditioned resid norm 1.248246596771e+00 true resid norm > 7.094899926215e+01 ||r(i)||/||b|| 2.335560030938e-05 > 256 KSP unpreconditioned resid norm 1.208952722414e+00 true resid norm > 7.101235824005e+01 ||r(i)||/||b|| 2.337645736134e-05 > 257 KSP unpreconditioned resid norm 1.208780664971e+00 true resid norm > 7.562936418444e+01 ||r(i)||/||b|| 2.489632299136e-05 > 258 KSP unpreconditioned resid norm 1.179956701653e+00 true resid norm > 7.812300941072e+01 ||r(i)||/||b|| 2.571720252207e-05 > 259 KSP unpreconditioned resid norm 1.179219541297e+00 true resid norm > 7.131201918549e+01 ||r(i)||/||b|| 2.347510232240e-05 > 260 KSP unpreconditioned resid norm 1.160215487467e+00 true resid norm > 7.222079766175e+01 ||r(i)||/||b|| 2.377426181841e-05 > 261 KSP unpreconditioned resid norm 1.159115040554e+00 true resid norm > 7.481372509179e+01 ||r(i)||/||b|| 2.462782391678e-05 > 262 KSP unpreconditioned resid norm 1.151973184765e+00 true resid norm > 7.709040836137e+01 ||r(i)||/||b|| 2.537728204907e-05 > 263 KSP unpreconditioned resid norm 1.150882463576e+00 true resid norm > 7.032588895526e+01 ||r(i)||/||b|| 2.315047951236e-05 > 264 KSP unpreconditioned resid norm 1.137617003277e+00 true resid norm > 7.004055871264e+01 ||r(i)||/||b|| 2.305655205500e-05 > 265 KSP unpreconditioned resid norm 1.137134003401e+00 true resid norm > 7.610459827221e+01 ||r(i)||/||b|| 2.505276462582e-05 > 266 KSP unpreconditioned resid norm 1.131425778253e+00 true resid norm > 7.852741072990e+01 ||r(i)||/||b|| 2.585032681802e-05 > 267 KSP unpreconditioned resid norm 1.131176695314e+00 true resid norm > 7.064571495865e+01 ||r(i)||/||b|| 2.325576258022e-05 > 268 KSP unpreconditioned resid norm 1.125420065063e+00 true resid norm > 7.138837220124e+01 ||r(i)||/||b|| 2.350023686323e-05 > 269 KSP unpreconditioned resid norm 1.124779989266e+00 true resid norm > 7.585594020759e+01 ||r(i)||/||b|| 2.497090923065e-05 > 270 KSP unpreconditioned resid norm 1.119805446125e+00 true resid norm > 7.703631305135e+01 ||r(i)||/||b|| 2.535947449079e-05 > 271 KSP unpreconditioned resid norm 1.119024433863e+00 true resid norm > 7.081439585094e+01 ||r(i)||/||b|| 2.331129040360e-05 > 272 KSP unpreconditioned resid norm 1.115694452861e+00 true resid norm > 7.134872343512e+01 ||r(i)||/||b|| 2.348718494222e-05 > 273 KSP unpreconditioned resid norm 1.113572716158e+00 true resid norm > 7.600475566242e+01 ||r(i)||/||b|| 2.501989757889e-05 > 274 KSP unpreconditioned resid norm 1.108711406381e+00 true resid norm > 7.738835220359e+01 ||r(i)||/||b|| 2.547536175937e-05 > 275 KSP unpreconditioned resid norm 1.107890435549e+00 true resid norm > 7.093429729336e+01 ||r(i)||/||b|| 2.335076058915e-05 > 276 KSP unpreconditioned resid norm 1.103340227961e+00 true resid norm > 7.145267197866e+01 ||r(i)||/||b|| 2.352140361564e-05 > 277 KSP unpreconditioned resid norm 1.102897652964e+00 true resid norm > 7.448617654625e+01 ||r(i)||/||b|| 2.451999867624e-05 > 278 KSP unpreconditioned resid norm 1.102576754158e+00 true resid norm > 7.707165090465e+01 ||r(i)||/||b|| 2.537110730854e-05 > 279 KSP unpreconditioned resid norm 1.102564028537e+00 true resid norm > 7.009637628868e+01 ||r(i)||/||b|| 2.307492656359e-05 > 280 KSP unpreconditioned resid norm 1.100828424712e+00 true resid norm > 7.059832880916e+01 ||r(i)||/||b|| 2.324016360096e-05 > 281 KSP unpreconditioned resid norm 1.100686341559e+00 true resid norm > 7.460867988528e+01 ||r(i)||/||b|| 2.456032537644e-05 > 282 KSP unpreconditioned resid norm 1.099417185996e+00 true resid norm > 7.763784632467e+01 ||r(i)||/||b|| 2.555749237477e-05 > 283 KSP unpreconditioned resid norm 1.099379061087e+00 true resid norm > 7.017139420999e+01 ||r(i)||/||b|| 2.309962160657e-05 > 284 KSP unpreconditioned resid norm 1.097928047676e+00 true resid norm > 6.983706716123e+01 ||r(i)||/||b|| 2.298956496018e-05 > 285 KSP unpreconditioned resid norm 1.096490152934e+00 true resid norm > 7.414445779601e+01 ||r(i)||/||b|| 2.440750876614e-05 > 286 KSP unpreconditioned resid norm 1.094691490227e+00 true resid norm > 7.634526287231e+01 ||r(i)||/||b|| 2.513198866374e-05 > 287 KSP unpreconditioned resid norm 1.093560358328e+00 true resid norm > 7.003716824146e+01 ||r(i)||/||b|| 2.305543595061e-05 > 288 KSP unpreconditioned resid norm 1.093357856424e+00 true resid norm > 6.964715939684e+01 ||r(i)||/||b|| 2.292704949292e-05 > 289 KSP unpreconditioned resid norm 1.091881434739e+00 true resid norm > 7.429955169250e+01 ||r(i)||/||b|| 2.445856390566e-05 > 290 KSP unpreconditioned resid norm 1.091817808496e+00 true resid norm > 7.607892786798e+01 ||r(i)||/||b|| 2.504431422190e-05 > 291 KSP unpreconditioned resid norm 1.090295101202e+00 true resid norm > 6.942248339413e+01 ||r(i)||/||b|| 2.285308871866e-05 > 292 KSP unpreconditioned resid norm 1.089995012773e+00 true resid norm > 6.995557798353e+01 ||r(i)||/||b|| 2.302857736947e-05 > 293 KSP unpreconditioned resid norm 1.089975910578e+00 true resid norm > 7.453210925277e+01 ||r(i)||/||b|| 2.453511919866e-05 > 294 KSP unpreconditioned resid norm 1.085570944646e+00 true resid norm > 7.629598425927e+01 ||r(i)||/||b|| 2.511576670710e-05 > 295 KSP unpreconditioned resid norm 1.085363565621e+00 true resid norm > 7.025539955712e+01 ||r(i)||/||b|| 2.312727520749e-05 > 296 KSP unpreconditioned resid norm 1.083348574106e+00 true resid norm > 7.003219621882e+01 ||r(i)||/||b|| 2.305379921754e-05 > 297 KSP unpreconditioned resid norm 1.082180374430e+00 true resid norm > 7.473048827106e+01 ||r(i)||/||b|| 2.460042330597e-05 > 298 KSP unpreconditioned resid norm 1.081326671068e+00 true resid norm > 7.660142838935e+01 ||r(i)||/||b|| 2.521631542651e-05 > 299 KSP unpreconditioned resid norm 1.078679751898e+00 true resid norm > 7.077868424247e+01 ||r(i)||/||b|| 2.329953454992e-05 > 300 KSP unpreconditioned resid norm 1.078656949888e+00 true resid norm > 7.074960394994e+01 ||r(i)||/||b|| 2.328996164972e-05 > Linear solve did not converge due to DIVERGED_ITS iterations 300 > KSP Object: 2 MPI processes > type: fgmres > GMRES: restart=300, using Modified Gram-Schmidt Orthogonalization > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=300, initial guess is zero > tolerances: relative=1e-09, absolute=1e-20, divergence=10000 > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 2 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization DIAG > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses (lumped, if requested) A00's diagonal's > inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_u_) 2 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_u_) 2 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0, needed 0 > Factored matrix follows: > Mat Object: 2 MPI processes > type: mpiaij > rows=184326, cols=184326 > package used to perform factorization: mumps > total: nonzeros=4.03041e+08, allocated nonzeros=4.03041e+08 > total number of mallocs used during MatSetValues calls =0 > MUMPS run parameters: > SYM (matrix type): 0 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequentia matrix ordering):7 > ICNTL(8) (scalling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): > 1 > ICNTL(13) (efficiency control): > 0 > ICNTL(14) (percentage of estimated workspace > increase): 20 > ICNTL(18) (input mat struct): > 3 > ICNTL(19) (Shur complement info): > 0 > ICNTL(20) (rhs sparse pattern): > 0 > ICNTL(21) (solution struct): > 1 > ICNTL(22) (in-core/out-of-core facility): > 0 > ICNTL(23) (max size of memory can be allocated > locally):0 > ICNTL(24) (detection of null pivot rows): > 0 > ICNTL(25) (computation of a null space basis): > 0 > ICNTL(26) (Schur options for rhs or solution): > 0 > ICNTL(27) (experimental parameter): > -24 > ICNTL(28) (use parallel or sequential ordering): > 1 > ICNTL(29) (parallel ordering): > 0 > ICNTL(30) (user-specified set of entries in inv(A)): > 0 > ICNTL(31) (factors is discarded in the solve phase): > 0 > ICNTL(33) (compute determinant): > 0 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): > 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0 > CNTL(4) (value of static pivoting): -1 > CNTL(5) (fixation for null pivots): 0 > RINFO(1) (local estimated flops for the elimination > after analysis): > [0] 5.59214e+11 > [1] 5.35237e+11 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 4.2839e+08 > [1] 3.799e+08 > RINFO(3) (local estimated flops for the elimination > after factorization): > [0] 5.59214e+11 > [1] 5.35237e+11 > INFO(15) (estimated size of (in MB) MUMPS internal > data for running numerical factorization): > [0] 2621 > [1] 2649 > INFO(16) (size of (in MB) MUMPS internal data used > during numerical factorization): > [0] 2621 > [1] 2649 > INFO(23) (num of pivots eliminated on this processor > after factorization): > [0] 90423 > [1] 93903 > RINFOG(1) (global estimated flops for the elimination > after analysis): 1.09445e+12 > RINFOG(2) (global estimated flops for the assembly > after factorization): 8.0829e+08 > RINFOG(3) (global estimated flops for the elimination > after factorization): 1.09445e+12 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0,0)*(2^0) > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 403041366 > INFOG(4) (estimated integer workspace for factors on > all processors after analysis): 2265748 > INFOG(5) (estimated maximum front size in the complete > tree): 6663 > INFOG(6) (number of nodes in the complete tree): 2812 > INFOG(7) (ordering option effectively use after > analysis): 5 > INFOG(8) (structural symmetry in percent of the > permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the > matrix factors after factorization): 403041366 > INFOG(10) (total integer space store the matrix > factors after factorization): 2265766 > INFOG(11) (order of largest frontal matrix after > factorization): 6663 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after > factorization): 0 > INFOG(14) (number of memory compress after > factorization): 0 > INFOG(15) (number of steps of iterative refinement > after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS > internal data for factorization after analysis: value on the most memory > consuming processor): 2649 > INFOG(17) (estimated size of all MUMPS internal data > for factorization after analysis: sum over all processors): 5270 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): 2649 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 5270 > INFOG(20) (estimated number of entries in the > factors): 403041366 > INFOG(21) (size in MB of memory effectively used > during factorization - value on the most memory consuming processor): 2121 > INFOG(22) (size in MB of memory effectively used > during factorization - sum over all processors): 4174 > INFOG(23) (after analysis: value of ICNTL(6) > effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) > effectively used): 1 > INFOG(25) (after factorization: number of pivots > modified by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots > encountered): 0 > INFOG(29) (after factorization: effective number of > entries in the factors (sum over all processors)): 403041366 > INFOG(30, 31) (after solution: size in Mbytes of > memory used during solution phase): 2467, 4922 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant > is requested): 0 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_u_) 2 MPI processes > type: mpiaij > rows=184326, cols=184326, bs=3 > total: nonzeros=3.32649e+07, allocated nonzeros=3.32649e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 26829 nodes, limit > used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_lu_) 2 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_lu_) 2 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0, needed 0 > Factored matrix follows: > Mat Object: 2 MPI processes > type: mpiaij > rows=2583, cols=2583 > package used to perform factorization: mumps > total: nonzeros=2.17621e+06, allocated nonzeros=2.17621e+06 > total number of mallocs used during MatSetValues calls =0 > MUMPS run parameters: > SYM (matrix type): 0 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequentia matrix ordering):7 > ICNTL(8) (scalling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): > 1 > ICNTL(13) (efficiency control): > 0 > ICNTL(14) (percentage of estimated workspace > increase): 20 > ICNTL(18) (input mat struct): > 3 > ICNTL(19) (Shur complement info): > 0 > ICNTL(20) (rhs sparse pattern): > 0 > ICNTL(21) (solution struct): > 1 > ICNTL(22) (in-core/out-of-core facility): > 0 > ICNTL(23) (max size of memory can be allocated > locally):0 > ICNTL(24) (detection of null pivot rows): > 0 > ICNTL(25) (computation of a null space basis): > 0 > ICNTL(26) (Schur options for rhs or solution): > 0 > ICNTL(27) (experimental parameter): > -24 > ICNTL(28) (use parallel or sequential ordering): > 1 > ICNTL(29) (parallel ordering): > 0 > ICNTL(30) (user-specified set of entries in inv(A)): > 0 > ICNTL(31) (factors is discarded in the solve phase): > 0 > ICNTL(33) (compute determinant): > 0 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): > 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0 > CNTL(4) (value of static pivoting): -1 > CNTL(5) (fixation for null pivots): 0 > RINFO(1) (local estimated flops for the elimination > after analysis): > [0] 5.12794e+08 > [1] 5.02142e+08 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 815031 > [1] 745263 > RINFO(3) (local estimated flops for the elimination > after factorization): > [0] 5.12794e+08 > [1] 5.02142e+08 > INFO(15) (estimated size of (in MB) MUMPS internal > data for running numerical factorization): > [0] 34 > [1] 34 > INFO(16) (size of (in MB) MUMPS internal data used > during numerical factorization): > [0] 34 > [1] 34 > INFO(23) (num of pivots eliminated on this processor > after factorization): > [0] 1158 > [1] 1425 > RINFOG(1) (global estimated flops for the elimination > after analysis): 1.01494e+09 > RINFOG(2) (global estimated flops for the assembly > after factorization): 1.56029e+06 > RINFOG(3) (global estimated flops for the elimination > after factorization): 1.01494e+09 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0,0)*(2^0) > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 2176209 > INFOG(4) (estimated integer workspace for factors on > all processors after analysis): 14427 > INFOG(5) (estimated maximum front size in the complete > tree): 699 > INFOG(6) (number of nodes in the complete tree): 15 > INFOG(7) (ordering option effectively use after > analysis): 2 > INFOG(8) (structural symmetry in percent of the > permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the > matrix factors after factorization): 2176209 > INFOG(10) (total integer space store the matrix > factors after factorization): 14427 > INFOG(11) (order of largest frontal matrix after > factorization): 699 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after > factorization): 0 > INFOG(14) (number of memory compress after > factorization): 0 > INFOG(15) (number of steps of iterative refinement > after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS > internal data for factorization after analysis: value on the most memory > consuming processor): 34 > INFOG(17) (estimated size of all MUMPS internal data > for factorization after analysis: sum over all processors): 68 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): 34 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 68 > INFOG(20) (estimated number of entries in the > factors): 2176209 > INFOG(21) (size in MB of memory effectively used > during factorization - value on the most memory consuming processor): 30 > INFOG(22) (size in MB of memory effectively used > during factorization - sum over all processors): 59 > INFOG(23) (after analysis: value of ICNTL(6) > effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) > effectively used): 1 > INFOG(25) (after factorization: number of pivots > modified by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots > encountered): 0 > INFOG(29) (after factorization: effective number of > entries in the factors (sum over all processors)): 2176209 > INFOG(30, 31) (after solution: size in Mbytes of > memory used during solution phase): 16, 32 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant > is requested): 0 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_lu_) 2 MPI processes > type: schurcomplement > rows=2583, cols=2583 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_lu_) 2 > MPI processes > type: mpiaij > rows=2583, cols=2583, bs=3 > total: nonzeros=117369, allocated nonzeros=117369 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > A10 > Mat Object: 2 MPI processes > type: mpiaij > rows=2583, cols=184326, rbs=3, cbs = 1 > total: nonzeros=292770, allocated nonzeros=292770 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > KSP of A00 > KSP Object: (fieldsplit_u_) 2 MPI > processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_u_) 2 MPI > processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 0, needed 0 > Factored matrix follows: > Mat Object: 2 MPI processes > type: mpiaij > rows=184326, cols=184326 > package used to perform factorization: mumps > total: nonzeros=4.03041e+08, allocated > nonzeros=4.03041e+08 > total number of mallocs used during MatSetValues > calls =0 > MUMPS run parameters: > SYM (matrix type): 0 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequentia matrix ordering):7 > ICNTL(8) (scalling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): > 1 > ICNTL(13) (efficiency control): > 0 > ICNTL(14) (percentage of estimated workspace > increase): 20 > ICNTL(18) (input mat struct): > 3 > ICNTL(19) (Shur complement info): > 0 > ICNTL(20) (rhs sparse pattern): > 0 > ICNTL(21) (solution struct): > 1 > ICNTL(22) (in-core/out-of-core facility): > 0 > ICNTL(23) (max size of memory can be allocated > locally):0 > ICNTL(24) (detection of null pivot rows): > 0 > ICNTL(25) (computation of a null space basis): > 0 > ICNTL(26) (Schur options for rhs or solution): > 0 > ICNTL(27) (experimental parameter): > -24 > ICNTL(28) (use parallel or sequential > ordering): 1 > ICNTL(29) (parallel ordering): > 0 > ICNTL(30) (user-specified set of entries in > inv(A)): 0 > ICNTL(31) (factors is discarded in the solve > phase): 0 > ICNTL(33) (compute determinant): > 0 > CNTL(1) (relative pivoting threshold): > 0.01 > CNTL(2) (stopping criterion of refinement): > 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0 > CNTL(4) (value of static pivoting): -1 > CNTL(5) (fixation for null pivots): 0 > RINFO(1) (local estimated flops for the > elimination after analysis): > [0] 5.59214e+11 > [1] 5.35237e+11 > RINFO(2) (local estimated flops for the > assembly after factorization): > [0] 4.2839e+08 > [1] 3.799e+08 > RINFO(3) (local estimated flops for the > elimination after factorization): > [0] 5.59214e+11 > [1] 5.35237e+11 > INFO(15) (estimated size of (in MB) MUMPS > internal data for running numerical factorization): > [0] 2621 > [1] 2649 > INFO(16) (size of (in MB) MUMPS internal data > used during numerical factorization): > [0] 2621 > [1] 2649 > INFO(23) (num of pivots eliminated on this > processor after factorization): > [0] 90423 > [1] 93903 > RINFOG(1) (global estimated flops for the > elimination after analysis): 1.09445e+12 > RINFOG(2) (global estimated flops for the > assembly after factorization): 8.0829e+08 > RINFOG(3) (global estimated flops for the > elimination after factorization): 1.09445e+12 > (RINFOG(12) RINFOG(13))*2^INFOG(34) > (determinant): (0,0)*(2^0) > INFOG(3) (estimated real workspace for factors > on all processors after analysis): 403041366 > INFOG(4) (estimated integer workspace for > factors on all processors after analysis): 2265748 > INFOG(5) (estimated maximum front size in the > complete tree): 6663 > INFOG(6) (number of nodes in the complete > tree): 2812 > INFOG(7) (ordering option effectively use > after analysis): 5 > INFOG(8) (structural symmetry in percent of > the permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to > store the matrix factors after factorization): 403041366 > INFOG(10) (total integer space store the > matrix factors after factorization): 2265766 > INFOG(11) (order of largest frontal matrix > after factorization): 6663 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after > factorization): 0 > INFOG(14) (number of memory compress after > factorization): 0 > INFOG(15) (number of steps of iterative > refinement after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS > internal data for factorization after analysis: value on the most memory > consuming processor): 2649 > INFOG(17) (estimated size of all MUMPS > internal data for factorization after analysis: sum over all processors): > 5270 > INFOG(18) (size of all MUMPS internal data > allocated during factorization: value on the most memory consuming > processor): 2649 > INFOG(19) (size of all MUMPS internal data > allocated during factorization: sum over all processors): 5270 > INFOG(20) (estimated number of entries in the > factors): 403041366 > INFOG(21) (size in MB of memory effectively > used during factorization - value on the most memory consuming processor): > 2121 > INFOG(22) (size in MB of memory effectively > used during factorization - sum over all processors): 4174 > INFOG(23) (after analysis: value of ICNTL(6) > effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) > effectively used): 1 > INFOG(25) (after factorization: number of > pivots modified by static pivoting): 0 > INFOG(28) (after factorization: number of null > pivots encountered): 0 > INFOG(29) (after factorization: effective > number of entries in the factors (sum over all processors)): 403041366 > INFOG(30, 31) (after solution: size in Mbytes > of memory used during solution phase): 2467, 4922 > INFOG(32) (after analysis: type of analysis > done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if > determinant is requested): 0 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_u_) > 2 MPI processes > type: mpiaij > rows=184326, cols=184326, bs=3 > total: nonzeros=3.32649e+07, allocated > nonzeros=3.32649e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 26829 > nodes, limit used is 5 > A01 > Mat Object: 2 MPI processes > type: mpiaij > rows=184326, cols=2583, rbs=3, cbs = 1 > total: nonzeros=292770, allocated nonzeros=292770 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 16098 nodes, > limit used is 5 > Mat Object: 2 MPI processes > type: mpiaij > rows=2583, cols=2583, rbs=3, cbs = 1 > total: nonzeros=1.25158e+06, allocated nonzeros=1.25158e+06 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > linear system matrix = precond matrix: > Mat Object: 2 MPI processes > type: mpiaij > rows=186909, cols=186909 > total: nonzeros=3.39678e+07, allocated nonzeros=3.39678e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 26829 nodes, limit used > is 5 > KSPSolve completed > > > Giang > > On Sun, Apr 17, 2016 at 1:15 AM, Matthew Knepley > wrote: > >> On Sat, Apr 16, 2016 at 6:54 PM, Hoang Giang Bui >> wrote: >> >>> Hello >>> >>> I'm solving an indefinite problem arising from mesh tying/contact using >>> Lagrange multiplier, the matrix has the form >>> >>> K = [A P^T >>> P 0] >>> >>> I used the FIELDSPLIT preconditioner with one field is the main variable >>> (displacement) and the other field for dual variable (Lagrange multiplier). >>> The block size for each field is 3. According to the manual, I first chose >>> the preconditioner based on Schur complement to treat this problem. >>> >> >> > >> For any solver question, please send us the output of >> >> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason >> >> > >> However, I will comment below >> >> >>> The parameters used for the solve is >>> -ksp_type gmres >>> >> >> You need 'fgmres' here with the options you have below. >> >> >>> -ksp_max_it 300 >>> -ksp_gmres_restart 300 >>> -ksp_gmres_modifiedgramschmidt >>> -pc_fieldsplit_type schur >>> -pc_fieldsplit_schur_fact_type diag >>> -pc_fieldsplit_schur_precondition selfp >>> >> >> > > >> It could be taking time in the MatMatMult() here if that matrix is dense. >> Is there any reason to >> believe that is a good preconditioner for your problem? >> > > >> >> >>> -pc_fieldsplit_detect_saddle_point >>> -fieldsplit_u_pc_type hypre >>> >> >> I would just use MUMPS here to start, especially if it works on the whole >> problem. Same with the one below. >> >> Matt >> >> >>> -fieldsplit_u_pc_hypre_type boomeramg >>> -fieldsplit_u_pc_hypre_boomeramg_coarsen_type PMIS >>> -fieldsplit_lu_pc_type hypre >>> -fieldsplit_lu_pc_hypre_type boomeramg >>> -fieldsplit_lu_pc_hypre_boomeramg_coarsen_type PMIS >>> >>> For the test case, a small problem is solved on 2 processes. Due to the >>> decomposition, the contact only happens in 1 proc, so the size of Lagrange >>> multiplier dofs on proc 0 is 0. >>> >>> 0: mIndexU.size(): 80490 >>> 0: mIndexLU.size(): 0 >>> 1: mIndexU.size(): 103836 >>> 1: mIndexLU.size(): 2583 >>> >>> However, with this setup the solver takes very long at KSPSolve before >>> going to iteration, and the first iteration seems forever so I have to stop >>> the calculation. I guessed that the solver takes time to compute the Schur >>> complement, but according to the manual only the diagonal of A is used to >>> approximate the Schur complement, so it should not take long to compute >>> this. >>> >>> Note that I ran the same problem with direct solver (MUMPS) and it's >>> able to produce the valid results. The parameter for the solve is pretty >>> standard >>> -ksp_type preonly >>> -pc_type lu >>> -pc_factor_mat_solver_package mumps >>> >>> Hence the matrix/rhs must not have any problem here. Do you have any >>> idea or suggestion for this case? >>> >>> >>> Giang >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Sun Apr 17 22:09:25 2016 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Mon, 18 Apr 2016 11:09:25 +0800 (CST) Subject: [petsc-users] how to compile the program with c++11 support. Message-ID: <7e1875c0.63a2.15427581943.Coremail.ztdepyahoo@163.com> Dear friends: I add the "-std=c++11" in the make file, but it still give me warning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Apr 17 23:12:17 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 17 Apr 2016 23:12:17 -0500 Subject: [petsc-users] how to compile the program with c++11 support. In-Reply-To: <7e1875c0.63a2.15427581943.Coremail.ztdepyahoo@163.com> References: <7e1875c0.63a2.15427581943.Coremail.ztdepyahoo@163.com> Message-ID: you have to be a bit more specific on the issue you are encountering. - are you using PETSc makefiles? - where did you add "-std=c++11" in the makefile? - what is the compile command and its warnings/errors you are getting from the makefile? [after whatever you did] Usually it should work if you add it to CXXFLAGS in makefile - or run 'make CXXFLAGS="-std=c++11" Or you can configure petsc with the option --with-cxx-dialect=C++11 Satish On Sun, 17 Apr 2016, ??? wrote: > Dear friends: > I add the "-std=c++11" in the make file, but it still give me warning. > > From a.croucher at auckland.ac.nz Sun Apr 17 23:31:58 2016 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 18 Apr 2016 16:31:58 +1200 Subject: [petsc-users] block matrix in serial In-Reply-To: References: <55F8CE54.5010707@auckland.ac.nz> Message-ID: <5714633E.8090709@auckland.ac.nz> hi Matt, did you (or anyone else) ever get a chance to look into these two bugs? AFAICT the blocksize discovery code hasn't been changed since then. I'm currently running a problem where the linear solver is struggling (running out of iterations), and I'm wondering if it might perform better if the matrix blocksize were being set correctly. May be nothing to do with it of course, but would at least be useful to eliminate that possibility. Cheers, Adrian //>//>/On 16/09/15 12:40, Matthew Knepley wrote: />/ > On Tue, Sep 15, 2015 at 9:05 PM, Adrian Croucher />/ > >> />/ > wrote: />/ > />/ > hi />/ > />/ > I have a test code (attached) that sets up a finite volume mesh />/ > using DMPlex, with 2 degrees of freedom per cell. />/ > />/ > I then create a matrix using DMCreateMatrix(), having used />/ > DMSetMatType() to set the matrix type to MATBAIJ or MATMPIBAIJ, to />/ > take advantage of the block structure. />/ > />/ > This works OK and gives me the expected matrix structure when I />/ > run on > 1 processor, but neither MATBAIJ or MATMPIBAIJ works if I />/ > run it in serial: />/ > />/ > />/ > Plex should automatically discover the block size from the Section. />/ > If not, it uses block size 1. I have to look at the example to see />/ > why the discovery is not working. Do you have any constraints? />//>/It looks like block discovery in parallel effectively always />/determines a block size of 1. Running with -mat_view ::ascii_info gives: />//>/Mat Object: 2 MPI processes />/type: mpibaij />/rows=20, cols=20 />/total: nonzeros=112, allocated nonzeros=112 />/total number of mallocs used during MatSetValues calls =0 />/block size is 1 />/^^^ />//>/The block size discovery does this: />//>/for (p = pStart; p < pEnd; ++p) { />/ierr = PetscSectionGetDof(sectionGlobal, p, &dof);CHKERRQ(ierr); />/ierr = PetscSectionGetConstraintDof(sectionGlobal, p, />/&cdof);CHKERRQ(ierr); />/if (dof-cdof) { />/if (bs < 0) { />/bs = dof-cdof; />/} else if (bs != dof-cdof) { />//* Layout does not admit a pointwise block size */ />/bs = 1; />/break; />/} />/} />/} />//>/In parallel, some of the dofs are remote (encoded as negative). So we />/run through seeing (dof - cdof) == 2, until we hit a "remote" point at />/when we see (dof - cdof) != 2 and then we break out and set bs = 1. />//>/In serial, we correctly determine bs == 2. But then />/DMGetLocalToGlobalMapping always does />//>/ierr = ISLocalToGlobalMappingCreate(PETSC_COMM_SELF, 1,size, />/ltog, PETSC_OWN_POINTER, &dm->ltogmap);CHKERRQ(ierr); />//>//>/i.e. is set with block size == 1. />//>/So there are two bugs here. />//>/1. In parallel, block size discovery in Mat creation has gone wrong />// Crap. Hoist on my own petard. Okay I will fix this. >/2. (Always), the lgmap has block size of 1, irrespective of the />/discovered block size from the section. />// Yep. This can also be fixed. It should work regardless, but would be better with blocking. Thanks Matt -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 84611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Mon Apr 18 06:02:25 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Mon, 18 Apr 2016 13:02:25 +0200 Subject: [petsc-users] Preallocation: Allocate more than the local column Message-ID: <4823217.bSuSqO28FM@asaru> Hello, I create a MATAIJ on 4 processors using local sizes (0) Set matrix A to local size 2 x 5 (1) Set matrix A to local size 2 x 2 (2) Set matrix A to local size 2 x 2 (3) Set matrix A to local size 2 x 2 which results in a global size of 8 x 11. Afaik the local size given is not relevant for the parallel layout, so each processors holds 2 rows x 11 cols. Accordingly I can set these using MatSetValue and alike But it seems to be important when I do preallocation, i.e. work on the local submatrices: (0) Local Submatrix Rows = 0 / 2, Local Submatrix Cols = 0 / 5 (1) Local Submatrix Rows = 2 / 4, Local Submatrix Cols = 5 / 7 (2) Local Submatrix Rows = 4 / 6, Local Submatrix Cols = 7 / 9 (3) Local Submatrix Rows = 6 / 8, Local Submatrix Cols = 9 / 11 (retrieved with MatGetOwnershipRange / MatGetOwnershipRangeColumn) therefore, if I do a preallocation using MatMPIAIJSetPreallocation I get errors like [1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home/florian/software/petsc/src/mat/impls/aij/seq/aij.c nnz cannot be greater than row length: local row 0 value 4 rowlength 2 Easiest way to work around that would be to set the number of local cols (here always 5/2/2/2) to 11. Setting the number of global columns to 11 would probably have the same effect. Problem is that each processor only knows the number of local columns, so I would need some communication for that. Another possibility that comes to my mind is to declare the Mat only after the preallocation has been determined and during preallocation the longest line has been computed. Are my assumptions correct so far? How would you cope with that? Best Regards, Florian From knepley at gmail.com Mon Apr 18 07:26:28 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 18 Apr 2016 08:26:28 -0400 Subject: [petsc-users] Preallocation: Allocate more than the local column In-Reply-To: <4823217.bSuSqO28FM@asaru> References: <4823217.bSuSqO28FM@asaru> Message-ID: On Mon, Apr 18, 2016 at 7:02 AM, Florian Lindner wrote: > Hello, > > I create a MATAIJ on 4 processors using local sizes > > (0) Set matrix A to local size 2 x 5 > (1) Set matrix A to local size 2 x 2 > (2) Set matrix A to local size 2 x 2 > (3) Set matrix A to local size 2 x 2 > > which results in a global size of 8 x 11. > > Afaik the local size given is not relevant for the parallel layout, so > each processors holds 2 rows x 11 cols. Accordingly I can set these using > MatSetValue and alike > > But it seems to be important when I do preallocation, i.e. work on the > local submatrices: > > (0) Local Submatrix Rows = 0 / 2, Local Submatrix Cols = 0 / 5 > (1) Local Submatrix Rows = 2 / 4, Local Submatrix Cols = 5 / 7 > (2) Local Submatrix Rows = 4 / 6, Local Submatrix Cols = 7 / 9 > (3) Local Submatrix Rows = 6 / 8, Local Submatrix Cols = 9 / 11 > > (retrieved with MatGetOwnershipRange / MatGetOwnershipRangeColumn) > > therefore, if I do a preallocation using MatMPIAIJSetPreallocation I get > errors like > > [1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in > /home/florian/software/petsc/src/mat/impls/aij/seq/aij.c > nnz cannot be greater than row length: local row 0 value 4 rowlength 2 > > Easiest way to work around that would be to set the number of local cols > (here always 5/2/2/2) to 11. Setting the number of global columns to 11 > would probably have the same effect. > So you want to preallocate a rectangular matrix? The error above is correct because the diagonal blocks on proc 1 is 2x2. You have to put the other nonzeros in the off-diagonal portion. Matt > Problem is that each processor only knows the number of local columns, so > I would need some communication for that. > > Another possibility that comes to my mind is to declare the Mat only after > the preallocation has been determined and during preallocation the longest > line has been computed. > > Are my assumptions correct so far? > How would you cope with that? > > Best Regards, > Florian > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgbk2008 at gmail.com Mon Apr 18 08:34:26 2016 From: hgbk2008 at gmail.com (Hoang Giang Bui) Date: Mon, 18 Apr 2016 15:34:26 +0200 Subject: [petsc-users] snes set function and Jacobian concurrently Message-ID: Hello Is it possible to combine the SNESSetFunction and SNESSetJacobian into one call, i.e. SNESSetFunctionAndJacobian which takes unique function pointer to compute rhs and Jacobian concurrently? The reason for it is that the finite element code I'm working on contribute to global rhs and stiffness at one call, so to cater for SNES would break the existing interface. Another supporting point for that feature is that for irreversible problem, the rhs (equilirium) and Jacobian shall not be decoupled, since they are not able to return to the past state after proceed. Giang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 18 08:40:28 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 18 Apr 2016 09:40:28 -0400 Subject: [petsc-users] snes set function and Jacobian concurrently In-Reply-To: References: Message-ID: On Mon, Apr 18, 2016 at 9:34 AM, Hoang Giang Bui wrote: > Hello > > Is it possible to combine the SNESSetFunction and SNESSetJacobian into one > call, i.e. SNESSetFunctionAndJacobian which takes unique function pointer > to compute rhs and Jacobian concurrently? The reason for it is that the > finite element code I'm working on contribute to global rhs and stiffness > at one call, so to cater for SNES would break the existing interface. > The Jacobian and Residual are not always formed together, so you would have to stash your computation in a user context and then check that the same X had been passed to both functions. For example, the Residual can be called many times in a line search routine. Matt > Another supporting point for that feature is that for irreversible > problem, the rhs (equilirium) and Jacobian shall not be decoupled, since > they are not able to return to the past state after proceed. > > Giang > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From natacha.bereux at gmail.com Mon Apr 18 10:19:07 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Mon, 18 Apr 2016 17:19:07 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Hello Satish, thank you very much for yor advices. They were very helpful ! The configure step finally succeeds if I use the following configure line: ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-metis-mumps --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi -lblacsF77init-openmpi -lblacsCinit-openmpi" --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib -lmetis -lGKlib" --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib -lzmumps -ldmumps -lmumps_common -lpord -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib -lesmumps -lscotch -lscotcherr -lscotcherrexit -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib -lmetis" --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include --with-blas-lapack-lib="-llapack -lopenblas" --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz LIBS=-lgomp I have to specify scotch shared libraries (-lscotch -lscotcherr -lscotcherrexit ) and metis shared library in --with-mumps-lib option otherwise the test (on dmump) in configure fails. Is it OK to do so ? I also tried to use the following shorter line ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi -lblacsF77init-openmpi -lblacsCinit-openmpi" --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib -lzmumps -ldmumps -lmumps_common -lpord -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib -lesmumps -lscotch -lscotcherr -lscotcherrexit -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib -lmetis" --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include --with-blas-lapack-lib="-llapack -lopenblas" --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz LIBS=-lgomp I do not use --with-metis-lib/--with-metis-include. I wonder if this is authorized : the libraries are given with the --with-mumps-lib, but the include are not defined. What is the good practice ? Best regards Natacha On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay wrote: > you'll have to roll-in the --with-blacs-lib option into > --with-scalapack-lib option > > Satish > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > Sorry, do not take into account my last email. > > I made some progress and I am now able to configure PETSc with a > > pre-installed version of metis. > > > > Problems come when I try to configure PETSc with MUMPS > > > > My command line is > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > --PETSC_ARCH=linux-metis-mumps > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > --with-metis-include=$METIS_PRE/include > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > --with-mumps-include=$MUMPS_PRE/include > > > > where METIS_PRE and MUMPS_PRE are the path to the local installs of metis > > and mumps) > > > > I get (at least) the following error > > /libdmumps.a(dend_driver.o): undefined reference to symbol > 'blacs_gridexit_' > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing from > > command line > > collect2: error: ld returned 1 exit status > > > > > > Would you have any idea of its meaning ? > > > > The configure.log is attached > > Thanks a lot if you can help me ! > > Natacha > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > natacha.bereux at gmail.com> > > wrote: > > > > > Hi Satish > > > thanks a lot for the answer. Unfortunately, it does not work yet. > > > More precisely : > > > --download-mumps works fine (and every --download-package option works > > > perfectly). I am then able to compile a PETSc library. > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > > > --with-package-lib=libXXXX.so also works > > > > > > But I would like to use static librairies, preinstalled on my computer > > > ... and this fails. > > > > > > For the moment I gave up compiling with MUMPS, and I am instead > trying to > > > compile with Metis 5. > > > I have a preinstalled version in a some directory lets say metis_dir > > > I try > > > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > > > --with-metis-include=metis_dir/include > > > this fails (see the attached config.log) > > > -with-metis-dir=metis_dir also fails > > > Is there a problem with static librairies ? > > > > > > Natacha > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay > wrote: > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > >> > > >> > Hello, > > >> > I am trying to compile Petsc (3.6.3) with external packages (MUMPS > and > > >> its > > >> > prerequisites). > > >> > More precisely I would like PETSc to use a pre-installed version of > > >> MUMPS. > > >> > > > >> > Petsc downloads and compiles the prerequisites (parmetis, scalapack > > >> etc) : > > >> > this works fine. > > >> > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > >> > > >> What version of MUMPS did you install? > > >> > > >> Why could you not use --download-mumps? > > >> > > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > > >> then specifying --download-metis --download-parmetis > > >> --download-scalapack [i.e different versions/builds of the same > > >> libraries] can result in conflcits. > > >> > > >> > > > >> > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > > >> install, > > >> > but the > > >> > configure step fails with the followiing message: > > >> > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > >> details): > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > >> > > > >> > I do not understand what is wrong. > > >> > I have attached the configure.log file. > > >> > > > >> > Any hint would be greatly appreciated ! > > >> > > >> >>>> > > >> Executing: mpicc -show > > >> stdout: gcc -I/usr/lib/openmpi/include > -I/usr/lib/openmpi/include/openmpi > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > >> Defined make macro "MPICC_SHOW" to "gcc > > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > >> <<<< > > >> Ok - so you are using system openmpi with gcc. > > >> > > >> > > >> >>>> > > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > -fPIC > > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > -lzmumps > > >> -lmumps_common -lpord > > >> > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lscalapack > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > >> -lgfortran -lm -lquadmath -lm > > >> > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lparmetis > > >> > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > -lmetis -lm > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > >> Possible ERROR while running linker: exit code 256 > > >> stderr: > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > >> `GOMP_critical_name_end' > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > >> `GOMP_critical_name_end' > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > >> `GOMP_critical_name_end' > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > >> `GOMP_critical_name_end' > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > >> `GOMP_critical_name_start' > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > >> `GOMP_critical_name_end' > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > function > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > >> dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > > >> dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > > >> dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > > >> dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > > >> dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > > >> > > >> > > >> > > >> Looks like this MUMPS is built with ptscotch and openmp. > > >> > > >> You can specify -lgomp for openmp. This can be done with the configure > > >> option > > >> LIBS=-lgomp > > >> > > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > > >> figureout > > >> how to specify it. > > >> > > >> Its best to specify all the packages [mumps and its dependencies] you > > >> built manually with the options: > > >> > > >> --with-package-include --with-package-lib > > >> > > >> Satish > > >> > > >> > Best regards, > > >> > > > >> > Natacha > > >> > > > >> > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2286819 bytes Desc: not available URL: From balay at mcs.anl.gov Mon Apr 18 10:52:06 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 18 Apr 2016 10:52:06 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Same with parmetis. On Mon, 18 Apr 2016, Natacha BEREUX wrote: > Hello Satish, > thank you very much for yor advices. They were very helpful ! > > The configure step finally succeeds if I use the following configure line: > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpi=1 > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > -lblacsF77init-openmpi -lblacsCinit-openmpi" > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > -lmetis -lGKlib" > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > -lzmumps -ldmumps -lmumps_common -lpord > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > -lesmumps -lscotch -lscotcherr -lscotcherrexit > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > -lmetis" > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > --with-blas-lapack-lib="-llapack -lopenblas" > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz LIBS=-lgomp > > I have to specify scotch shared libraries (-lscotch -lscotcherr > -lscotcherrexit ) and metis shared library in --with-mumps-lib option > otherwise the test (on dmump) in configure fails. > Is it OK to do so ? Hm - its best to avoid duplication of libraries. i.e specifying scotch via mumps-libs and also via --download-ptscotch will cause problems. Why not specify scotch with --with-ptscotch-include,--with-ptscotch-lib options? > > I also tried to use the following shorter line > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > -lblacsF77init-openmpi -lblacsCinit-openmpi" > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > -lzmumps -ldmumps -lmumps_common -lpord > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > -lesmumps -lscotch -lscotcherr -lscotcherrexit > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > -lmetis" > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > --with-blas-lapack-lib="-llapack -lopenblas" > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz LIBS=-lgomp > > I do not use --with-metis-lib/--with-metis-include. > I wonder if this is authorized : the libraries are given with the > --with-mumps-lib, but the include are not defined. This is fine. [except for the dupliation of ptscotch] > > What is the good practice ? Either is work - but its best to specify each package with its own options listed by configure. The issues usually are: - Is this package primarily a depencency of an externalpakage or if there is a petsc interface to it? For ex: PETSc has interface to mumps, parmetis - but not scalapack. So if you club parmetis into mumps-libs then petsc interface to parmetis would not be enabled. If you are not using this feature - it doesn't matter if its not enabled. - does this externalpakage also require the includes in the public interface? If dmumps_c.h requires metis.h [when mumps is built with metis] - then you might have to specify metis include also with --with-mumps-include. Otherwise - it doesn't matter. - are there bugs in petsc configure that can trigger wrong error checks? Because mumps depends on scalapack, and optionally on metis,parmetis,ptscotch [i.e only one of them is required - but not all] - there is an error-check in configure to make sure atleast one of them is specified for --download-mumps. Perhaps this check should not trigger error for user built mumps. Did you build MUMPS with both metis, ptscotch? [and not parmetis?] Satish > > Best regards > Natacha > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay wrote: > > > you'll have to roll-in the --with-blacs-lib option into > > --with-scalapack-lib option > > > > Satish > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > Sorry, do not take into account my last email. > > > I made some progress and I am now able to configure PETSc with a > > > pre-installed version of metis. > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > My command line is > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > --PETSC_ARCH=linux-metis-mumps > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > --with-metis-include=$METIS_PRE/include > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local installs of metis > > > and mumps) > > > > > > I get (at least) the following error > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > 'blacs_gridexit_' > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing from > > > command line > > > collect2: error: ld returned 1 exit status > > > > > > > > > Would you have any idea of its meaning ? > > > > > > The configure.log is attached > > > Thanks a lot if you can help me ! > > > Natacha > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > natacha.bereux at gmail.com> > > > wrote: > > > > > > > Hi Satish > > > > thanks a lot for the answer. Unfortunately, it does not work yet. > > > > More precisely : > > > > --download-mumps works fine (and every --download-package option works > > > > perfectly). I am then able to compile a PETSc library. > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more generally > > > > --with-package-lib=libXXXX.so also works > > > > > > > > But I would like to use static librairies, preinstalled on my computer > > > > ... and this fails. > > > > > > > > For the moment I gave up compiling with MUMPS, and I am instead > > trying to > > > > compile with Metis 5. > > > > I have a preinstalled version in a some directory lets say metis_dir > > > > I try > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, metis_dir/lib/libGKlib.a] > > > > --with-metis-include=metis_dir/include > > > > this fails (see the attached config.log) > > > > -with-metis-dir=metis_dir also fails > > > > Is there a problem with static librairies ? > > > > > > > > Natacha > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay > > wrote: > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > >> > > > >> > Hello, > > > >> > I am trying to compile Petsc (3.6.3) with external packages (MUMPS > > and > > > >> its > > > >> > prerequisites). > > > >> > More precisely I would like PETSc to use a pre-installed version of > > > >> MUMPS. > > > >> > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, scalapack > > > >> etc) : > > > >> > this works fine. > > > >> > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > >> > > > >> What version of MUMPS did you install? > > > >> > > > >> Why could you not use --download-mumps? > > > >> > > > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > > > >> then specifying --download-metis --download-parmetis > > > >> --download-scalapack [i.e different versions/builds of the same > > > >> libraries] can result in conflcits. > > > >> > > > >> > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory of MUMPS > > > >> install, > > > >> > but the > > > >> > configure step fails with the followiing message: > > > >> > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > >> details): > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > >> > > > > >> > I do not understand what is wrong. > > > >> > I have attached the configure.log file. > > > >> > > > > >> > Any hint would be greatly appreciated ! > > > >> > > > >> >>>> > > > >> Executing: mpicc -show > > > >> stdout: gcc -I/usr/lib/openmpi/include > > -I/usr/lib/openmpi/include/openmpi > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > >> <<<< > > > >> Ok - so you are using system openmpi with gcc. > > > >> > > > >> > > > >> >>>> > > > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > > -fPIC > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > > -lzmumps > > > >> -lmumps_common -lpord > > > >> > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -lscalapack > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > >> -lgfortran -lm -lquadmath -lm > > > >> > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -lparmetis > > > >> > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > -lmetis -lm > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > > >> Possible ERROR while running linker: exit code 256 > > > >> stderr: > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > >> `GOMP_critical_name_start' > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > >> `GOMP_critical_name_end' > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > function > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > >> dana_lr.F:(.text+0x115): undefined reference to `scotchfgraphbuild_' > > > >> dana_lr.F:(.text+0x131): undefined reference to `scotchfstratinit_' > > > >> dana_lr.F:(.text+0x151): undefined reference to `scotchfgraphpart_' > > > >> dana_lr.F:(.text+0x15e): undefined reference to `scotchfstratexit_' > > > >> dana_lr.F:(.text+0x16b): undefined reference to `scotchfgraphexit_' > > > >> > > > >> > > > >> > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > >> > > > >> You can specify -lgomp for openmp. This can be done with the configure > > > >> option > > > >> LIBS=-lgomp > > > >> > > > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll have to > > > >> figureout > > > >> how to specify it. > > > >> > > > >> Its best to specify all the packages [mumps and its dependencies] you > > > >> built manually with the options: > > > >> > > > >> --with-package-include --with-package-lib > > > >> > > > >> Satish > > > >> > > > >> > Best regards, > > > >> > > > > >> > Natacha > > > >> > > > > >> > > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Mon Apr 18 13:41:29 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Apr 2016 13:41:29 -0500 Subject: [petsc-users] Preallocation: Allocate more than the local column In-Reply-To: <4823217.bSuSqO28FM@asaru> References: <4823217.bSuSqO28FM@asaru> Message-ID: <329D3F4D-15F2-4F99-9C23-C2D848E0798A@mcs.anl.gov> Florian, I don't really understand your question so I will answer a question that I do understand that may or not be related to your question. To do a proper preallocation each process needs to know the rows of the matrix that it will own plus which columns are in the "diagonal portion" of the matrix. Usually one figures this information out based on the decomposition of the "mesh" that they have decided on. One cannot have the matrix decide on the decomposition of the rows/columns while also trying to provide the preallocation information since the preallocation information depends exactly on the decomposition of the rows/columns. Thus you need to determine how you want to decompose the matrix YOURSELF before you create the matrix and provide the preallocation information. Barry > On Apr 18, 2016, at 6:02 AM, Florian Lindner wrote: > > Hello, > > I create a MATAIJ on 4 processors using local sizes > > (0) Set matrix A to local size 2 x 5 > (1) Set matrix A to local size 2 x 2 > (2) Set matrix A to local size 2 x 2 > (3) Set matrix A to local size 2 x 2 > > which results in a global size of 8 x 11. > > Afaik the local size given is not relevant for the parallel layout, so each processors holds 2 rows x 11 cols. Accordingly I can set these using MatSetValue and alike > > But it seems to be important when I do preallocation, i.e. work on the local submatrices: > > (0) Local Submatrix Rows = 0 / 2, Local Submatrix Cols = 0 / 5 > (1) Local Submatrix Rows = 2 / 4, Local Submatrix Cols = 5 / 7 > (2) Local Submatrix Rows = 4 / 6, Local Submatrix Cols = 7 / 9 > (3) Local Submatrix Rows = 6 / 8, Local Submatrix Cols = 9 / 11 > > (retrieved with MatGetOwnershipRange / MatGetOwnershipRangeColumn) > > therefore, if I do a preallocation using MatMPIAIJSetPreallocation I get errors like > > [1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home/florian/software/petsc/src/mat/impls/aij/seq/aij.c > nnz cannot be greater than row length: local row 0 value 4 rowlength 2 > > Easiest way to work around that would be to set the number of local cols (here always 5/2/2/2) to 11. Setting the number of global columns to 11 would probably have the same effect. > > Problem is that each processor only knows the number of local columns, so I would need some communication for that. > > Another possibility that comes to my mind is to declare the Mat only after the preallocation has been determined and during preallocation the longest line has been computed. > > Are my assumptions correct so far? > How would you cope with that? > > Best Regards, > Florian > > From bsmith at mcs.anl.gov Mon Apr 18 13:58:24 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Apr 2016 13:58:24 -0500 Subject: [petsc-users] snes set function and Jacobian concurrently In-Reply-To: References: Message-ID: <98A2ECBF-6B53-47A1-BD2F-066A5D022600@mcs.anl.gov> http://www.mcs.anl.gov/petsc/documentation/faq.html#functionjacobian > On Apr 18, 2016, at 8:34 AM, Hoang Giang Bui wrote: > > Hello > > Is it possible to combine the SNESSetFunction and SNESSetJacobian into one call, i.e. SNESSetFunctionAndJacobian which takes unique function pointer to compute rhs and Jacobian concurrently? The reason for it is that the finite element code I'm working on contribute to global rhs and stiffness at one call, so to cater for SNES would break the existing interface. > > Another supporting point for that feature is that for irreversible problem, the rhs (equilirium) and Jacobian shall not be decoupled, since they are not able to return to the past state after proceed. > > Giang > From mrestelli at gmail.com Tue Apr 19 03:26:12 2016 From: mrestelli at gmail.com (marco restelli) Date: Tue, 19 Apr 2016 10:26:12 +0200 Subject: [petsc-users] Deactivate PETSc error handling Message-ID: Dear all, running a code linked against PETSc I get the following error: PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero This message from PETSc is suppressing a much more useful error message from the compiler, which typically indicates the source line where the floating point exception happened. Is there a way to switch off any error handling from PETSc? Thank you, Marco From richardtmills at gmail.com Tue Apr 19 03:58:13 2016 From: richardtmills at gmail.com (Richard Mills) Date: Tue, 19 Apr 2016 01:58:13 -0700 Subject: [petsc-users] Deactivate PETSc error handling In-Reply-To: References: Message-ID: Hi Marco, Try running with "-no_signal_handler" on the command line. Cheers, Richard On Tue, Apr 19, 2016 at 1:26 AM, marco restelli wrote: > Dear all, > running a code linked against PETSc I get the following error: > > PETSC ERROR: Caught signal number 8 FPE: Floating Point > Exception,probably divide by zero > > This message from PETSc is suppressing a much more useful error > message from the compiler, which typically indicates the source line > where the floating point exception happened. Is there a way to switch > off any error handling from PETSc? > > Thank you, > Marco > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrestelli at gmail.com Tue Apr 19 04:33:57 2016 From: mrestelli at gmail.com (marco restelli) Date: Tue, 19 Apr 2016 11:33:57 +0200 Subject: [petsc-users] Deactivate PETSc error handling In-Reply-To: References: Message-ID: 2016-04-19 10:58 GMT+0200, Richard Mills : > Hi Marco, > > Try running with "-no_signal_handler" on the command line. Perfect, thanks! Marco From ztdepyahoo at 163.com Tue Apr 19 05:51:30 2016 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 19 Apr 2016 18:51:30 +0800 (CST) Subject: [petsc-users] can i use MatCreatDense to creat a parallel matrix for KSP Message-ID: <35bc2691.106c1.1542e258215.Coremail.ztdepyahoo@163.com> Dear friends: I am receiving error about " not for unassembled matrix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From natacha.bereux at gmail.com Tue Apr 19 07:17:47 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Tue, 19 Apr 2016 14:17:47 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Hello, Thanks a lot for your explanations. We have a "home-made" install of MUMPS, built with metis and scotch (and - at the moment- neither with parmetis nor with ptscotch). When I use --with-mumps-lib/--with-mumps-include, I am forced to specify scotch and metis libs inside mumps libs (otherwise the link of the check program dmumps_c fails). And this occurs even if I have already defined the location of scotch libraries through --with-scotch-lib/--with-scotch-include and --with-metis-lib/--with-metis-include options. Defining PETSc interfaces to scotch and metis packages is (at least for me) not sufficient : I have to specify these libraries locations in --with-mumps-lib. (as shown in the configure.log). This is not really a problem and doing so fixes the configure step. I agree that duplicating scotch to define ptscotch is weird. I did so because we do not have a parallel built of scotch. >From PETSc point of view, MUMPS requires either Parmetis or PTScotch : so I provided ptscotch through --download-ptscotch option. I do not intend to use it, that is only for configuring purpose ! Natacha On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay wrote: > Same with parmetis. > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > Hello Satish, > > thank you very much for yor advices. They were very helpful ! > > > > The configure step finally succeeds if I use the following configure > line: > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-mpi=1 > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > -lmetis -lGKlib" > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > -lzmumps -ldmumps -lmumps_common -lpord > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > -lmetis" > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > --with-blas-lapack-lib="-llapack -lopenblas" > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > LIBS=-lgomp > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > -lscotcherrexit ) and metis shared library in --with-mumps-lib option > > otherwise the test (on dmump) in configure fails. > > Is it OK to do so ? > > Hm - its best to avoid duplication of libraries. > > i.e specifying scotch via mumps-libs and also via --download-ptscotch will > cause problems. > > Why not specify scotch with --with-ptscotch-include,--with-ptscotch-lib > options? > > > > > I also tried to use the following shorter line > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > -lzmumps -ldmumps -lmumps_common -lpord > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > -lmetis" > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > --with-blas-lapack-lib="-llapack -lopenblas" > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > LIBS=-lgomp > > > > I do not use --with-metis-lib/--with-metis-include. > > I wonder if this is authorized : the libraries are given with the > > --with-mumps-lib, but the include are not defined. > > This is fine. [except for the dupliation of ptscotch] > > > > > What is the good practice ? > > Either is work - but its best to specify each package with its own > options listed by configure. The issues usually are: > > - Is this package primarily a depencency of an externalpakage or if there > is a petsc interface to it? > > For ex: PETSc has interface to mumps, parmetis - but not scalapack. So > if you club parmetis into mumps-libs then petsc interface to parmetis > would not be enabled. If you are not using this feature - it doesn't > matter if its not enabled. > > - does this externalpakage also require the includes in the public > interface? > > If dmumps_c.h requires metis.h [when mumps is built with metis] > - then you might have to specify metis include also with > --with-mumps-include. Otherwise - it doesn't matter. > > - are there bugs in petsc configure that can trigger wrong error checks? > > Because mumps depends on scalapack, and optionally on > metis,parmetis,ptscotch [i.e only one of them is required - but not > all] - there is an error-check in configure to make sure atleast one > of them is specified for --download-mumps. Perhaps this check should > not trigger error for user built mumps. > > Did you build MUMPS with both metis, ptscotch? [and not parmetis?] > > Satish > > > > > Best regards > > Natacha > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay wrote: > > > > > you'll have to roll-in the --with-blacs-lib option into > > > --with-scalapack-lib option > > > > > > Satish > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > Sorry, do not take into account my last email. > > > > I made some progress and I am now able to configure PETSc with a > > > > pre-installed version of metis. > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > My command line is > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > --PETSC_ARCH=linux-metis-mumps > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local installs of > metis > > > > and mumps) > > > > > > > > I get (at least) the following error > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > 'blacs_gridexit_' > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing > from > > > > command line > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > The configure.log is attached > > > > Thanks a lot if you can help me ! > > > > Natacha > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > natacha.bereux at gmail.com> > > > > wrote: > > > > > > > > > Hi Satish > > > > > thanks a lot for the answer. Unfortunately, it does not work yet. > > > > > More precisely : > > > > > --download-mumps works fine (and every --download-package option > works > > > > > perfectly). I am then able to compile a PETSc library. > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more > generally > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > But I would like to use static librairies, preinstalled on my > computer > > > > > ... and this fails. > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am instead > > > trying to > > > > > compile with Metis 5. > > > > > I have a preinstalled version in a some directory lets say > metis_dir > > > > > I try > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > metis_dir/lib/libGKlib.a] > > > > > --with-metis-include=metis_dir/include > > > > > this fails (see the attached config.log) > > > > > -with-metis-dir=metis_dir also fails > > > > > Is there a problem with static librairies ? > > > > > > > > > > Natacha > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay > > > wrote: > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > >> > > > > >> > Hello, > > > > >> > I am trying to compile Petsc (3.6.3) with external packages > (MUMPS > > > and > > > > >> its > > > > >> > prerequisites). > > > > >> > More precisely I would like PETSc to use a pre-installed > version of > > > > >> MUMPS. > > > > >> > > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, > scalapack > > > > >> etc) : > > > > >> > this works fine. > > > > >> > > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > > >> > > > > >> What version of MUMPS did you install? > > > > >> > > > > >> Why could you not use --download-mumps? > > > > >> > > > > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > > > > >> then specifying --download-metis --download-parmetis > > > > >> --download-scalapack [i.e different versions/builds of the same > > > > >> libraries] can result in conflcits. > > > > >> > > > > >> > > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory of > MUMPS > > > > >> install, > > > > >> > but the > > > > >> > configure step fails with the followiing message: > > > > >> > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > > >> details): > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > >> > > > > > >> > I do not understand what is wrong. > > > > >> > I have attached the configure.log file. > > > > >> > > > > > >> > Any hint would be greatly appreciated ! > > > > >> > > > > >> >>>> > > > > >> Executing: mpicc -show > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > -I/usr/lib/openmpi/include/openmpi > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > -pthread > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > >> <<<< > > > > >> Ok - so you are using system openmpi with gcc. > > > > >> > > > > >> > > > > >> >>>> > > > > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > > > -fPIC > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > > > -lzmumps > > > > >> -lmumps_common -lpord > > > > >> > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -lscalapack > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > -L/usr/lib/openmpi/lib > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > >> -lgfortran -lm -lquadmath -lm > > > > >> > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -lparmetis > > > > >> > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > -lmetis -lm > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread > -ldl > > > > >> Possible ERROR while running linker: exit code 256 > > > > >> stderr: > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > >> `GOMP_critical_name_start' > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > >> `GOMP_critical_name_end' > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > function > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > >> dana_lr.F:(.text+0x115): undefined reference to > `scotchfgraphbuild_' > > > > >> dana_lr.F:(.text+0x131): undefined reference to > `scotchfstratinit_' > > > > >> dana_lr.F:(.text+0x151): undefined reference to > `scotchfgraphpart_' > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > `scotchfstratexit_' > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > `scotchfgraphexit_' > > > > >> > > > > >> > > > > >> > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > >> > > > > >> You can specify -lgomp for openmp. This can be done with the > configure > > > > >> option > > > > >> LIBS=-lgomp > > > > >> > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll > have to > > > > >> figureout > > > > >> how to specify it. > > > > >> > > > > >> Its best to specify all the packages [mumps and its dependencies] > you > > > > >> built manually with the options: > > > > >> > > > > >> --with-package-include --with-package-lib > > > > >> > > > > >> Satish > > > > >> > > > > >> > Best regards, > > > > >> > > > > > >> > Natacha > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1928950 bytes Desc: not available URL: From ztdepyahoo at 163.com Tue Apr 19 07:20:58 2016 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 19 Apr 2016 20:20:58 +0800 (CST) Subject: [petsc-users] Which command gather the distributed vector to the root rank Message-ID: <3ef040b5.111dd.1542e776a67.Coremail.ztdepyahoo@163.com> Dear friends: I Want to gather the vector to the root rank. which command perform this task. i only find that with viewer, we can write it to the root rank. Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 19 07:31:16 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Apr 2016 07:31:16 -0500 Subject: [petsc-users] Which command gather the distributed vector to the root rank In-Reply-To: <3ef040b5.111dd.1542e776a67.Coremail.ztdepyahoo@163.com> References: <3ef040b5.111dd.1542e776a67.Coremail.ztdepyahoo@163.com> Message-ID: On Tue, Apr 19, 2016 at 7:20 AM, ??? wrote: > Dear friends: > I Want to gather the vector to the root rank. which command perform > this task. i only find that with viewer, we can write it to the root rank. > http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-to-mpi-vec Matt > Regards > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 19 07:34:06 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Apr 2016 07:34:06 -0500 Subject: [petsc-users] can i use MatCreatDense to creat a parallel matrix for KSP In-Reply-To: <35bc2691.106c1.1542e258215.Coremail.ztdepyahoo@163.com> References: <35bc2691.106c1.1542e258215.Coremail.ztdepyahoo@163.com> Message-ID: Did you call MatAssemblyBegin/End()? Matt On Tue, Apr 19, 2016 at 5:51 AM, ??? wrote: > Dear friends: > I am receiving error about " not for unassembled matrix" > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Apr 19 09:33:51 2016 From: hzhang at mcs.anl.gov (Hong) Date: Tue, 19 Apr 2016 09:33:51 -0500 Subject: [petsc-users] Which command gather the distributed vector to the root rank In-Reply-To: <3ef040b5.111dd.1542e776a67.Coremail.ztdepyahoo@163.com> References: <3ef040b5.111dd.1542e776a67.Coremail.ztdepyahoo@163.com> Message-ID: ???: > Dear friends: > I Want to gather the vector to the root rank. which command perform > this task. i only find that with viewer, we can write it to the root rank. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html Hong > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 19 11:26:36 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 19 Apr 2016 11:26:36 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Natacha, Are you using mumps serially or parallely? My understanding is - MUMPS - when used parallely requires either parmetis or ptscotch. And when used sequentially - it does not require anything [i.e metis/scotch might be optional] Currently PETSc configure is trying to detect/use mumps parallely - hence its insisting on parmetis or ptscotch. Assuming ptscotch doesn't conflict with scotch [I don't know if ptscotch sources also includes scotch scources or not] - its fine to use --download-ptscotch as a workarround. [or --download-parmetis - assuming it can build with your install of metis] Ideally you should not need a workarround - hence my asking for specifics of usage - to see if our configure check for mumps needs fixing. Wrt specifying scotch or metis - PETSc configure supports metis - but not scotch. So --with-metis-lib/--with-metis-include should work - but not --with-scotch-lib/--with-scotch-include But its fine to lump both into mumps options - as they are primarily dependencies of mumps anyway. Satish On Tue, 19 Apr 2016, Natacha BEREUX wrote: > Hello, > Thanks a lot for your explanations. > We have a "home-made" install of MUMPS, built with metis and scotch (and - > at the moment- neither with parmetis nor with ptscotch). > > When I use --with-mumps-lib/--with-mumps-include, I am forced to specify > scotch and metis libs inside mumps libs (otherwise the link of the check > program dmumps_c fails). And this occurs even if I have already defined the > location of scotch libraries through > --with-scotch-lib/--with-scotch-include and > --with-metis-lib/--with-metis-include options. > Defining PETSc interfaces to scotch and metis packages is (at least for > me) not sufficient : I have to specify these libraries locations in > --with-mumps-lib. (as shown in the configure.log). > > This is not really a problem and doing so fixes the configure step. > > I agree that duplicating scotch to define ptscotch is weird. I did so > because we do not have a parallel built of scotch. > From PETSc point of view, MUMPS requires either Parmetis or PTScotch : so I > provided ptscotch through --download-ptscotch option. > I do not intend to use it, that is only for configuring purpose ! > > Natacha > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay wrote: > > > Same with parmetis. > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > Hello Satish, > > > thank you very much for yor advices. They were very helpful ! > > > > > > The configure step finally succeeds if I use the following configure > > line: > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > --with-mpi=1 > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > -lmetis -lGKlib" > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > -lmetis" > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > LIBS=-lgomp > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib option > > > otherwise the test (on dmump) in configure fails. > > > Is it OK to do so ? > > > > Hm - its best to avoid duplication of libraries. > > > > i.e specifying scotch via mumps-libs and also via --download-ptscotch will > > cause problems. > > > > Why not specify scotch with --with-ptscotch-include,--with-ptscotch-lib > > options? > > > > > > > > I also tried to use the following shorter line > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > -lmetis" > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > LIBS=-lgomp > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > I wonder if this is authorized : the libraries are given with the > > > --with-mumps-lib, but the include are not defined. > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > What is the good practice ? > > > > Either is work - but its best to specify each package with its own > > options listed by configure. The issues usually are: > > > > - Is this package primarily a depencency of an externalpakage or if there > > is a petsc interface to it? > > > > For ex: PETSc has interface to mumps, parmetis - but not scalapack. So > > if you club parmetis into mumps-libs then petsc interface to parmetis > > would not be enabled. If you are not using this feature - it doesn't > > matter if its not enabled. > > > > - does this externalpakage also require the includes in the public > > interface? > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > - then you might have to specify metis include also with > > --with-mumps-include. Otherwise - it doesn't matter. > > > > - are there bugs in petsc configure that can trigger wrong error checks? > > > > Because mumps depends on scalapack, and optionally on > > metis,parmetis,ptscotch [i.e only one of them is required - but not > > all] - there is an error-check in configure to make sure atleast one > > of them is specified for --download-mumps. Perhaps this check should > > not trigger error for user built mumps. > > > > Did you build MUMPS with both metis, ptscotch? [and not parmetis?] > > > > Satish > > > > > > > > Best regards > > > Natacha > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay wrote: > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > --with-scalapack-lib option > > > > > > > > Satish > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > Sorry, do not take into account my last email. > > > > > I made some progress and I am now able to configure PETSc with a > > > > > pre-installed version of metis. > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > My command line is > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local installs of > > metis > > > > > and mumps) > > > > > > > > > > I get (at least) the following error > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > 'blacs_gridexit_' > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing > > from > > > > > command line > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > The configure.log is attached > > > > > Thanks a lot if you can help me ! > > > > > Natacha > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > natacha.bereux at gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Satish > > > > > > thanks a lot for the answer. Unfortunately, it does not work yet. > > > > > > More precisely : > > > > > > --download-mumps works fine (and every --download-package option > > works > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more > > generally > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > But I would like to use static librairies, preinstalled on my > > computer > > > > > > ... and this fails. > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am instead > > > > trying to > > > > > > compile with Metis 5. > > > > > > I have a preinstalled version in a some directory lets say > > metis_dir > > > > > > I try > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > metis_dir/lib/libGKlib.a] > > > > > > --with-metis-include=metis_dir/include > > > > > > this fails (see the attached config.log) > > > > > > -with-metis-dir=metis_dir also fails > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay > > > > wrote: > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > >> > > > > > >> > Hello, > > > > > >> > I am trying to compile Petsc (3.6.3) with external packages > > (MUMPS > > > > and > > > > > >> its > > > > > >> > prerequisites). > > > > > >> > More precisely I would like PETSc to use a pre-installed > > version of > > > > > >> MUMPS. > > > > > >> > > > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, > > scalapack > > > > > >> etc) : > > > > > >> > this works fine. > > > > > >> > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > > > >> > > > > > >> What version of MUMPS did you install? > > > > > >> > > > > > >> Why could you not use --download-mumps? > > > > > >> > > > > > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > > > > > >> then specifying --download-metis --download-parmetis > > > > > >> --download-scalapack [i.e different versions/builds of the same > > > > > >> libraries] can result in conflcits. > > > > > >> > > > > > >> > > > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory of > > MUMPS > > > > > >> install, > > > > > >> > but the > > > > > >> > configure step fails with the followiing message: > > > > > >> > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > > > >> details): > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > > >> > > > > > > >> > I do not understand what is wrong. > > > > > >> > I have attached the configure.log file. > > > > > >> > > > > > > >> > Any hint would be greatly appreciated ! > > > > > >> > > > > > >> >>>> > > > > > >> Executing: mpicc -show > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > -I/usr/lib/openmpi/include/openmpi > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > > > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > > -pthread > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > >> <<<< > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > >> > > > > > >> > > > > > >> >>>> > > > > > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > > > > -fPIC > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > > > > -lzmumps > > > > > >> -lmumps_common -lpord > > > > > >> > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > -lscalapack > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > -L/usr/lib/openmpi/lib > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > >> > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > -lparmetis > > > > > >> > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > -lmetis -lm > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread > > -ldl > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > >> stderr: > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > >> `GOMP_critical_name_start' > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > >> `GOMP_critical_name_end' > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > function > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > `scotchfgraphbuild_' > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > `scotchfstratinit_' > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > `scotchfgraphpart_' > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > `scotchfstratexit_' > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > `scotchfgraphexit_' > > > > > >> > > > > > >> > > > > > >> > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > >> > > > > > >> You can specify -lgomp for openmp. This can be done with the > > configure > > > > > >> option > > > > > >> LIBS=-lgomp > > > > > >> > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll > > have to > > > > > >> figureout > > > > > >> how to specify it. > > > > > >> > > > > > >> Its best to specify all the packages [mumps and its dependencies] > > you > > > > > >> built manually with the options: > > > > > >> > > > > > >> --with-package-include --with-package-lib > > > > > >> > > > > > >> Satish > > > > > >> > > > > > >> > Best regards, > > > > > >> > > > > > > >> > Natacha > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Tue Apr 19 11:37:03 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 19 Apr 2016 11:37:03 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: I see ptscotch sources includes scotch aswell. BTW: What is the reason for installing mumps separately? Does it have some patches you are experimeinting with? Is it built with extra options that --download-mumps does not enable? Something else? thanks, Satish On Tue, 19 Apr 2016, Satish Balay wrote: > Natacha, > > Are you using mumps serially or parallely? > > My understanding is - MUMPS - when used parallely requires either > parmetis or ptscotch. > > And when used sequentially - it does not require anything [i.e > metis/scotch might be optional] > > Currently PETSc configure is trying to detect/use mumps parallely - > hence its insisting on parmetis or ptscotch. > > Assuming ptscotch doesn't conflict with scotch [I don't know if > ptscotch sources also includes scotch scources or not] - its fine to > use --download-ptscotch as a workarround. [or --download-parmetis - > assuming it can build with your install of metis] > > Ideally you should not need a workarround - hence my asking for > specifics of usage - to see if our configure check for mumps needs > fixing. > > Wrt specifying scotch or metis - PETSc configure supports metis - but > not scotch. So --with-metis-lib/--with-metis-include should work - > but not --with-scotch-lib/--with-scotch-include > > But its fine to lump both into mumps options - as they are primarily > dependencies of mumps anyway. > > Satish > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > Hello, > > Thanks a lot for your explanations. > > We have a "home-made" install of MUMPS, built with metis and scotch (and - > > at the moment- neither with parmetis nor with ptscotch). > > > > When I use --with-mumps-lib/--with-mumps-include, I am forced to specify > > scotch and metis libs inside mumps libs (otherwise the link of the check > > program dmumps_c fails). And this occurs even if I have already defined the > > location of scotch libraries through > > --with-scotch-lib/--with-scotch-include and > > --with-metis-lib/--with-metis-include options. > > Defining PETSc interfaces to scotch and metis packages is (at least for > > me) not sufficient : I have to specify these libraries locations in > > --with-mumps-lib. (as shown in the configure.log). > > > > This is not really a problem and doing so fixes the configure step. > > > > I agree that duplicating scotch to define ptscotch is weird. I did so > > because we do not have a parallel built of scotch. > > From PETSc point of view, MUMPS requires either Parmetis or PTScotch : so I > > provided ptscotch through --download-ptscotch option. > > I do not intend to use it, that is only for configuring purpose ! > > > > Natacha > > > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay wrote: > > > > > Same with parmetis. > > > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > > > Hello Satish, > > > > thank you very much for yor advices. They were very helpful ! > > > > > > > > The configure step finally succeeds if I use the following configure > > > line: > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > --with-mpi=1 > > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > -lmetis -lGKlib" > > > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > -lmetis" > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > LIBS=-lgomp > > > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib option > > > > otherwise the test (on dmump) in configure fails. > > > > Is it OK to do so ? > > > > > > Hm - its best to avoid duplication of libraries. > > > > > > i.e specifying scotch via mumps-libs and also via --download-ptscotch will > > > cause problems. > > > > > > Why not specify scotch with --with-ptscotch-include,--with-ptscotch-lib > > > options? > > > > > > > > > > > I also tried to use the following shorter line > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > -lmetis" > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > LIBS=-lgomp > > > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > > I wonder if this is authorized : the libraries are given with the > > > > --with-mumps-lib, but the include are not defined. > > > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > > > > What is the good practice ? > > > > > > Either is work - but its best to specify each package with its own > > > options listed by configure. The issues usually are: > > > > > > - Is this package primarily a depencency of an externalpakage or if there > > > is a petsc interface to it? > > > > > > For ex: PETSc has interface to mumps, parmetis - but not scalapack. So > > > if you club parmetis into mumps-libs then petsc interface to parmetis > > > would not be enabled. If you are not using this feature - it doesn't > > > matter if its not enabled. > > > > > > - does this externalpakage also require the includes in the public > > > interface? > > > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > > - then you might have to specify metis include also with > > > --with-mumps-include. Otherwise - it doesn't matter. > > > > > > - are there bugs in petsc configure that can trigger wrong error checks? > > > > > > Because mumps depends on scalapack, and optionally on > > > metis,parmetis,ptscotch [i.e only one of them is required - but not > > > all] - there is an error-check in configure to make sure atleast one > > > of them is specified for --download-mumps. Perhaps this check should > > > not trigger error for user built mumps. > > > > > > Did you build MUMPS with both metis, ptscotch? [and not parmetis?] > > > > > > Satish > > > > > > > > > > > Best regards > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay wrote: > > > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > > --with-scalapack-lib option > > > > > > > > > > Satish > > > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > Sorry, do not take into account my last email. > > > > > > I made some progress and I am now able to configure PETSc with a > > > > > > pre-installed version of metis. > > > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > > > My command line is > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local installs of > > > metis > > > > > > and mumps) > > > > > > > > > > > > I get (at least) the following error > > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > > 'blacs_gridexit_' > > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO missing > > > from > > > > > > command line > > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > > > The configure.log is attached > > > > > > Thanks a lot if you can help me ! > > > > > > Natacha > > > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > > natacha.bereux at gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Satish > > > > > > > thanks a lot for the answer. Unfortunately, it does not work yet. > > > > > > > More precisely : > > > > > > > --download-mumps works fine (and every --download-package option > > > works > > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more > > > generally > > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > > > But I would like to use static librairies, preinstalled on my > > > computer > > > > > > > ... and this fails. > > > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am instead > > > > > trying to > > > > > > > compile with Metis 5. > > > > > > > I have a preinstalled version in a some directory lets say > > > metis_dir > > > > > > > I try > > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > > metis_dir/lib/libGKlib.a] > > > > > > > --with-metis-include=metis_dir/include > > > > > > > this fails (see the attached config.log) > > > > > > > -with-metis-dir=metis_dir also fails > > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay > > > > > wrote: > > > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > >> > > > > > > >> > Hello, > > > > > > >> > I am trying to compile Petsc (3.6.3) with external packages > > > (MUMPS > > > > > and > > > > > > >> its > > > > > > >> > prerequisites). > > > > > > >> > More precisely I would like PETSc to use a pre-installed > > > version of > > > > > > >> MUMPS. > > > > > > >> > > > > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, > > > scalapack > > > > > > >> etc) : > > > > > > >> > this works fine. > > > > > > >> > > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > > > > >> > > > > > > >> What version of MUMPS did you install? > > > > > > >> > > > > > > >> Why could you not use --download-mumps? > > > > > > >> > > > > > > >> Using a different metis/parmetis/slcalapack to install MUMPS - and > > > > > > >> then specifying --download-metis --download-parmetis > > > > > > >> --download-scalapack [i.e different versions/builds of the same > > > > > > >> libraries] can result in conflcits. > > > > > > >> > > > > > > >> > > > > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory of > > > MUMPS > > > > > > >> install, > > > > > > >> > but the > > > > > > >> > configure step fails with the followiing message: > > > > > > >> > > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > > > > >> details): > > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did not work > > > > > > >> > > > > > > > >> > I do not understand what is wrong. > > > > > > >> > I have attached the configure.log file. > > > > > > >> > > > > > > > >> > Any hint would be greatly appreciated ! > > > > > > >> > > > > > > >> >>>> > > > > > > >> Executing: mpicc -show > > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc > > > > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > > > > >> -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > > > -pthread > > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > > >> <<<< > > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > > >> > > > > > > >> > > > > > > >> >>>> > > > > > > >> Executing: mpicc -o /tmp/petsc-0u_4WI/config.libraries/conftest > > > > > -fPIC > > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O > > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps -lsmumps > > > > > -lzmumps > > > > > > >> -lmumps_common -lpord > > > > > > >> > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > -lscalapack > > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > > -L/usr/lib/openmpi/lib > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > > >> > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > -lparmetis > > > > > > >> > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > >> -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > -lmetis -lm > > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread > > > -ldl > > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > > >> stderr: > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > > >> `GOMP_critical_name_start' > > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > > >> `GOMP_critical_name_end' > > > > > > >> /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > > function > > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > > `scotchfgraphbuild_' > > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > > `scotchfstratinit_' > > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > > `scotchfgraphpart_' > > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > > `scotchfstratexit_' > > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > > `scotchfgraphexit_' > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > > >> > > > > > > >> You can specify -lgomp for openmp. This can be done with the > > > configure > > > > > > >> option > > > > > > >> LIBS=-lgomp > > > > > > >> > > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed we'll > > > have to > > > > > > >> figureout > > > > > > >> how to specify it. > > > > > > >> > > > > > > >> Its best to specify all the packages [mumps and its dependencies] > > > you > > > > > > >> built manually with the options: > > > > > > >> > > > > > > >> --with-package-include --with-package-lib > > > > > > >> > > > > > > >> Satish > > > > > > >> > > > > > > >> > Best regards, > > > > > > >> > > > > > > > >> > Natacha > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From natacha.bereux at gmail.com Tue Apr 19 12:39:27 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Tue, 19 Apr 2016 19:39:27 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Yes, we have a custom MUMPS version (because my company, EDF, belongs to MUMPS consortium). Thereby, we get some patches and functionnalities in advance with respect to the public version. The option --download-mumps also works fine (I only have to define the path to a local archive of sources). Our build does not use extra options. We prefer to compile MUMPS alone, and in a second time build a compatible PETSc library. In fact we aim to use MUMPS in two ways : - direct call of MUMPS driver (this is the legacy way) - through PETSc interface (this is what I am experiencing) Best regards Natacha On Tue, Apr 19, 2016 at 6:37 PM, Satish Balay wrote: > I see ptscotch sources includes scotch aswell. > > BTW: What is the reason for installing mumps separately? Does it have > some patches you are experimeinting with? Is it built with extra > options that --download-mumps does not enable? Something else? > > thanks, > Satish > > On Tue, 19 Apr 2016, Satish Balay wrote: > > > Natacha, > > > > Are you using mumps serially or parallely? > > > > My understanding is - MUMPS - when used parallely requires either > > parmetis or ptscotch. > > > > And when used sequentially - it does not require anything [i.e > > metis/scotch might be optional] > > > > Currently PETSc configure is trying to detect/use mumps parallely - > > hence its insisting on parmetis or ptscotch. > > > > Assuming ptscotch doesn't conflict with scotch [I don't know if > > ptscotch sources also includes scotch scources or not] - its fine to > > use --download-ptscotch as a workarround. [or --download-parmetis - > > assuming it can build with your install of metis] > > > > Ideally you should not need a workarround - hence my asking for > > specifics of usage - to see if our configure check for mumps needs > > fixing. > > > > Wrt specifying scotch or metis - PETSc configure supports metis - but > > not scotch. So --with-metis-lib/--with-metis-include should work - > > but not --with-scotch-lib/--with-scotch-include > > > > But its fine to lump both into mumps options - as they are primarily > > dependencies of mumps anyway. > > > > Satish > > > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > > > Hello, > > > Thanks a lot for your explanations. > > > We have a "home-made" install of MUMPS, built with metis and scotch > (and - > > > at the moment- neither with parmetis nor with ptscotch). > > > > > > When I use --with-mumps-lib/--with-mumps-include, I am forced to > specify > > > scotch and metis libs inside mumps libs (otherwise the link of the > check > > > program dmumps_c fails). And this occurs even if I have already > defined the > > > location of scotch libraries through > > > --with-scotch-lib/--with-scotch-include and > > > --with-metis-lib/--with-metis-include options. > > > Defining PETSc interfaces to scotch and metis packages is (at least for > > > me) not sufficient : I have to specify these libraries locations in > > > --with-mumps-lib. (as shown in the configure.log). > > > > > > This is not really a problem and doing so fixes the configure step. > > > > > > I agree that duplicating scotch to define ptscotch is weird. I did so > > > because we do not have a parallel built of scotch. > > > From PETSc point of view, MUMPS requires either Parmetis or PTScotch : > so I > > > provided ptscotch through --download-ptscotch option. > > > I do not intend to use it, that is only for configuring purpose ! > > > > > > Natacha > > > > > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay > wrote: > > > > > > > Same with parmetis. > > > > > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > Hello Satish, > > > > > thank you very much for yor advices. They were very helpful ! > > > > > > > > > > The configure step finally succeeds if I use the following > configure > > > > line: > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > --with-mpi=1 > > > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > -lmetis -lGKlib" > > > > > > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > -lmetis" > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > LIBS=-lgomp > > > > > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib > option > > > > > otherwise the test (on dmump) in configure fails. > > > > > Is it OK to do so ? > > > > > > > > Hm - its best to avoid duplication of libraries. > > > > > > > > i.e specifying scotch via mumps-libs and also via > --download-ptscotch will > > > > cause problems. > > > > > > > > Why not specify scotch with > --with-ptscotch-include,--with-ptscotch-lib > > > > options? > > > > > > > > > > > > > > I also tried to use the following shorter line > > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > -lmetis" > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > LIBS=-lgomp > > > > > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > > > I wonder if this is authorized : the libraries are given with the > > > > > --with-mumps-lib, but the include are not defined. > > > > > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > > > > > > > What is the good practice ? > > > > > > > > Either is work - but its best to specify each package with its own > > > > options listed by configure. The issues usually are: > > > > > > > > - Is this package primarily a depencency of an externalpakage or if > there > > > > is a petsc interface to it? > > > > > > > > For ex: PETSc has interface to mumps, parmetis - but not scalapack. > So > > > > if you club parmetis into mumps-libs then petsc interface to parmetis > > > > would not be enabled. If you are not using this feature - it doesn't > > > > matter if its not enabled. > > > > > > > > - does this externalpakage also require the includes in the public > > > > interface? > > > > > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > > > - then you might have to specify metis include also with > > > > --with-mumps-include. Otherwise - it doesn't matter. > > > > > > > > - are there bugs in petsc configure that can trigger wrong error > checks? > > > > > > > > Because mumps depends on scalapack, and optionally on > > > > metis,parmetis,ptscotch [i.e only one of them is required - but not > > > > all] - there is an error-check in configure to make sure atleast one > > > > of them is specified for --download-mumps. Perhaps this check should > > > > not trigger error for user built mumps. > > > > > > > > Did you build MUMPS with both metis, ptscotch? [and not parmetis?] > > > > > > > > Satish > > > > > > > > > > > > > > Best regards > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay > wrote: > > > > > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > > > --with-scalapack-lib option > > > > > > > > > > > > Satish > > > > > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > Sorry, do not take into account my last email. > > > > > > > I made some progress and I am now able to configure PETSc > with a > > > > > > > pre-installed version of metis. > > > > > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > > > > > My command line is > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local > installs of > > > > metis > > > > > > > and mumps) > > > > > > > > > > > > > > I get (at least) the following error > > > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > > > 'blacs_gridexit_' > > > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO > missing > > > > from > > > > > > > command line > > > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > > > > > The configure.log is attached > > > > > > > Thanks a lot if you can help me ! > > > > > > > Natacha > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > > > natacha.bereux at gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Satish > > > > > > > > thanks a lot for the answer. Unfortunately, it does not work > yet. > > > > > > > > More precisely : > > > > > > > > --download-mumps works fine (and every --download-package > option > > > > works > > > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more > > > > generally > > > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > > > > > But I would like to use static librairies, preinstalled on > my > > > > computer > > > > > > > > ... and this fails. > > > > > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am > instead > > > > > > trying to > > > > > > > > compile with Metis 5. > > > > > > > > I have a preinstalled version in a some directory lets say > > > > metis_dir > > > > > > > > I try > > > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > > > metis_dir/lib/libGKlib.a] > > > > > > > > --with-metis-include=metis_dir/include > > > > > > > > this fails (see the attached config.log) > > > > > > > > -with-metis-dir=metis_dir also fails > > > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay < > balay at mcs.anl.gov> > > > > > > wrote: > > > > > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > > >> > > > > > > > >> > Hello, > > > > > > > >> > I am trying to compile Petsc (3.6.3) with external > packages > > > > (MUMPS > > > > > > and > > > > > > > >> its > > > > > > > >> > prerequisites). > > > > > > > >> > More precisely I would like PETSc to use a pre-installed > > > > version of > > > > > > > >> MUMPS. > > > > > > > >> > > > > > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, > > > > scalapack > > > > > > > >> etc) : > > > > > > > >> > this works fine. > > > > > > > >> > > > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > > > > > >> > > > > > > > >> What version of MUMPS did you install? > > > > > > > >> > > > > > > > >> Why could you not use --download-mumps? > > > > > > > >> > > > > > > > >> Using a different metis/parmetis/slcalapack to install > MUMPS - and > > > > > > > >> then specifying --download-metis --download-parmetis > > > > > > > >> --download-scalapack [i.e different versions/builds of the > same > > > > > > > >> libraries] can result in conflcits. > > > > > > > >> > > > > > > > >> > > > > > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory > of > > > > MUMPS > > > > > > > >> install, > > > > > > > >> > but the > > > > > > > >> > configure step fails with the followiing message: > > > > > > > >> > > > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see > configure.log for > > > > > > > >> details): > > > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did > not work > > > > > > > >> > > > > > > > > >> > I do not understand what is wrong. > > > > > > > >> > I have attached the configure.log file. > > > > > > > >> > > > > > > > > >> > Any hint would be greatly appreciated ! > > > > > > > >> > > > > > > > >> >>>> > > > > > > > >> Executing: mpicc -show > > > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl > -lhwloc > > > > > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > > > > > >> -I/usr/lib/openmpi/include > -I/usr/lib/openmpi/include/openmpi > > > > -pthread > > > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > > > >> <<<< > > > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > > > > > > > >> Executing: mpicc -o > /tmp/petsc-0u_4WI/config.libraries/conftest > > > > > > -fPIC > > > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -O > > > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps > -lsmumps > > > > > > -lzmumps > > > > > > > >> -lmumps_common -lpord > > > > > > > >> > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > >> > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > -lscalapack > > > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > > > -L/usr/lib/openmpi/lib > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > > > >> > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > >> > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > -lparmetis > > > > > > > >> > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > >> > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > -lmetis -lm > > > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s > -lpthread > > > > -ldl > > > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > > > >> stderr: > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function > `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function > `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > >> function > `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > > > >> `GOMP_critical_name_start' > > > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > > > >> `GOMP_critical_name_end' > > > > > > > >> > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > > > function > > > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > > > `scotchfgraphbuild_' > > > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > > > `scotchfstratinit_' > > > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > > > `scotchfgraphpart_' > > > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > > > `scotchfstratexit_' > > > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > > > `scotchfgraphexit_' > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > > > >> > > > > > > > >> You can specify -lgomp for openmp. This can be done with the > > > > configure > > > > > > > >> option > > > > > > > >> LIBS=-lgomp > > > > > > > >> > > > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed > we'll > > > > have to > > > > > > > >> figureout > > > > > > > >> how to specify it. > > > > > > > >> > > > > > > > >> Its best to specify all the packages [mumps and its > dependencies] > > > > you > > > > > > > >> built manually with the options: > > > > > > > >> > > > > > > > >> --with-package-include --with-package-lib > > > > > > > >> > > > > > > > >> Satish > > > > > > > >> > > > > > > > >> > Best regards, > > > > > > > >> > > > > > > > > >> > Natacha > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 20 11:43:17 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 20 Apr 2016 11:43:17 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Natacha, With https://bitbucket.org/petsc/petsc/commits/c0c82a18b4559f6e977fb5e0c503b006f23ed070 you should be able to build mumps as you wish - and specify the dependencies in anyway [all clubed with mumps - or metis seprately - but scotch clubbed into metis] And I think --download-mumps=your-copy-of-mumps.tar.gz would be easier to use. But you might have other reasons to install mumps standalone - and specify to petsc [and this should also be fine] [for now this fix is in petsc git repo - and will be available in the next full release] thanks, Satish On Tue, 19 Apr 2016, Natacha BEREUX wrote: > Yes, we have a custom MUMPS version (because my company, EDF, belongs to > MUMPS consortium). Thereby, we get some patches and functionnalities in > advance with respect to the public version. > > The option --download-mumps also works fine (I only have to define the path > to a local archive of sources). Our build does not use extra options. > We prefer to compile MUMPS alone, and in a second time build a compatible > PETSc library. > > In fact we aim to use MUMPS in two ways : > - direct call of MUMPS driver (this is the legacy way) > - through PETSc interface (this is what I am experiencing) > Best regards > Natacha > > > > > > On Tue, Apr 19, 2016 at 6:37 PM, Satish Balay wrote: > > > I see ptscotch sources includes scotch aswell. > > > > BTW: What is the reason for installing mumps separately? Does it have > > some patches you are experimeinting with? Is it built with extra > > options that --download-mumps does not enable? Something else? > > > > thanks, > > Satish > > > > On Tue, 19 Apr 2016, Satish Balay wrote: > > > > > Natacha, > > > > > > Are you using mumps serially or parallely? > > > > > > My understanding is - MUMPS - when used parallely requires either > > > parmetis or ptscotch. > > > > > > And when used sequentially - it does not require anything [i.e > > > metis/scotch might be optional] > > > > > > Currently PETSc configure is trying to detect/use mumps parallely - > > > hence its insisting on parmetis or ptscotch. > > > > > > Assuming ptscotch doesn't conflict with scotch [I don't know if > > > ptscotch sources also includes scotch scources or not] - its fine to > > > use --download-ptscotch as a workarround. [or --download-parmetis - > > > assuming it can build with your install of metis] > > > > > > Ideally you should not need a workarround - hence my asking for > > > specifics of usage - to see if our configure check for mumps needs > > > fixing. > > > > > > Wrt specifying scotch or metis - PETSc configure supports metis - but > > > not scotch. So --with-metis-lib/--with-metis-include should work - > > > but not --with-scotch-lib/--with-scotch-include > > > > > > But its fine to lump both into mumps options - as they are primarily > > > dependencies of mumps anyway. > > > > > > Satish > > > > > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > > > > > Hello, > > > > Thanks a lot for your explanations. > > > > We have a "home-made" install of MUMPS, built with metis and scotch > > (and - > > > > at the moment- neither with parmetis nor with ptscotch). > > > > > > > > When I use --with-mumps-lib/--with-mumps-include, I am forced to > > specify > > > > scotch and metis libs inside mumps libs (otherwise the link of the > > check > > > > program dmumps_c fails). And this occurs even if I have already > > defined the > > > > location of scotch libraries through > > > > --with-scotch-lib/--with-scotch-include and > > > > --with-metis-lib/--with-metis-include options. > > > > Defining PETSc interfaces to scotch and metis packages is (at least for > > > > me) not sufficient : I have to specify these libraries locations in > > > > --with-mumps-lib. (as shown in the configure.log). > > > > > > > > This is not really a problem and doing so fixes the configure step. > > > > > > > > I agree that duplicating scotch to define ptscotch is weird. I did so > > > > because we do not have a parallel built of scotch. > > > > From PETSc point of view, MUMPS requires either Parmetis or PTScotch : > > so I > > > > provided ptscotch through --download-ptscotch option. > > > > I do not intend to use it, that is only for configuring purpose ! > > > > > > > > Natacha > > > > > > > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay > > wrote: > > > > > > > > > Same with parmetis. > > > > > > > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > Hello Satish, > > > > > > thank you very much for yor advices. They were very helpful ! > > > > > > > > > > > > The configure step finally succeeds if I use the following > > configure > > > > > line: > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > --with-mpi=1 > > > > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > -lmetis -lGKlib" > > > > > > > > > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > -lmetis" > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > LIBS=-lgomp > > > > > > > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib > > option > > > > > > otherwise the test (on dmump) in configure fails. > > > > > > Is it OK to do so ? > > > > > > > > > > Hm - its best to avoid duplication of libraries. > > > > > > > > > > i.e specifying scotch via mumps-libs and also via > > --download-ptscotch will > > > > > cause problems. > > > > > > > > > > Why not specify scotch with > > --with-ptscotch-include,--with-ptscotch-lib > > > > > options? > > > > > > > > > > > > > > > > > I also tried to use the following shorter line > > > > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > -lmetis" > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > LIBS=-lgomp > > > > > > > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > > > > I wonder if this is authorized : the libraries are given with the > > > > > > --with-mumps-lib, but the include are not defined. > > > > > > > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > > > > > > > > > > What is the good practice ? > > > > > > > > > > Either is work - but its best to specify each package with its own > > > > > options listed by configure. The issues usually are: > > > > > > > > > > - Is this package primarily a depencency of an externalpakage or if > > there > > > > > is a petsc interface to it? > > > > > > > > > > For ex: PETSc has interface to mumps, parmetis - but not scalapack. > > So > > > > > if you club parmetis into mumps-libs then petsc interface to parmetis > > > > > would not be enabled. If you are not using this feature - it doesn't > > > > > matter if its not enabled. > > > > > > > > > > - does this externalpakage also require the includes in the public > > > > > interface? > > > > > > > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > > > > - then you might have to specify metis include also with > > > > > --with-mumps-include. Otherwise - it doesn't matter. > > > > > > > > > > - are there bugs in petsc configure that can trigger wrong error > > checks? > > > > > > > > > > Because mumps depends on scalapack, and optionally on > > > > > metis,parmetis,ptscotch [i.e only one of them is required - but not > > > > > all] - there is an error-check in configure to make sure atleast one > > > > > of them is specified for --download-mumps. Perhaps this check should > > > > > not trigger error for user built mumps. > > > > > > > > > > Did you build MUMPS with both metis, ptscotch? [and not parmetis?] > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > Best regards > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay > > wrote: > > > > > > > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > > > > --with-scalapack-lib option > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > > > Sorry, do not take into account my last email. > > > > > > > > I made some progress and I am now able to configure PETSc > > with a > > > > > > > > pre-installed version of metis. > > > > > > > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > > > > > > > My command line is > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local > > installs of > > > > > metis > > > > > > > > and mumps) > > > > > > > > > > > > > > > > I get (at least) the following error > > > > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > > > > 'blacs_gridexit_' > > > > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO > > missing > > > > > from > > > > > > > > command line > > > > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > > > > > > > The configure.log is attached > > > > > > > > Thanks a lot if you can help me ! > > > > > > > > Natacha > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > > > > natacha.bereux at gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Satish > > > > > > > > > thanks a lot for the answer. Unfortunately, it does not work > > yet. > > > > > > > > > More precisely : > > > > > > > > > --download-mumps works fine (and every --download-package > > option > > > > > works > > > > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or more > > > > > generally > > > > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > > > > > > > But I would like to use static librairies, preinstalled on > > my > > > > > computer > > > > > > > > > ... and this fails. > > > > > > > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am > > instead > > > > > > > trying to > > > > > > > > > compile with Metis 5. > > > > > > > > > I have a preinstalled version in a some directory lets say > > > > > metis_dir > > > > > > > > > I try > > > > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > > > > metis_dir/lib/libGKlib.a] > > > > > > > > > --with-metis-include=metis_dir/include > > > > > > > > > this fails (see the attached config.log) > > > > > > > > > -with-metis-dir=metis_dir also fails > > > > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay < > > balay at mcs.anl.gov> > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > > > >> > > > > > > > > >> > Hello, > > > > > > > > >> > I am trying to compile Petsc (3.6.3) with external > > packages > > > > > (MUMPS > > > > > > > and > > > > > > > > >> its > > > > > > > > >> > prerequisites). > > > > > > > > >> > More precisely I would like PETSc to use a pre-installed > > > > > version of > > > > > > > > >> MUMPS. > > > > > > > > >> > > > > > > > > > >> > Petsc downloads and compiles the prerequisites (parmetis, > > > > > scalapack > > > > > > > > >> etc) : > > > > > > > > >> > this works fine. > > > > > > > > >> > > > > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed with? > > > > > > > > >> > > > > > > > > >> What version of MUMPS did you install? > > > > > > > > >> > > > > > > > > >> Why could you not use --download-mumps? > > > > > > > > >> > > > > > > > > >> Using a different metis/parmetis/slcalapack to install > > MUMPS - and > > > > > > > > >> then specifying --download-metis --download-parmetis > > > > > > > > >> --download-scalapack [i.e different versions/builds of the > > same > > > > > > > > >> libraries] can result in conflcits. > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > I define MUMPS location by --with-mumps-dir=top-directory > > of > > > > > MUMPS > > > > > > > > >> install, > > > > > > > > >> > but the > > > > > > > > >> > configure step fails with the followiing message: > > > > > > > > >> > > > > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see > > configure.log for > > > > > > > > >> details): > > > > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did > > not work > > > > > > > > >> > > > > > > > > > >> > I do not understand what is wrong. > > > > > > > > >> > I have attached the configure.log file. > > > > > > > > >> > > > > > > > > > >> > Any hint would be greatly appreciated ! > > > > > > > > >> > > > > > > > > >> >>>> > > > > > > > > >> Executing: mpicc -show > > > > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl > > -lhwloc > > > > > > > > >> Defined make macro "MPICC_SHOW" to "gcc > > > > > > > > >> -I/usr/lib/openmpi/include > > -I/usr/lib/openmpi/include/openmpi > > > > > -pthread > > > > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > > > > >> <<<< > > > > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> >>>> > > > > > > > > >> Executing: mpicc -o > > /tmp/petsc-0u_4WI/config.libraries/conftest > > > > > > > -fPIC > > > > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing > > -Wno-unknown-pragmas -O > > > > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps -ldmumps > > -lsmumps > > > > > > > -lzmumps > > > > > > > > >> -lmumps_common -lpord > > > > > > > > >> > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > >> > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > -lscalapack > > > > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > > > > -L/usr/lib/openmpi/lib > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > > > > >> > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > >> > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > -lparmetis > > > > > > > > >> > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > >> > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > -lmetis -lm > > > > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > >> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s > > -lpthread > > > > > -ldl > > > > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > > > > >> stderr: > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function > > `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > >> function > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > >> > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > > > > function > > > > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > > > > `scotchfgraphbuild_' > > > > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > > > > `scotchfstratinit_' > > > > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > > > > `scotchfgraphpart_' > > > > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > > > > `scotchfstratexit_' > > > > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > > > > `scotchfgraphexit_' > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > > > > >> > > > > > > > > >> You can specify -lgomp for openmp. This can be done with the > > > > > configure > > > > > > > > >> option > > > > > > > > >> LIBS=-lgomp > > > > > > > > >> > > > > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed > > we'll > > > > > have to > > > > > > > > >> figureout > > > > > > > > >> how to specify it. > > > > > > > > >> > > > > > > > > >> Its best to specify all the packages [mumps and its > > dependencies] > > > > > you > > > > > > > > >> built manually with the options: > > > > > > > > >> > > > > > > > > >> --with-package-include --with-package-lib > > > > > > > > >> > > > > > > > > >> Satish > > > > > > > > >> > > > > > > > > >> > Best regards, > > > > > > > > >> > > > > > > > > > >> > Natacha > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From natacha.bereux at gmail.com Wed Apr 20 15:51:22 2016 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Wed, 20 Apr 2016 22:51:22 +0200 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Thanks a lot Satish, I've learned a lot on Petsc build process ! I am away for the next two weeks and will test the patch when I come back . Thanks again Natacha On Wed, Apr 20, 2016 at 6:43 PM, Satish Balay wrote: > Natacha, > > With > https://bitbucket.org/petsc/petsc/commits/c0c82a18b4559f6e977fb5e0c503b006f23ed070 > you should be able to build mumps as you wish - and specify the > dependencies in anyway > [all clubed with mumps - or metis seprately - but scotch clubbed into > metis] > > And I think --download-mumps=your-copy-of-mumps.tar.gz would be easier to > use. But you > might have other reasons to install mumps standalone - and specify to > petsc [and this > should also be fine] > > [for now this fix is in petsc git repo - and will be available in the next > full release] > > thanks, > Satish > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > Yes, we have a custom MUMPS version (because my company, EDF, belongs to > > MUMPS consortium). Thereby, we get some patches and functionnalities in > > advance with respect to the public version. > > > > The option --download-mumps also works fine (I only have to define the > path > > to a local archive of sources). Our build does not use extra options. > > We prefer to compile MUMPS alone, and in a second time build a compatible > > PETSc library. > > > > In fact we aim to use MUMPS in two ways : > > - direct call of MUMPS driver (this is the legacy way) > > - through PETSc interface (this is what I am experiencing) > > Best regards > > Natacha > > > > > > > > > > > > On Tue, Apr 19, 2016 at 6:37 PM, Satish Balay wrote: > > > > > I see ptscotch sources includes scotch aswell. > > > > > > BTW: What is the reason for installing mumps separately? Does it have > > > some patches you are experimeinting with? Is it built with extra > > > options that --download-mumps does not enable? Something else? > > > > > > thanks, > > > Satish > > > > > > On Tue, 19 Apr 2016, Satish Balay wrote: > > > > > > > Natacha, > > > > > > > > Are you using mumps serially or parallely? > > > > > > > > My understanding is - MUMPS - when used parallely requires either > > > > parmetis or ptscotch. > > > > > > > > And when used sequentially - it does not require anything [i.e > > > > metis/scotch might be optional] > > > > > > > > Currently PETSc configure is trying to detect/use mumps parallely - > > > > hence its insisting on parmetis or ptscotch. > > > > > > > > Assuming ptscotch doesn't conflict with scotch [I don't know if > > > > ptscotch sources also includes scotch scources or not] - its fine to > > > > use --download-ptscotch as a workarround. [or --download-parmetis - > > > > assuming it can build with your install of metis] > > > > > > > > Ideally you should not need a workarround - hence my asking for > > > > specifics of usage - to see if our configure check for mumps needs > > > > fixing. > > > > > > > > Wrt specifying scotch or metis - PETSc configure supports metis - but > > > > not scotch. So --with-metis-lib/--with-metis-include should work - > > > > but not --with-scotch-lib/--with-scotch-include > > > > > > > > But its fine to lump both into mumps options - as they are primarily > > > > dependencies of mumps anyway. > > > > > > > > Satish > > > > > > > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > Hello, > > > > > Thanks a lot for your explanations. > > > > > We have a "home-made" install of MUMPS, built with metis and scotch > > > (and - > > > > > at the moment- neither with parmetis nor with ptscotch). > > > > > > > > > > When I use --with-mumps-lib/--with-mumps-include, I am forced to > > > specify > > > > > scotch and metis libs inside mumps libs (otherwise the link of the > > > check > > > > > program dmumps_c fails). And this occurs even if I have already > > > defined the > > > > > location of scotch libraries through > > > > > --with-scotch-lib/--with-scotch-include and > > > > > --with-metis-lib/--with-metis-include options. > > > > > Defining PETSc interfaces to scotch and metis packages is (at > least for > > > > > me) not sufficient : I have to specify these libraries locations > in > > > > > --with-mumps-lib. (as shown in the configure.log). > > > > > > > > > > This is not really a problem and doing so fixes the configure step. > > > > > > > > > > I agree that duplicating scotch to define ptscotch is weird. I did > so > > > > > because we do not have a parallel built of scotch. > > > > > From PETSc point of view, MUMPS requires either Parmetis or > PTScotch : > > > so I > > > > > provided ptscotch through --download-ptscotch option. > > > > > I do not intend to use it, that is only for configuring purpose ! > > > > > > > > > > Natacha > > > > > > > > > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay > > > wrote: > > > > > > > > > > > Same with parmetis. > > > > > > > > > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > Hello Satish, > > > > > > > thank you very much for yor advices. They were very helpful ! > > > > > > > > > > > > > > The configure step finally succeeds if I use the following > > > configure > > > > > > line: > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > --with-mpi=1 > > > > > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > -lmetis -lGKlib" > > > > > > > > > > > > > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > -lmetis" > > > > > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > > LIBS=-lgomp > > > > > > > > > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > > > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib > > > option > > > > > > > otherwise the test (on dmump) in configure fails. > > > > > > > Is it OK to do so ? > > > > > > > > > > > > Hm - its best to avoid duplication of libraries. > > > > > > > > > > > > i.e specifying scotch via mumps-libs and also via > > > --download-ptscotch will > > > > > > cause problems. > > > > > > > > > > > > Why not specify scotch with > > > --with-ptscotch-include,--with-ptscotch-lib > > > > > > options? > > > > > > > > > > > > > > > > > > > > I also tried to use the following shorter line > > > > > > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > -lmetis" > > > > > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > > LIBS=-lgomp > > > > > > > > > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > > > > > I wonder if this is authorized : the libraries are given with > the > > > > > > > --with-mumps-lib, but the include are not defined. > > > > > > > > > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > > > > > > > > > > > > > What is the good practice ? > > > > > > > > > > > > Either is work - but its best to specify each package with its > own > > > > > > options listed by configure. The issues usually are: > > > > > > > > > > > > - Is this package primarily a depencency of an externalpakage or > if > > > there > > > > > > is a petsc interface to it? > > > > > > > > > > > > For ex: PETSc has interface to mumps, parmetis - but not > scalapack. > > > So > > > > > > if you club parmetis into mumps-libs then petsc interface to > parmetis > > > > > > would not be enabled. If you are not using this feature - it > doesn't > > > > > > matter if its not enabled. > > > > > > > > > > > > - does this externalpakage also require the includes in the > public > > > > > > interface? > > > > > > > > > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > > > > > - then you might have to specify metis include also with > > > > > > --with-mumps-include. Otherwise - it doesn't matter. > > > > > > > > > > > > - are there bugs in petsc configure that can trigger wrong error > > > checks? > > > > > > > > > > > > Because mumps depends on scalapack, and optionally on > > > > > > metis,parmetis,ptscotch [i.e only one of them is required - but > not > > > > > > all] - there is an error-check in configure to make sure atleast > one > > > > > > of them is specified for --download-mumps. Perhaps this check > should > > > > > > not trigger error for user built mumps. > > > > > > > > > > > > Did you build MUMPS with both metis, ptscotch? [and not > parmetis?] > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay < > balay at mcs.anl.gov> > > > wrote: > > > > > > > > > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > > > > > --with-scalapack-lib option > > > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > > > > > Sorry, do not take into account my last email. > > > > > > > > > I made some progress and I am now able to configure PETSc > > > with a > > > > > > > > > pre-installed version of metis. > > > > > > > > > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > > > > > > > > > My command line is > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx > --with-fc=mpif90 > > > > > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local > > > installs of > > > > > > metis > > > > > > > > > and mumps) > > > > > > > > > > > > > > > > > > I get (at least) the following error > > > > > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > > > > > 'blacs_gridexit_' > > > > > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO > > > missing > > > > > > from > > > > > > > > > command line > > > > > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > > > > > > > > > The configure.log is attached > > > > > > > > > Thanks a lot if you can help me ! > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > > > > > natacha.bereux at gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Satish > > > > > > > > > > thanks a lot for the answer. Unfortunately, it does not > work > > > yet. > > > > > > > > > > More precisely : > > > > > > > > > > --download-mumps works fine (and every > --download-package > > > option > > > > > > works > > > > > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or > more > > > > > > generally > > > > > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > > > > > > > > > But I would like to use static librairies, preinstalled > on > > > my > > > > > > computer > > > > > > > > > > ... and this fails. > > > > > > > > > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am > > > instead > > > > > > > > trying to > > > > > > > > > > compile with Metis 5. > > > > > > > > > > I have a preinstalled version in a some directory lets > say > > > > > > metis_dir > > > > > > > > > > I try > > > > > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > > > > > metis_dir/lib/libGKlib.a] > > > > > > > > > > --with-metis-include=metis_dir/include > > > > > > > > > > this fails (see the attached config.log) > > > > > > > > > > -with-metis-dir=metis_dir also fails > > > > > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay < > > > balay at mcs.anl.gov> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > >> > > > > > > > > > >> > Hello, > > > > > > > > > >> > I am trying to compile Petsc (3.6.3) with external > > > packages > > > > > > (MUMPS > > > > > > > > and > > > > > > > > > >> its > > > > > > > > > >> > prerequisites). > > > > > > > > > >> > More precisely I would like PETSc to use a > pre-installed > > > > > > version of > > > > > > > > > >> MUMPS. > > > > > > > > > >> > > > > > > > > > > >> > Petsc downloads and compiles the prerequisites > (parmetis, > > > > > > scalapack > > > > > > > > > >> etc) : > > > > > > > > > >> > this works fine. > > > > > > > > > >> > > > > > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed > with? > > > > > > > > > >> > > > > > > > > > >> What version of MUMPS did you install? > > > > > > > > > >> > > > > > > > > > >> Why could you not use --download-mumps? > > > > > > > > > >> > > > > > > > > > >> Using a different metis/parmetis/slcalapack to install > > > MUMPS - and > > > > > > > > > >> then specifying --download-metis --download-parmetis > > > > > > > > > >> --download-scalapack [i.e different versions/builds of > the > > > same > > > > > > > > > >> libraries] can result in conflcits. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > I define MUMPS location by > --with-mumps-dir=top-directory > > > of > > > > > > MUMPS > > > > > > > > > >> install, > > > > > > > > > >> > but the > > > > > > > > > >> > configure step fails with the followiing message: > > > > > > > > > >> > > > > > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see > > > configure.log for > > > > > > > > > >> details): > > > > > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did > > > not work > > > > > > > > > >> > > > > > > > > > > >> > I do not understand what is wrong. > > > > > > > > > >> > I have attached the configure.log file. > > > > > > > > > >> > > > > > > > > > > >> > Any hint would be greatly appreciated ! > > > > > > > > > >> > > > > > > > > > >> >>>> > > > > > > > > > >> Executing: mpicc -show > > > > > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl > > > -lhwloc > > > > > > > > > >> Defined make macro "MPICC_SHOW" to > "gcc > > > > > > > > > >> -I/usr/lib/openmpi/include > > > -I/usr/lib/openmpi/include/openmpi > > > > > > -pthread > > > > > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > > > > > >> <<<< > > > > > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> >>>> > > > > > > > > > >> Executing: mpicc -o > > > /tmp/petsc-0u_4WI/config.libraries/conftest > > > > > > > > -fPIC > > > > > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing > > > -Wno-unknown-pragmas -O > > > > > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps > -ldmumps > > > -lsmumps > > > > > > > > -lzmumps > > > > > > > > > >> -lmumps_common -lpord > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > >> > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > -lscalapack > > > > > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > -L/usr/lib/openmpi/lib > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran > -lm > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran > -lm > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > >> > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > -lparmetis > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > >> > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > -lmetis -lm > > > > > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib > -L/usr/lib/openmpi/lib > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > >> -L/lib/x86_64-linux-gnu > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s > > > -lpthread > > > > > > -ldl > > > > > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > > > > > >> stderr: > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function > > > `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function > `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function > `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function > > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > >> function > > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > >> > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > > > > > function > > > > > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > > > > > `scotchfgraphbuild_' > > > > > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > > > > > `scotchfstratinit_' > > > > > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > > > > > `scotchfgraphpart_' > > > > > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > > > > > `scotchfstratexit_' > > > > > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > > > > > `scotchfgraphexit_' > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > > > > > >> > > > > > > > > > >> You can specify -lgomp for openmp. This can be done > with the > > > > > > configure > > > > > > > > > >> option > > > > > > > > > >> LIBS=-lgomp > > > > > > > > > >> > > > > > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed > > > we'll > > > > > > have to > > > > > > > > > >> figureout > > > > > > > > > >> how to specify it. > > > > > > > > > >> > > > > > > > > > >> Its best to specify all the packages [mumps and its > > > dependencies] > > > > > > you > > > > > > > > > >> built manually with the options: > > > > > > > > > >> > > > > > > > > > >> --with-package-include --with-package-lib > > > > > > > > > >> > > > > > > > > > >> Satish > > > > > > > > > >> > > > > > > > > > >> > Best regards, > > > > > > > > > >> > > > > > > > > > > >> > Natacha > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 20 16:01:46 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 20 Apr 2016 16:01:46 -0500 Subject: [petsc-users] Compiling petsc with a user-defined MUMPS directory In-Reply-To: References: Message-ID: Natacha, The patch is not likely to work with petsc-3.6. You can obtain petsc git repo and use the branch 'balay/git-externalpackage-update'. [It should be in branch 'master' by next week] http://www.mcs.anl.gov/petsc/developers/index.html Satish On Wed, 20 Apr 2016, Natacha BEREUX wrote: > Thanks a lot Satish, > I've learned a lot on Petsc build process ! > I am away for the next two weeks and will test the patch when I come back . > Thanks again > Natacha > > > On Wed, Apr 20, 2016 at 6:43 PM, Satish Balay wrote: > > > Natacha, > > > > With > > https://bitbucket.org/petsc/petsc/commits/c0c82a18b4559f6e977fb5e0c503b006f23ed070 > > you should be able to build mumps as you wish - and specify the > > dependencies in anyway > > [all clubed with mumps - or metis seprately - but scotch clubbed into > > metis] > > > > And I think --download-mumps=your-copy-of-mumps.tar.gz would be easier to > > use. But you > > might have other reasons to install mumps standalone - and specify to > > petsc [and this > > should also be fine] > > > > [for now this fix is in petsc git repo - and will be available in the next > > full release] > > > > thanks, > > Satish > > > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > > > Yes, we have a custom MUMPS version (because my company, EDF, belongs to > > > MUMPS consortium). Thereby, we get some patches and functionnalities in > > > advance with respect to the public version. > > > > > > The option --download-mumps also works fine (I only have to define the > > path > > > to a local archive of sources). Our build does not use extra options. > > > We prefer to compile MUMPS alone, and in a second time build a compatible > > > PETSc library. > > > > > > In fact we aim to use MUMPS in two ways : > > > - direct call of MUMPS driver (this is the legacy way) > > > - through PETSc interface (this is what I am experiencing) > > > Best regards > > > Natacha > > > > > > > > > > > > > > > > > > On Tue, Apr 19, 2016 at 6:37 PM, Satish Balay wrote: > > > > > > > I see ptscotch sources includes scotch aswell. > > > > > > > > BTW: What is the reason for installing mumps separately? Does it have > > > > some patches you are experimeinting with? Is it built with extra > > > > options that --download-mumps does not enable? Something else? > > > > > > > > thanks, > > > > Satish > > > > > > > > On Tue, 19 Apr 2016, Satish Balay wrote: > > > > > > > > > Natacha, > > > > > > > > > > Are you using mumps serially or parallely? > > > > > > > > > > My understanding is - MUMPS - when used parallely requires either > > > > > parmetis or ptscotch. > > > > > > > > > > And when used sequentially - it does not require anything [i.e > > > > > metis/scotch might be optional] > > > > > > > > > > Currently PETSc configure is trying to detect/use mumps parallely - > > > > > hence its insisting on parmetis or ptscotch. > > > > > > > > > > Assuming ptscotch doesn't conflict with scotch [I don't know if > > > > > ptscotch sources also includes scotch scources or not] - its fine to > > > > > use --download-ptscotch as a workarround. [or --download-parmetis - > > > > > assuming it can build with your install of metis] > > > > > > > > > > Ideally you should not need a workarround - hence my asking for > > > > > specifics of usage - to see if our configure check for mumps needs > > > > > fixing. > > > > > > > > > > Wrt specifying scotch or metis - PETSc configure supports metis - but > > > > > not scotch. So --with-metis-lib/--with-metis-include should work - > > > > > but not --with-scotch-lib/--with-scotch-include > > > > > > > > > > But its fine to lump both into mumps options - as they are primarily > > > > > dependencies of mumps anyway. > > > > > > > > > > Satish > > > > > > > > > > On Tue, 19 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > Hello, > > > > > > Thanks a lot for your explanations. > > > > > > We have a "home-made" install of MUMPS, built with metis and scotch > > > > (and - > > > > > > at the moment- neither with parmetis nor with ptscotch). > > > > > > > > > > > > When I use --with-mumps-lib/--with-mumps-include, I am forced to > > > > specify > > > > > > scotch and metis libs inside mumps libs (otherwise the link of the > > > > check > > > > > > program dmumps_c fails). And this occurs even if I have already > > > > defined the > > > > > > location of scotch libraries through > > > > > > --with-scotch-lib/--with-scotch-include and > > > > > > --with-metis-lib/--with-metis-include options. > > > > > > Defining PETSc interfaces to scotch and metis packages is (at > > least for > > > > > > me) not sufficient : I have to specify these libraries locations > > in > > > > > > --with-mumps-lib. (as shown in the configure.log). > > > > > > > > > > > > This is not really a problem and doing so fixes the configure step. > > > > > > > > > > > > I agree that duplicating scotch to define ptscotch is weird. I did > > so > > > > > > because we do not have a parallel built of scotch. > > > > > > From PETSc point of view, MUMPS requires either Parmetis or > > PTScotch : > > > > so I > > > > > > provided ptscotch through --download-ptscotch option. > > > > > > I do not intend to use it, that is only for configuring purpose ! > > > > > > > > > > > > Natacha > > > > > > > > > > > > On Mon, Apr 18, 2016 at 5:52 PM, Satish Balay > > > > wrote: > > > > > > > > > > > > > Same with parmetis. > > > > > > > > > > > > > > On Mon, 18 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > > > Hello Satish, > > > > > > > > thank you very much for yor advices. They were very helpful ! > > > > > > > > > > > > > > > > The configure step finally succeeds if I use the following > > > > configure > > > > > > > line: > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > > --with-mpi=1 > > > > > > > > --with-debugging=0 --PETSC_ARCH=linux-metis-mumps > > > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > > > > > > > > > --with-metis-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > > -lmetis -lGKlib" > > > > > > > > > > > > > > > > > > > > > --with-metis-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/include > > > > > > > > > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > > -lmetis" > > > > > > > > > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > > > LIBS=-lgomp > > > > > > > > > > > > > > > > I have to specify scotch shared libraries (-lscotch -lscotcherr > > > > > > > > -lscotcherrexit ) and metis shared library in --with-mumps-lib > > > > option > > > > > > > > otherwise the test (on dmump) in configure fails. > > > > > > > > Is it OK to do so ? > > > > > > > > > > > > > > Hm - its best to avoid duplication of libraries. > > > > > > > > > > > > > > i.e specifying scotch via mumps-libs and also via > > > > --download-ptscotch will > > > > > > > cause problems. > > > > > > > > > > > > > > Why not specify scotch with > > > > --with-ptscotch-include,--with-ptscotch-lib > > > > > > > options? > > > > > > > > > > > > > > > > > > > > > > > I also tried to use the following shorter line > > > > > > > > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > > > > > > --with-mpi=1 --with-debugging=0 --PETSC_ARCH=linux-mumps- > > > > > > > > --with-scalapack-lib="-lscalapack-openmpi -lblacs-openmpi > > > > > > > > -lblacsF77init-openmpi -lblacsCinit-openmpi" > > > > > > > > > > > > > > > > > > > > > --with-mumps-lib="-L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/lib > > > > > > > > -lzmumps -ldmumps -lmumps_common -lpord > > > > > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Scotch_aster-604_aster1/lib > > > > > > > > -lesmumps -lscotch -lscotcherr -lscotcherrexit > > > > > > > > > > > > > > > > > > > > > -L/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Metis_aster-510_aster/lib > > > > > > > > -lmetis" > > > > > > > > > > > > > > > > > > > > > --with-mumps-include=/home/H03755/dev/codeaster-prerequisites/v13/prerequisites/Mumps-501_consortium_aster5/MPI/include > > > > > > > > --with-blas-lapack-lib="-llapack -lopenblas" > > > > > > > > --download-ptscotch=/home/H03755/Librairies/scotch_6.0.3.tar.gz > > > > > > > LIBS=-lgomp > > > > > > > > > > > > > > > > I do not use --with-metis-lib/--with-metis-include. > > > > > > > > I wonder if this is authorized : the libraries are given with > > the > > > > > > > > --with-mumps-lib, but the include are not defined. > > > > > > > > > > > > > > This is fine. [except for the dupliation of ptscotch] > > > > > > > > > > > > > > > > > > > > > > > What is the good practice ? > > > > > > > > > > > > > > Either is work - but its best to specify each package with its > > own > > > > > > > options listed by configure. The issues usually are: > > > > > > > > > > > > > > - Is this package primarily a depencency of an externalpakage or > > if > > > > there > > > > > > > is a petsc interface to it? > > > > > > > > > > > > > > For ex: PETSc has interface to mumps, parmetis - but not > > scalapack. > > > > So > > > > > > > if you club parmetis into mumps-libs then petsc interface to > > parmetis > > > > > > > would not be enabled. If you are not using this feature - it > > doesn't > > > > > > > matter if its not enabled. > > > > > > > > > > > > > > - does this externalpakage also require the includes in the > > public > > > > > > > interface? > > > > > > > > > > > > > > If dmumps_c.h requires metis.h [when mumps is built with metis] > > > > > > > - then you might have to specify metis include also with > > > > > > > --with-mumps-include. Otherwise - it doesn't matter. > > > > > > > > > > > > > > - are there bugs in petsc configure that can trigger wrong error > > > > checks? > > > > > > > > > > > > > > Because mumps depends on scalapack, and optionally on > > > > > > > metis,parmetis,ptscotch [i.e only one of them is required - but > > not > > > > > > > all] - there is an error-check in configure to make sure atleast > > one > > > > > > > of them is specified for --download-mumps. Perhaps this check > > should > > > > > > > not trigger error for user built mumps. > > > > > > > > > > > > > > Did you build MUMPS with both metis, ptscotch? [and not > > parmetis?] > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 6:07 PM, Satish Balay < > > balay at mcs.anl.gov> > > > > wrote: > > > > > > > > > > > > > > > > > you'll have to roll-in the --with-blacs-lib option into > > > > > > > > > --with-scalapack-lib option > > > > > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > On Thu, 14 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > > > > > > > > > > Sorry, do not take into account my last email. > > > > > > > > > > I made some progress and I am now able to configure PETSc > > > > with a > > > > > > > > > > pre-installed version of metis. > > > > > > > > > > > > > > > > > > > > Problems come when I try to configure PETSc with MUMPS > > > > > > > > > > > > > > > > > > > > My command line is > > > > > > > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx > > --with-fc=mpif90 > > > > > > > > > > --with-ssl=0 --with-mpi=1 --with-debugging=1 > > > > > > > > > > --PETSC_ARCH=linux-metis-mumps > > > > > > > > > > --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-blacs-lib=[/usr/lib/libblacs-openmpi.so,/usr/lib/libblacsCinit-openmpi.so,/usr/lib/libblacsF77init-openmpi.so] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-metis-lib=[${METIS_PRE}/lib/libmetis.a,${METIS_PRE}/lib/libGKlib.a] > > > > > > > > > > --with-metis-include=$METIS_PRE/include > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-mumps-lib=[$MUMPS_PRE/lib/libdmumps.a,$MUMPS_PRE/lib/libmumps_common.a,$MUMPS_PRE/lib/libpord.a] > > > > > > > > > > --with-mumps-include=$MUMPS_PRE/include > > > > > > > > > > > > > > > > > > > > where METIS_PRE and MUMPS_PRE are the path to the local > > > > installs of > > > > > > > metis > > > > > > > > > > and mumps) > > > > > > > > > > > > > > > > > > > > I get (at least) the following error > > > > > > > > > > /libdmumps.a(dend_driver.o): undefined reference to symbol > > > > > > > > > 'blacs_gridexit_' > > > > > > > > > > /usr/lib/libblacs-openmpi.so.1: error adding symbols: DSO > > > > missing > > > > > > > from > > > > > > > > > > command line > > > > > > > > > > collect2: error: ld returned 1 exit status > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Would you have any idea of its meaning ? > > > > > > > > > > > > > > > > > > > > The configure.log is attached > > > > > > > > > > Thanks a lot if you can help me ! > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > On Thu, Apr 14, 2016 at 5:19 PM, Natacha BEREUX < > > > > > > > > > natacha.bereux at gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Satish > > > > > > > > > > > thanks a lot for the answer. Unfortunately, it does not > > work > > > > yet. > > > > > > > > > > > More precisely : > > > > > > > > > > > --download-mumps works fine (and every > > --download-package > > > > option > > > > > > > works > > > > > > > > > > > perfectly). I am then able to compile a PETSc library. > > > > > > > > > > > --with-package-lib=/usr/lib/libscalapack-openmpi.so or > > more > > > > > > > generally > > > > > > > > > > > --with-package-lib=libXXXX.so also works > > > > > > > > > > > > > > > > > > > > > > But I would like to use static librairies, preinstalled > > on > > > > my > > > > > > > computer > > > > > > > > > > > ... and this fails. > > > > > > > > > > > > > > > > > > > > > > For the moment I gave up compiling with MUMPS, and I am > > > > instead > > > > > > > > > trying to > > > > > > > > > > > compile with Metis 5. > > > > > > > > > > > I have a preinstalled version in a some directory lets > > say > > > > > > > metis_dir > > > > > > > > > > > I try > > > > > > > > > > > -with-metis-lib=[metis_dir/lib/libmetis.a, > > > > > > > metis_dir/lib/libGKlib.a] > > > > > > > > > > > --with-metis-include=metis_dir/include > > > > > > > > > > > this fails (see the attached config.log) > > > > > > > > > > > -with-metis-dir=metis_dir also fails > > > > > > > > > > > Is there a problem with static librairies ? > > > > > > > > > > > > > > > > > > > > > > Natacha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 12, 2016 at 6:19 PM, Satish Balay < > > > > balay at mcs.anl.gov> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> On Tue, 12 Apr 2016, Natacha BEREUX wrote: > > > > > > > > > > >> > > > > > > > > > > >> > Hello, > > > > > > > > > > >> > I am trying to compile Petsc (3.6.3) with external > > > > packages > > > > > > > (MUMPS > > > > > > > > > and > > > > > > > > > > >> its > > > > > > > > > > >> > prerequisites). > > > > > > > > > > >> > More precisely I would like PETSc to use a > > pre-installed > > > > > > > version of > > > > > > > > > > >> MUMPS. > > > > > > > > > > >> > > > > > > > > > > > >> > Petsc downloads and compiles the prerequisites > > (parmetis, > > > > > > > scalapack > > > > > > > > > > >> etc) : > > > > > > > > > > >> > this works fine. > > > > > > > > > > >> > > > > > > > > > > >> What metis/parmetis/slcalapack is this MUMPS installed > > with? > > > > > > > > > > >> > > > > > > > > > > >> What version of MUMPS did you install? > > > > > > > > > > >> > > > > > > > > > > >> Why could you not use --download-mumps? > > > > > > > > > > >> > > > > > > > > > > >> Using a different metis/parmetis/slcalapack to install > > > > MUMPS - and > > > > > > > > > > >> then specifying --download-metis --download-parmetis > > > > > > > > > > >> --download-scalapack [i.e different versions/builds of > > the > > > > same > > > > > > > > > > >> libraries] can result in conflcits. > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > >> > I define MUMPS location by > > --with-mumps-dir=top-directory > > > > of > > > > > > > MUMPS > > > > > > > > > > >> install, > > > > > > > > > > >> > but the > > > > > > > > > > >> > configure step fails with the followiing message: > > > > > > > > > > >> > > > > > > > > > > > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see > > > > configure.log for > > > > > > > > > > >> details): > > > > > > > > > > >> > --with-mumps-dir=/home/H03755/Librairies/Mumps_MPI did > > > > not work > > > > > > > > > > >> > > > > > > > > > > > >> > I do not understand what is wrong. > > > > > > > > > > >> > I have attached the configure.log file. > > > > > > > > > > >> > > > > > > > > > > > >> > Any hint would be greatly appreciated ! > > > > > > > > > > >> > > > > > > > > > > >> >>>> > > > > > > > > > > >> Executing: mpicc -show > > > > > > > > > > >> stdout: gcc -I/usr/lib/openmpi/include > > > > > > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > > > > > >> -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl > > > > -lhwloc > > > > > > > > > > >> Defined make macro "MPICC_SHOW" to > > "gcc > > > > > > > > > > >> -I/usr/lib/openmpi/include > > > > -I/usr/lib/openmpi/include/openmpi > > > > > > > -pthread > > > > > > > > > > >> -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc" > > > > > > > > > > >> <<<< > > > > > > > > > > >> Ok - so you are using system openmpi with gcc. > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> >>>> > > > > > > > > > > >> Executing: mpicc -o > > > > /tmp/petsc-0u_4WI/config.libraries/conftest > > > > > > > > > -fPIC > > > > > > > > > > >> -Wall -Wwrite-strings -Wno-strict-aliasing > > > > -Wno-unknown-pragmas -O > > > > > > > > > > >> /tmp/petsc-0u_4WI/config.libraries/conftest.o > > > > > > > > > > >> -Wl,-rpath,/home/H03755/Librairies/Mumps_MPI/lib > > > > > > > > > > >> -L/home/H03755/Librairies/Mumps_MPI/lib -lcmumps > > -ldmumps > > > > -lsmumps > > > > > > > > > -lzmumps > > > > > > > > > > >> -lmumps_common -lpord > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > > >> > > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > -lscalapack > > > > > > > > > > >> -llapack -lblas -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > -L/usr/lib/openmpi/lib > > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran > > -lm > > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > > >> -lgfortran -lm -lquadmath -lm -llapack -lblas > > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran > > -lm > > > > > > > > > > >> -Wl,-rpath,/usr/lib/openmpi/lib > > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > > >> -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > > >> -lgfortran -lm -lquadmath -lm > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > > >> > > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > -lparmetis > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -Wl,-rpath,/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > > >> > > > > -L/home/H03755/Librairies/petsc-3.6.3/linux-debug-mumps-ext/lib > > > > > > > > > -lmetis -lm > > > > > > > > > > >> -lm -Wl,-rpath,/usr/lib/openmpi/lib > > -L/usr/lib/openmpi/lib > > > > > > > > > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > > >> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > > > > > > > > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu > > -Wl,-rpath,/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/lib/x86_64-linux-gnu > > -Wl,-rpath,/usr/lib/x86_64-linux-gnu > > > > > > > > > > >> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s > > > > -lpthread > > > > > > > -ldl > > > > > > > > > > >> Possible ERROR while running linker: exit code 256 > > > > > > > > > > >> stderr: > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function > > > > `__dmumps_lr_stats_MOD_update_flop_stats_lrb_product': > > > > > > > > > > >> dlr_stats.F:(.text+0x3079): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x30fa): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> dlr_stats.F:(.text+0x310e): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x318f): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function `__dmumps_lr_stats_MOD_update_flop_stats_trsm': > > > > > > > > > > >> dlr_stats.F:(.text+0x33a9): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x33f9): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> dlr_stats.F:(.text+0x340a): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x345a): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function > > `__dmumps_lr_stats_MOD_update_flop_stats_panel': > > > > > > > > > > >> dlr_stats.F:(.text+0x3576): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x35a7): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> dlr_stats.F:(.text+0x35b8): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x35e9): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function > > `__dmumps_lr_stats_MOD_update_flop_stats_demote': > > > > > > > > > > >> dlr_stats.F:(.text+0x36ac): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x36ce): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> dlr_stats.F:(.text+0x36df): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x3701): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function > > > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_demote': > > > > > > > > > > >> dlr_stats.F:(.text+0x37c1): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x37e3): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dlr_stats.o): In > > > > > > > > > > >> function > > > > `__dmumps_lr_stats_MOD_update_flop_stats_cb_promote': > > > > > > > > > > >> dlr_stats.F:(.text+0x3839): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_start' > > > > > > > > > > >> dlr_stats.F:(.text+0x3856): undefined reference to > > > > > > > > > > >> `GOMP_critical_name_end' > > > > > > > > > > >> > > > > /home/H03755/Librairies/Mumps_MPI/lib/libdmumps.a(dana_lr.o): In > > > > > > > > > function > > > > > > > > > > >> `__dmumps_ana_lr_MOD_mumps_scotch_kway': > > > > > > > > > > >> dana_lr.F:(.text+0x115): undefined reference to > > > > > > > `scotchfgraphbuild_' > > > > > > > > > > >> dana_lr.F:(.text+0x131): undefined reference to > > > > > > > `scotchfstratinit_' > > > > > > > > > > >> dana_lr.F:(.text+0x151): undefined reference to > > > > > > > `scotchfgraphpart_' > > > > > > > > > > >> dana_lr.F:(.text+0x15e): undefined reference to > > > > > > > `scotchfstratexit_' > > > > > > > > > > >> dana_lr.F:(.text+0x16b): undefined reference to > > > > > > > `scotchfgraphexit_' > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> Looks like this MUMPS is built with ptscotch and openmp. > > > > > > > > > > >> > > > > > > > > > > >> You can specify -lgomp for openmp. This can be done > > with the > > > > > > > configure > > > > > > > > > > >> option > > > > > > > > > > >> LIBS=-lgomp > > > > > > > > > > >> > > > > > > > > > > >> Wrt and PTSCOTCH stuff depending upon how its installed > > > > we'll > > > > > > > have to > > > > > > > > > > >> figureout > > > > > > > > > > >> how to specify it. > > > > > > > > > > >> > > > > > > > > > > >> Its best to specify all the packages [mumps and its > > > > dependencies] > > > > > > > you > > > > > > > > > > >> built manually with the options: > > > > > > > > > > >> > > > > > > > > > > >> --with-package-include --with-package-lib > > > > > > > > > > >> > > > > > > > > > > >> Satish > > > > > > > > > > >> > > > > > > > > > > >> > Best regards, > > > > > > > > > > >> > > > > > > > > > > > >> > Natacha > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From deng106 at purdue.edu Thu Apr 21 22:59:13 2016 From: deng106 at purdue.edu (Wei Deng) Date: Thu, 21 Apr 2016 23:59:13 -0400 Subject: [petsc-users] About the MatGetOrdering in Petsc Message-ID: <001001d19c4b$58a8c1f0$09fa45d0$@purdue.edu> Dear Sir/Madam, When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. Do you know why? The attachment is the code with which I have trouble. Thanks a lot. Wei Deng Purdue Uni. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: main_0.cpp URL: From deng106 at purdue.edu Fri Apr 22 08:36:54 2016 From: deng106 at purdue.edu (Wei Deng) Date: Fri, 22 Apr 2016 09:36:54 -0400 Subject: [petsc-users] =?gb2312?b?tPC4tDogQWJvdXQgdGhlIE1hdEdldE9yZGVyaW5n?= =?gb2312?b?IGluIFBldHNj?= Message-ID: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? ???: Wei Deng [mailto:deng106 at purdue.edu] ????: Thursday, April 21, 2016 11:59 PM ???: 'petsc-users at mcs.anl.gov' ??: About the MatGetOrdering in Petsc Dear Sir/Madam, When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. Do you know why? The attachment is the code with which I have trouble. Thanks a lot. Wei Deng Purdue Uni. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at rice.edu Fri Apr 22 10:23:29 2016 From: knepley at rice.edu (Matthew Knepley) Date: Fri, 22 Apr 2016 10:23:29 -0500 Subject: [petsc-users] Problem about the MatGetOrdering in Petsc In-Reply-To: <006b01d19c5b$ab74a380$025dea80$@purdue.edu> References: <006b01d19c5b$ab74a380$025dea80$@purdue.edu> Message-ID: On Fri, Apr 22, 2016 at 12:56 AM, Wei Deng wrote: > Dear Prof. Knepley, > > > > I am a student from Prof. Sameh?s class, I am now trying to use RCM > (Reverse Cuthill Mckee) to transform a sparse matrix into a banded one. > > > > In a test case which can be perfectly transformed into band in matlab, I > can?t achieve it in Petsc using the same data. The code can work, but I > don?t know why does this problem appear. Could you spend some to give me > some advice about this? > This is indeed RCM. I have cleaned up the code and taken snapshots of the matrices. I am not sure what you are using in Matlab, but here we are just calling SPARSEPAK: https://bitbucket.org/petsc/petsc/src/78026435dbba2cd6fc5f6af7e1f769fa9012b35f/src/mat/order/genrcm.c?at=master&fileviewer=file-view-default Thanks, Matt > Really appreciate about that. > > > > > > Thanks a lot. > > > > > > > > Wei Deng > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: main.c Type: text/x-csrc Size: 2292 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-04-22 at 10.20.31 AM.png Type: image/png Size: 22348 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-04-22 at 10.20.39 AM.png Type: image/png Size: 22501 bytes Desc: not available URL: From rlmackie862 at gmail.com Fri Apr 22 10:38:51 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Fri, 22 Apr 2016 08:38:51 -0700 Subject: [petsc-users] MatSolve_SeqAIJ_NaturalOrdering Message-ID: <18D8617D-D571-45C8-92D5-3499136716AB@gmail.com> After profiling our code, we have found that most of the time is spent in MatSolve_SeqAIJ_NaturalOrdering, which upon inspection is just doing simple forward and backward solves of already factored ILU matrices. We think that we should be able to see improvement by replacing these with optimized versions from Intel MKL (or other optimized BLAS). For example, Intel MKL has these routines: https://software.intel.com/en-us/node/468572 Is it possible to replace the PETSc triangular solves with a more optimized version? Thanks, Randy M. From jed at jedbrown.org Fri Apr 22 10:43:57 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Apr 2016 09:43:57 -0600 Subject: [petsc-users] MatSolve_SeqAIJ_NaturalOrdering In-Reply-To: <18D8617D-D571-45C8-92D5-3499136716AB@gmail.com> References: <18D8617D-D571-45C8-92D5-3499136716AB@gmail.com> Message-ID: <87twit4pj6.fsf@jedbrown.org> Randall Mackie writes: > After profiling our code, we have found that most of the time is spent in MatSolve_SeqAIJ_NaturalOrdering, which upon inspection is just doing simple forward and backward solves of already factored ILU matrices. > > We think that we should be able to see improvement by replacing these with optimized versions from Intel MKL (or other optimized BLAS). Doing so would actually give up memory optimizations that are only in PETSc. http://hpc.sagepub.com/content/25/4/386 (also http://www.mcs.anl.gov/uploads/cels/papers/P1658.pdf) Run with -log_summary and look at the rows for MatSolve versus MatMult. Is the flop/s rate (last column) close? If so, then you're already getting a high fraction of memory bandwidth. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Apr 22 11:29:30 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Apr 2016 11:29:30 -0500 Subject: [petsc-users] =?utf-8?b?562U5aSNOiBBYm91dCB0aGUgTWF0R2V0T3JkZXJp?= =?utf-8?q?ng_in_Petsc?= In-Reply-To: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> Message-ID: It can be applied to any graph but produces only a single permutation (that is it returns the same permutation for the rows and the columns even if the matrix is non-symmetric. From the source code /*****************************************************************/ /*****************************************************************/ /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ /*****************************************************************/ /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ /* BY CALLING THE SUBROUTINE RCM.*/ /* INPUT PARAMETERS -*/ /* NEQNS - NUMBER OF EQUATIONS*/ /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ /* OUTPUT PARAMETER -*/ /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ /* WORKING PARAMETERS -*/ /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ /* IS NUMBERED.*/ /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ /* PROGRAM SUBROUTINES -*/ /* FNROOT, RCM.*/ /*****************************************************************/ #undef __FUNCT__ > On Apr 22, 2016, at 8:36 AM, Wei Deng wrote: > > In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? > > ???: Wei Deng [mailto:deng106 at purdue.edu] > ????: Thursday, April 21, 2016 11:59 PM > ???: 'petsc-users at mcs.anl.gov' > ??: About the MatGetOrdering in Petsc > > Dear Sir/Madam, > > > When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. > > Do you know why? > > The attachment is the code with which I have trouble. > > > Thanks a lot. > > > Wei Deng > Purdue Uni. From bsmith at mcs.anl.gov Fri Apr 22 11:37:50 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Apr 2016 11:37:50 -0500 Subject: [petsc-users] =?utf-8?b?562U5aSNOiBBYm91dCB0aGUgTWF0R2V0T3JkZXJp?= =?utf-8?q?ng_in_Petsc?= In-Reply-To: <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> Message-ID: <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that these algorithms are for symmetric matrices. Perhaps MATLAB detects a non-symmetric matrix and does something different in that case. Barry > On Apr 22, 2016, at 11:32 AM, Wei Deng wrote: > > So that means that this RCM may be not sufficiently work for a non-symmetric matrix? > > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > ????: Friday, April 22, 2016 12:30 PM > ???: Wei Deng > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > It can be applied to any graph but produces only a single permutation (that is it returns the same permutation for the rows and the columns even if the matrix is non-symmetric. From the source code > > /*****************************************************************/ > /*****************************************************************/ > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > /*****************************************************************/ > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > /* BY CALLING THE SUBROUTINE RCM.*/ > > /* INPUT PARAMETERS -*/ > /* NEQNS - NUMBER OF EQUATIONS*/ > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > /* OUTPUT PARAMETER -*/ > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > /* WORKING PARAMETERS -*/ > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > /* IS NUMBERED.*/ > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > /* PROGRAM SUBROUTINES -*/ > /* FNROOT, RCM.*/ > /*****************************************************************/ > #undef __FUNCT__ > >> On Apr 22, 2016, at 8:36 AM, Wei Deng wrote: >> >> In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? >> >> ???: Wei Deng [mailto:deng106 at purdue.edu] >> ????: Thursday, April 21, 2016 11:59 PM >> ???: 'petsc-users at mcs.anl.gov' >> ??: About the MatGetOrdering in Petsc >> >> Dear Sir/Madam, >> >> >> When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. >> >> Do you know why? >> >> The attachment is the code with which I have trouble. >> >> >> Thanks a lot. >> >> >> Wei Deng >> Purdue Uni. > > From deng106 at purdue.edu Fri Apr 22 12:18:57 2016 From: deng106 at purdue.edu (Wei Deng) Date: Fri, 22 Apr 2016 13:18:57 -0400 Subject: [petsc-users] =?utf-8?b?562U5aSNOiAg562U5aSNOiBBYm91dCB0aGUgTWF0?= =?utf-8?q?GetOrdering_in_Petsc?= In-Reply-To: <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> Message-ID: <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> This is a symmetric matrix and use sysrcm in matlab transform this into banded, however, call rcm in petsc still fails. filename = 'C:\Users\Wei\Desktop\bcspwr03.mat'; load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) name = 'Zd_Jac6.mat' path = 'C:\Users\Wei\Documents\Courses\Parallelism Numerical Linear Algebra\CS515\matlab\' filename = strcat(path,name); load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) -----????----- ???: Barry Smith [mailto:bsmith at mcs.anl.gov] ????: Friday, April 22, 2016 12:38 PM ???: Wei Deng ; PETSc users list ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that these algorithms are for symmetric matrices. Perhaps MATLAB detects a non-symmetric matrix and does something different in that case. Barry > On Apr 22, 2016, at 11:32 AM, Wei Deng wrote: > > So that means that this RCM may be not sufficiently work for a non-symmetric matrix? > > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > ????: Friday, April 22, 2016 12:30 PM > ???: Wei Deng > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > It can be applied to any graph but produces only a single permutation (that is it returns the same permutation for the rows and the columns even if the matrix is non-symmetric. From the source code > > /*****************************************************************/ > /*****************************************************************/ > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > /*****************************************************************/ > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > /* BY CALLING THE SUBROUTINE RCM.*/ > > /* INPUT PARAMETERS -*/ > /* NEQNS - NUMBER OF EQUATIONS*/ > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > /* OUTPUT PARAMETER -*/ > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > /* WORKING PARAMETERS -*/ > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > /* IS NUMBERED.*/ > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > /* PROGRAM SUBROUTINES -*/ > /* FNROOT, RCM.*/ > /*****************************************************************/ > #undef __FUNCT__ > >> On Apr 22, 2016, at 8:36 AM, Wei Deng wrote: >> >> In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? >> >> ???: Wei Deng [mailto:deng106 at purdue.edu] >> ????: Thursday, April 21, 2016 11:59 PM >> ???: 'petsc-users at mcs.anl.gov' >> ??: About the MatGetOrdering in Petsc >> >> Dear Sir/Madam, >> >> >> When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. >> >> Do you know why? >> >> The attachment is the code with which I have trouble. >> >> >> Thanks a lot. >> >> >> Wei Deng >> Purdue Uni. > > -------------- next part -------------- A non-text attachment was scrubbed... Name: bcspwr03.mtx Type: application/octet-stream Size: 2446 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Capture.jpg Type: image/jpeg Size: 90097 bytes Desc: not available URL: From knepley at gmail.com Fri Apr 22 13:31:50 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Apr 2016 13:31:50 -0500 Subject: [petsc-users] =?utf-8?b?562U5aSNOiDnrZTlpI06IEFib3V0IHRoZSBNYXRH?= =?utf-8?q?etOrdering_in_Petsc?= In-Reply-To: <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> Message-ID: On Fri, Apr 22, 2016 at 12:18 PM, Wei Deng wrote: > This is a symmetric matrix and use sysrcm in matlab transform this into > banded, however, call rcm in petsc still fails. > Fails is not descriptive enough. Please send the screenshot. Matt > filename = 'C:\Users\Wei\Desktop\bcspwr03.mat'; > load(filename) > A=Problem.A; > p = symrcm(A); > R = A(p,p); > % change to dense > %B = full(R); > bandwidth(R) > subplot(1,2,2),spy(A),title(name) > subplot(1,2,2),spy(R),title(name) > > name = 'Zd_Jac6.mat' > path = 'C:\Users\Wei\Documents\Courses\Parallelism Numerical Linear > Algebra\CS515\matlab\' > filename = strcat(path,name); > load(filename) > A=Problem.A; > p = symrcm(A); > R = A(p,p); > % change to dense > %B = full(R); > bandwidth(R) > subplot(1,2,2),spy(A),title(name) > subplot(1,2,2),spy(R),title(name) > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > ????: Friday, April 22, 2016 12:38 PM > ???: Wei Deng ; PETSc users list < > petsc-users at mcs.anl.gov> > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that > these algorithms are for symmetric matrices. > > Perhaps MATLAB detects a non-symmetric matrix and does something different > in that case. > > Barry > > > On Apr 22, 2016, at 11:32 AM, Wei Deng wrote: > > > > So that means that this RCM may be not sufficiently work for a > non-symmetric matrix? > > > > > > -----????----- > > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > > ????: Friday, April 22, 2016 12:30 PM > > ???: Wei Deng > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > > > > It can be applied to any graph but produces only a single permutation > (that is it returns the same permutation for the rows and the columns even > if the matrix is non-symmetric. From the source code > > > > /*****************************************************************/ > > /*****************************************************************/ > > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > > /*****************************************************************/ > > > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > > /* BY CALLING THE SUBROUTINE RCM.*/ > > > > /* INPUT PARAMETERS -*/ > > /* NEQNS - NUMBER OF EQUATIONS*/ > > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > > > /* OUTPUT PARAMETER -*/ > > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > > > /* WORKING PARAMETERS -*/ > > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > > /* IS NUMBERED.*/ > > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > > > /* PROGRAM SUBROUTINES -*/ > > /* FNROOT, RCM.*/ > > /*****************************************************************/ > > #undef __FUNCT__ > > > >> On Apr 22, 2016, at 8:36 AM, Wei Deng wrote: > >> > >> In other words, does the function MatGetOrdering only support symmetric > matrix? It can?t work well in other cases? > >> > >> ???: Wei Deng [mailto:deng106 at purdue.edu] > >> ????: Thursday, April 21, 2016 11:59 PM > >> ???: 'petsc-users at mcs.anl.gov' > >> ??: About the MatGetOrdering in Petsc > >> > >> Dear Sir/Madam, > >> > >> > >> When I was trying to change a sparse matrix into a banded one, I can > use RCM in matlab to change it perfectly. But this does not work in Petsc, > most points were moved to the diagonal position. However, there are still > some points left, leading the width unacceptable. > >> > >> Do you know why? > >> > >> The attachment is the code with which I have trouble. > >> > >> > >> Thanks a lot. > >> > >> > >> Wei Deng > >> Purdue Uni. > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From chengj5 at rpi.edu Fri Apr 22 16:09:47 2016 From: chengj5 at rpi.edu (Jie Cheng) Date: Fri, 22 Apr 2016 17:09:47 -0400 Subject: [petsc-users] Krylov Method Takes Too Long to Solve Message-ID: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> Hi I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: ksp KSP and PC on KSP preconditioner follow --------------------------------- KSP Object: (ksp_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (ksp_) 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=16906, cols=16906 package used to perform factorization: petsc total: nonzeros=988540, allocated nonzeros=988540 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 7582 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=16906, cols=16906 total: nonzeros=988540, allocated nonzeros=988540 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 7582 nodes, limit used is 5 --------------------------------- linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=16906, cols=16906 total: nonzeros=988540, allocated nonzeros=988540 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 7582 nodes, limit used is 5 Could anyone give me any suggestion please? Thanks Jie Cheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Apr 22 16:12:15 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Apr 2016 15:12:15 -0600 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> Message-ID: <874mat4ac0.fsf@jedbrown.org> Always send the output with -log_summary when asking about performance. Jie Cheng writes: > Hi > > I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. > > The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. > > A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ksp > KSP and PC on KSP preconditioner follow > --------------------------------- > KSP Object: (ksp_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (ksp_) 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > package used to perform factorization: petsc > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > --------------------------------- > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > > Could anyone give me any suggestion please? > > Thanks > Jie Cheng -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Apr 22 16:25:51 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Apr 2016 16:25:51 -0500 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> Message-ID: <3D8FE75E-341A-4F1A-BF17-5F0121DD11D7@mcs.anl.gov> How large are the problems you really want to solve? This is a pretty small problem for iterative methods, direct solver may be faster. ILU/ICC methods are not particularly good for solid mechanics and probably terrible for mix formations (if I understand what you mean by mixed formulation). Barry > On Apr 22, 2016, at 4:09 PM, Jie Cheng wrote: > > Hi > > I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. > > The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. > > A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ksp > KSP and PC on KSP preconditioner follow > --------------------------------- > KSP Object: (ksp_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (ksp_) 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > package used to perform factorization: petsc > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > --------------------------------- > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > > Could anyone give me any suggestion please? > > Thanks > Jie Cheng From jed at jedbrown.org Fri Apr 22 17:30:32 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Apr 2016 16:30:32 -0600 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <4D9A6276-408E-4EA6-9E9B-F5104D99C8AF@rpi.edu> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> <874mat4ac0.fsf@jedbrown.org> <4D9A6276-408E-4EA6-9E9B-F5104D99C8AF@rpi.edu> Message-ID: <87shyd2s53.fsf@jedbrown.org> Allow me to call your attention to some fine print: ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Jie Cheng writes: > Hello Jed > > Thanks for your reply. Here is the log: (the first a couple of lines are printed by my own code during a time step, ?iteration? means the number of Newton?s iteration, ?GMRES_iteration? denotes the number of GMRES iterations it takes in a particular Newton?s iteration, err1 and err2 are my criteria of convergence) > > > Number of nonzeros allocated: 988540 > Step = 1, LoadFactor = 1.0000e-02 > REASON: 2 > Iteration = 1 GMRES_Iteration = 2 Err1 = 1.0000e+00, Err2 = 1.8375e-11 > REASON: 2 > Iteration = 2 GMRES_Iteration = 4 Err1 = 4.1151e-02, Err2 = 5.9467e-11 > REASON: 2 > Iteration = 3 GMRES_Iteration = 2 Err1 = 1.0265e-02, Err2 = 2.1268e-12 > REASON: 2 > Iteration = 4 GMRES_Iteration = 2 Err1 = 8.9824e-04, Err2 = 1.2622e-14 > REASON: 2 > Iteration = 5 GMRES_Iteration = 2 Err1 = 1.5741e-04, Err2 = 8.1248e-16 > REASON: 2 > Iteration = 6 GMRES_Iteration = 2 Err1 = 7.1515e-06, Err2 = 1.6605e-16 > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > ./main on a arch-darwin-c-debug named jiecheng.local with 1 processor, by Jie Fri Apr 22 18:06:33 2016 > Using Petsc Release Version 3.6.3, Dec, 03, 2015 > > Max Max/Min Avg Total > Time (sec): 4.119e+02 1.00000 4.119e+02 > Objects: 6.800e+01 1.00000 6.800e+01 > Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 > Flops/sec: 6.287e+08 1.00000 6.287e+08 6.287e+08 > Memory: 3.308e+07 1.00000 3.308e+07 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 4.1191e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecMDot 49906 1.0 2.1200e+01 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 10 0 0 0 5 10 0 0 0 1231 > VecNorm 51599 1.0 1.5052e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1159 > VecScale 51587 1.0 5.6397e+00 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 155 > VecCopy 1682 1.0 5.0184e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 1781 1.0 3.5445e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 3324 1.0 4.0115e-01 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 280 > VecAYPX 12 1.0 1.0931e-03 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 371 > VecMAXPY 51581 1.0 2.4876e+01 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 11 0 0 0 6 11 0 0 0 1117 > VecAssemblyBegin 18 1.0 1.6809e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 18 1.0 1.2112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNormalize 51581 1.0 8.1458e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 321 > MatMult 51555 1.0 1.5226e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 37 39 0 0 0 37 39 0 0 0 664 > MatSolve 51561 1.0 1.7415e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 42 39 0 0 0 42 39 0 0 0 580 > MatLUFactorNum 6 1.0 6.3280e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 353 > MatILUFactorSym 1 1.0 2.1535e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 6 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 6 1.0 1.7512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 1 1.0 6.8307e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 1 1.0 1.3239e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 6 1.0 9.4421e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPGMRESOrthog 49906 1.0 4.6091e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 0.0e+00 11 20 0 0 0 11 20 0 0 0 1133 > KSPSetUp 7 1.0 3.3112e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 6 1.0 3.8880e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 666 > PCSetUp 6 1.0 6.8447e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 326 > PCApply 20 1.0 3.8806e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 667 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Vector 56 56 7659456 0 > Matrix 2 2 24271340 0 > Krylov Solver 2 2 36720 0 > Preconditioner 2 2 1832 0 > Index Set 5 5 237080 0 > Viewer 1 0 0 0 > ======================================================================================================================== > Average time to get PetscTime(): 9.53674e-08 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: --with-mpi-dir=/usr/local > ----------------------------------------- > Libraries compiled on Tue Mar 22 14:05:36 2016 on calcium-68.dynamic2.rpi.edu > Machine characteristics: Darwin-15.4.0-x86_64-i386-64bit > Using PETSc directory: /usr/local/petsc/petsc-3.6.3 > Using PETSc arch: arch-darwin-c-debug > ----------------------------------------- > > Using C compiler: /usr/local/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 ${COPTFLAGS} ${CFLAGS} > Using Fortran compiler: /usr/local/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 ${FOPTFLAGS} ${FFLAGS} > ----------------------------------------- > > Using include paths: -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/include > ----------------------------------------- > > Using C linker: /usr/local/bin/mpicc > Using Fortran linker: /usr/local/bin/mpif90 > Using libraries: -Wl,-rpath,/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -L/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -lpetsc -llapack -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -Wl,-rpath,/usr/local/gfortran/lib -L/usr/local/gfortran/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/usr/local/lib -L/usr/local/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -ldl > ----------------------------------------- > >> On Apr 22, 2016, at 5:12 PM, Jed Brown wrote: >> >> Always send the output with -log_summary when asking about performance. >> >> Jie Cheng writes: >> >>> Hi >>> >>> I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. >>> >>> The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. >>> >>> A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: >>> >>> KSP Object: 1 MPI processes >>> type: gmres >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>> GMRES: happy breakdown tolerance 1e-30 >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object: 1 MPI processes >>> type: ksp >>> KSP and PC on KSP preconditioner follow >>> --------------------------------- >>> KSP Object: (ksp_) 1 MPI processes >>> type: gmres >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>> GMRES: happy breakdown tolerance 1e-30 >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (ksp_) 1 MPI processes >>> type: ilu >>> ILU: out-of-place factorization >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=16906, cols=16906 >>> package used to perform factorization: petsc >>> total: nonzeros=988540, allocated nonzeros=988540 >>> total number of mallocs used during MatSetValues calls =0 >>> using I-node routines: found 7582 nodes, limit used is 5 >>> linear system matrix = precond matrix: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=16906, cols=16906 >>> total: nonzeros=988540, allocated nonzeros=988540 >>> total number of mallocs used during MatSetValues calls =0 >>> using I-node routines: found 7582 nodes, limit used is 5 >>> --------------------------------- >>> linear system matrix = precond matrix: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=16906, cols=16906 >>> total: nonzeros=988540, allocated nonzeros=988540 >>> total number of mallocs used during MatSetValues calls =0 >>> using I-node routines: found 7582 nodes, limit used is 5 >>> >>> Could anyone give me any suggestion please? >>> >>> Thanks >>> Jie Cheng -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From deng106 at purdue.edu Fri Apr 22 20:03:39 2016 From: deng106 at purdue.edu (Wei Deng) Date: Fri, 22 Apr 2016 21:03:39 -0400 Subject: [petsc-users] =?utf-8?b?562U5aSNOiAg562U5aSNOiDnrZTlpI06IEFib3V0?= =?utf-8?q?_the_MatGetOrdering_in_Petsc?= In-Reply-To: References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> Message-ID: <003301d19cfb$fcc5a270$f650e750$@purdue.edu> There is no bug, as the figure shows, rcm in matlab works well for symmetric matrix, however, petsc rcm has some bad points, like (117,9) ???: Matthew Knepley [mailto:knepley at gmail.com] ????: Friday, April 22, 2016 2:32 PM ???: Wei Deng ??: Barry Smith ; PETSc users list ??: Re: [petsc-users] ??: ??: About the MatGetOrdering in Petsc On Fri, Apr 22, 2016 at 12:18 PM, Wei Deng > wrote: This is a symmetric matrix and use sysrcm in matlab transform this into banded, however, call rcm in petsc still fails. Fails is not descriptive enough. Please send the screenshot. Matt filename = 'C:\Users\Wei\Desktop\bcspwr03.mat'; load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) name = 'Zd_Jac6.mat' path = 'C:\Users\Wei\Documents\Courses\Parallelism Numerical Linear Algebra\CS515\matlab\' filename = strcat(path,name); load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) -----????----- ???: Barry Smith [mailto:bsmith at mcs.anl.gov ] ????: Friday, April 22, 2016 12:38 PM ???: Wei Deng >; PETSc users list > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that these algorithms are for symmetric matrices. Perhaps MATLAB detects a non-symmetric matrix and does something different in that case. Barry > On Apr 22, 2016, at 11:32 AM, Wei Deng > wrote: > > So that means that this RCM may be not sufficiently work for a non-symmetric matrix? > > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov ] > ????: Friday, April 22, 2016 12:30 PM > ???: Wei Deng > > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > It can be applied to any graph but produces only a single permutation (that is it returns the same permutation for the rows and the columns even if the matrix is non-symmetric. From the source code > > /*****************************************************************/ > /*****************************************************************/ > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > /*****************************************************************/ > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > /* BY CALLING THE SUBROUTINE RCM.*/ > > /* INPUT PARAMETERS -*/ > /* NEQNS - NUMBER OF EQUATIONS*/ > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > /* OUTPUT PARAMETER -*/ > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > /* WORKING PARAMETERS -*/ > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > /* IS NUMBERED.*/ > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > /* PROGRAM SUBROUTINES -*/ > /* FNROOT, RCM.*/ > /*****************************************************************/ > #undef __FUNCT__ > >> On Apr 22, 2016, at 8:36 AM, Wei Deng > wrote: >> >> In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? >> >> ???: Wei Deng [mailto:deng106 at purdue.edu ] >> ????: Thursday, April 21, 2016 11:59 PM >> ???: 'petsc-users at mcs.anl.gov ' > >> ??: About the MatGetOrdering in Petsc >> >> Dear Sir/Madam, >> >> >> When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. >> >> Do you know why? >> >> The attachment is the code with which I have trouble. >> >> >> Thanks a lot. >> >> >> Wei Deng >> Purdue Uni. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Matlab result.jpg Type: image/jpeg Size: 90097 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSC RCM result.jpg Type: image/jpeg Size: 127155 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bcspwr03.mat Type: application/octet-stream Size: 1314 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: main_0.cpp URL: From chengj5 at rpi.edu Fri Apr 22 22:50:18 2016 From: chengj5 at rpi.edu (Jie Cheng) Date: Fri, 22 Apr 2016 23:50:18 -0400 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <87shyd2s53.fsf@jedbrown.org> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> <874mat4ac0.fsf@jedbrown.org> <4D9A6276-408E-4EA6-9E9B-F5104D99C8AF@rpi.edu> <87shyd2s53.fsf@jedbrown.org> Message-ID: <32396333-AB5A-4F7A-96A4-4EAF5B38B3D7@rpi.edu> Hi Jed and Barry Thanks for your help. After I reconfigured PETSc without debugging, it did become much faster. But still not fast enough. The problem remains: what could be the factor that slows the solver down? In this example, only 4662 nodes and 2920 elements are used. Typically I use 3D hexahedral element, the number of degrees of freedom in the mixed formulation I use is 3*4662+2920 = 16906. At this point, my code runs only in serial. I am planning to parallelize it and then work with much finer mesh in the future. If as Barry said, the number of degrees of freedom is too small for iterative solvers, then which direct solver do you recommend to use? Here is the log when I ran without debugging. It is not fast enough because this is only 1 step, and there are 99 more steps. Thank you. Jie ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./main on a arch-darwin-c-opt named jiecheng.local with 1 processor, by Jie Fri Apr 22 22:36:38 2016 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 1.520e+02 1.00000 1.520e+02 Objects: 6.800e+01 1.00000 6.800e+01 Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 Flops/sec: 1.704e+09 1.00000 1.704e+09 1.704e+09 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.5195e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecMDot 49906 1.0 8.4904e+00 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 3074 VecNorm 51599 1.0 1.0644e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1639 VecScale 51587 1.0 4.5893e-01 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1900 VecCopy 1682 1.0 2.9097e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1781 1.0 1.9067e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 3324 1.0 5.8797e-02 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1911 VecAYPX 12 1.0 3.0708e-04 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1321 VecMAXPY 51581 1.0 7.2805e+00 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 11 0 0 0 5 11 0 0 0 3817 VecAssemblyBegin 18 1.0 4.0531e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 18 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 51581 1.0 1.5588e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1678 MatMult 51555 1.0 6.2301e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 41 39 0 0 0 41 39 0 0 0 1622 MatSolve 51561 1.0 5.8966e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 39 39 0 0 0 39 39 0 0 0 1714 MatLUFactorNum 6 1.0 1.4240e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1569 MatILUFactorSym 1 1.0 1.0796e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 6 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 6 1.0 6.2988e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.9605e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 5.3310e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 6 1.0 5.8441e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 49906 1.0 1.5348e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 0.0e+00 10 20 0 0 0 10 20 0 0 0 3401 KSPSetUp 7 1.0 1.5509e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 6 1.0 1.3919e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 92100 0 0 0 92100 0 0 0 1861 PCSetUp 6 1.0 1.5490e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1442 PCApply 20 1.0 1.3901e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 91100 0 0 0 91100 0 0 0 1861 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 56 56 7659456 0 Matrix 2 2 24271340 0 Krylov Solver 2 2 36720 0 Preconditioner 2 2 1832 0 Index Set 5 5 237080 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-debugging=no > On Apr 22, 2016, at 6:30 PM, Jed Brown wrote: > > Allow me to call your attention to some fine print: > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > Please always use "reply-all" so that your messages go to the list. > This is standard mailing list etiquette. It is important to preserve > threading for people who find this discussion later and so that we do > not waste our time re-answering the same questions that have already > been answered in private side-conversations. You'll likely get an > answer faster that way too. > > Jie Cheng writes: > >> Hello Jed >> >> Thanks for your reply. Here is the log: (the first a couple of lines are printed by my own code during a time step, ?iteration? means the number of Newton?s iteration, ?GMRES_iteration? denotes the number of GMRES iterations it takes in a particular Newton?s iteration, err1 and err2 are my criteria of convergence) >> >> >> Number of nonzeros allocated: 988540 >> Step = 1, LoadFactor = 1.0000e-02 >> REASON: 2 >> Iteration = 1 GMRES_Iteration = 2 Err1 = 1.0000e+00, Err2 = 1.8375e-11 >> REASON: 2 >> Iteration = 2 GMRES_Iteration = 4 Err1 = 4.1151e-02, Err2 = 5.9467e-11 >> REASON: 2 >> Iteration = 3 GMRES_Iteration = 2 Err1 = 1.0265e-02, Err2 = 2.1268e-12 >> REASON: 2 >> Iteration = 4 GMRES_Iteration = 2 Err1 = 8.9824e-04, Err2 = 1.2622e-14 >> REASON: 2 >> Iteration = 5 GMRES_Iteration = 2 Err1 = 1.5741e-04, Err2 = 8.1248e-16 >> REASON: 2 >> Iteration = 6 GMRES_Iteration = 2 Err1 = 7.1515e-06, Err2 = 1.6605e-16 >> ************************************************************************************************************************ >> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** >> ************************************************************************************************************************ >> >> ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- >> >> ./main on a arch-darwin-c-debug named jiecheng.local with 1 processor, by Jie Fri Apr 22 18:06:33 2016 >> Using Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> Max Max/Min Avg Total >> Time (sec): 4.119e+02 1.00000 4.119e+02 >> Objects: 6.800e+01 1.00000 6.800e+01 >> Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 >> Flops/sec: 6.287e+08 1.00000 6.287e+08 6.287e+08 >> Memory: 3.308e+07 1.00000 3.308e+07 >> MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 >> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 >> MPI Reductions: 0.000e+00 0.00000 >> >> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) >> e.g., VecAXPY() for real vectors of length N --> 2N flops >> and VecAXPY() for complex vectors of length N --> 8N flops >> >> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg %Total counts %Total Avg %Total counts %Total >> 0: Main Stage: 4.1191e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >> >> ------------------------------------------------------------------------------------------------------------------------ >> See the 'Profiling' chapter of the users' manual for details on interpreting output. >> Phase summary info: >> Count: number of times phase was executed >> Time and Flops: Max - maximum over all processors >> Ratio - ratio of maximum to minimum over all processors >> Mess: number of messages sent >> Avg. len: average message length (bytes) >> Reduct: number of global reductions >> Global: entire computation >> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). >> %T - percent time in this phase %F - percent flops in this phase >> %M - percent messages in this phase %L - percent message lengths in this phase >> %R - percent reductions in this phase >> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) >> ------------------------------------------------------------------------------------------------------------------------ >> >> >> ########################################################## >> # # >> # WARNING!!! # >> # # >> # This code was compiled with a debugging option, # >> # To get timing results run ./configure # >> # using --with-debugging=no, the performance will # >> # be generally two or three times faster. # >> # # >> ########################################################## >> >> >> Event Count Time (sec) Flops --- Global --- --- Stage --- Total >> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> ------------------------------------------------------------------------------------------------------------------------ >> >> --- Event Stage 0: Main Stage >> >> VecMDot 49906 1.0 2.1200e+01 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 10 0 0 0 5 10 0 0 0 1231 >> VecNorm 51599 1.0 1.5052e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1159 >> VecScale 51587 1.0 5.6397e+00 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 155 >> VecCopy 1682 1.0 5.0184e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecSet 1781 1.0 3.5445e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAXPY 3324 1.0 4.0115e-01 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 280 >> VecAYPX 12 1.0 1.0931e-03 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 371 >> VecMAXPY 51581 1.0 2.4876e+01 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 11 0 0 0 6 11 0 0 0 1117 >> VecAssemblyBegin 18 1.0 1.6809e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyEnd 18 1.0 1.2112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecNormalize 51581 1.0 8.1458e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 321 >> MatMult 51555 1.0 1.5226e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 37 39 0 0 0 37 39 0 0 0 664 >> MatSolve 51561 1.0 1.7415e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 42 39 0 0 0 42 39 0 0 0 580 >> MatLUFactorNum 6 1.0 6.3280e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 353 >> MatILUFactorSym 1 1.0 2.1535e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 6 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 6 1.0 1.7512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatGetRowIJ 1 1.0 6.8307e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatGetOrdering 1 1.0 1.3239e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatZeroEntries 6 1.0 9.4421e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> KSPGMRESOrthog 49906 1.0 4.6091e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 0.0e+00 11 20 0 0 0 11 20 0 0 0 1133 >> KSPSetUp 7 1.0 3.3112e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> KSPSolve 6 1.0 3.8880e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 666 >> PCSetUp 6 1.0 6.8447e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 326 >> PCApply 20 1.0 3.8806e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 667 >> ------------------------------------------------------------------------------------------------------------------------ >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' Mem. >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> >> Vector 56 56 7659456 0 >> Matrix 2 2 24271340 0 >> Krylov Solver 2 2 36720 0 >> Preconditioner 2 2 1832 0 >> Index Set 5 5 237080 0 >> Viewer 1 0 0 0 >> ======================================================================================================================== >> Average time to get PetscTime(): 9.53674e-08 >> #PETSc Option Table entries: >> -log_summary >> #End of PETSc Option Table entries >> Compiled without FORTRAN kernels >> Compiled with full precision matrices (default) >> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >> Configure options: --with-mpi-dir=/usr/local >> ----------------------------------------- >> Libraries compiled on Tue Mar 22 14:05:36 2016 on calcium-68.dynamic2.rpi.edu >> Machine characteristics: Darwin-15.4.0-x86_64-i386-64bit >> Using PETSc directory: /usr/local/petsc/petsc-3.6.3 >> Using PETSc arch: arch-darwin-c-debug >> ----------------------------------------- >> >> Using C compiler: /usr/local/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 ${COPTFLAGS} ${CFLAGS} >> Using Fortran compiler: /usr/local/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 ${FOPTFLAGS} ${FFLAGS} >> ----------------------------------------- >> >> Using include paths: -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/include >> ----------------------------------------- >> >> Using C linker: /usr/local/bin/mpicc >> Using Fortran linker: /usr/local/bin/mpif90 >> Using libraries: -Wl,-rpath,/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -L/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -lpetsc -llapack -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -Wl,-rpath,/usr/local/gfortran/lib -L/usr/local/gfortran/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/usr/local/lib -L/usr/local/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -ldl >> ----------------------------------------- >> >>> On Apr 22, 2016, at 5:12 PM, Jed Brown wrote: >>> >>> Always send the output with -log_summary when asking about performance. >>> >>> Jie Cheng writes: >>> >>>> Hi >>>> >>>> I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. >>>> >>>> The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. >>>> >>>> A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: >>>> >>>> KSP Object: 1 MPI processes >>>> type: gmres >>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>>> GMRES: happy breakdown tolerance 1e-30 >>>> maximum iterations=10000, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: 1 MPI processes >>>> type: ksp >>>> KSP and PC on KSP preconditioner follow >>>> --------------------------------- >>>> KSP Object: (ksp_) 1 MPI processes >>>> type: gmres >>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>>> GMRES: happy breakdown tolerance 1e-30 >>>> maximum iterations=10000, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: (ksp_) 1 MPI processes >>>> type: ilu >>>> ILU: out-of-place factorization >>>> 0 levels of fill >>>> tolerance for zero pivot 2.22045e-14 >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Mat Object: 1 MPI processes >>>> type: seqaij >>>> rows=16906, cols=16906 >>>> package used to perform factorization: petsc >>>> total: nonzeros=988540, allocated nonzeros=988540 >>>> total number of mallocs used during MatSetValues calls =0 >>>> using I-node routines: found 7582 nodes, limit used is 5 >>>> linear system matrix = precond matrix: >>>> Mat Object: 1 MPI processes >>>> type: seqaij >>>> rows=16906, cols=16906 >>>> total: nonzeros=988540, allocated nonzeros=988540 >>>> total number of mallocs used during MatSetValues calls =0 >>>> using I-node routines: found 7582 nodes, limit used is 5 >>>> --------------------------------- >>>> linear system matrix = precond matrix: >>>> Mat Object: 1 MPI processes >>>> type: seqaij >>>> rows=16906, cols=16906 >>>> total: nonzeros=988540, allocated nonzeros=988540 >>>> total number of mallocs used during MatSetValues calls =0 >>>> using I-node routines: found 7582 nodes, limit used is 5 >>>> >>>> Could anyone give me any suggestion please? >>>> >>>> Thanks >>>> Jie Cheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 22 22:56:17 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Apr 2016 22:56:17 -0500 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <32396333-AB5A-4F7A-96A4-4EAF5B38B3D7@rpi.edu> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> <874mat4ac0.fsf@jedbrown.org> <4D9A6276-408E-4EA6-9E9B-F5104D99C8AF@rpi.edu> <87shyd2s53.fsf@jedbrown.org> <32396333-AB5A-4F7A-96A4-4EAF5B38B3D7@rpi.edu> Message-ID: On Fri, Apr 22, 2016 at 10:50 PM, Jie Cheng wrote: > Hi Jed and Barry > > Thanks for your help. After I reconfigured PETSc without debugging, it did > become much faster. But still not fast enough. The problem remains: what > could be the factor that slows the solver down? In this example, only 4662 > nodes and 2920 elements are used. Typically I use 3D hexahedral element, > the number of degrees of freedom in the mixed formulation I use is > 3*4662+2920 = 16906. At this point, my code runs only in serial. I am > planning to parallelize it and then work with much finer mesh in the > future. If as Barry said, the number of degrees of freedom is too small for > iterative solvers, then which direct solver do you recommend to use? > SuperLU or MUMPS. Iterative solvers usually do not win until > 50K unknowns or maybe 100K depending on your system. Also, the solver you showed last time made no sense. You had GMRES/ILU(0) preconditioning GMRES. You should remove the outer solver, and use a better preconditioner since iLU(0) is quite weak. Thanks, Matt > Here is the log when I ran without debugging. It is not fast enough > because this is only 1 step, and there are 99 more steps. > > Thank you. > Jie > > ---------------------------------------------- PETSc Performance Summary: > ---------------------------------------------- > > ./main on a arch-darwin-c-opt named jiecheng.local with 1 processor, by > Jie Fri Apr 22 22:36:38 2016 > Using Petsc Release Version 3.6.3, Dec, 03, 2015 > > Max Max/Min Avg Total > Time (sec): 1.520e+02 1.00000 1.520e+02 > Objects: 6.800e+01 1.00000 6.800e+01 > Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 > Flops/sec: 1.704e+09 1.00000 1.704e+09 1.704e+09 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N > --> 2N flops > and VecAXPY() for complex vectors of length N > --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 1.5195e+02 100.0% 2.5896e+11 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and > PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this > phase > %M - percent messages in this phase %L - percent message lengths > in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time > over all processors) > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecMDot 49906 1.0 8.4904e+00 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 6 10 0 0 0 6 10 0 0 0 3074 > VecNorm 51599 1.0 1.0644e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 1 1 0 0 0 1 1 0 0 0 1639 > VecScale 51587 1.0 4.5893e-01 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1900 > VecCopy 1682 1.0 2.9097e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 1781 1.0 1.9067e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 3324 1.0 5.8797e-02 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1911 > VecAYPX 12 1.0 3.0708e-04 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1321 > VecMAXPY 51581 1.0 7.2805e+00 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 5 11 0 0 0 5 11 0 0 0 3817 > VecAssemblyBegin 18 1.0 4.0531e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 18 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNormalize 51581 1.0 1.5588e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 1 1 0 0 0 1 1 0 0 0 1678 > MatMult 51555 1.0 6.2301e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 41 39 0 0 0 41 39 0 0 0 1622 > MatSolve 51561 1.0 5.8966e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 39 39 0 0 0 39 39 0 0 0 1714 > MatLUFactorNum 6 1.0 1.4240e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1569 > MatILUFactorSym 1 1.0 1.0796e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 6 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 6 1.0 6.2988e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 1 1.0 5.9605e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 1 1.0 5.3310e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 6 1.0 5.8441e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPGMRESOrthog 49906 1.0 1.5348e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 10 20 0 0 0 10 20 0 0 0 3401 > KSPSetUp 7 1.0 1.5509e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 6 1.0 1.3919e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 92100 0 0 0 92100 0 0 0 1861 > PCSetUp 6 1.0 1.5490e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1442 > PCApply 20 1.0 1.3901e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 91100 0 0 0 91100 0 0 0 1861 > > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Vector 56 56 7659456 0 > Matrix 2 2 24271340 0 > Krylov Solver 2 2 36720 0 > Preconditioner 2 2 1832 0 > Index Set 5 5 237080 0 > Viewer 1 0 0 0 > > ======================================================================================================================== > Average time to get PetscTime(): 2.14577e-07 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: --with-debugging=no > > On Apr 22, 2016, at 6:30 PM, Jed Brown wrote: > > Allow me to call your attention to some fine print: > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > Please always use "reply-all" so that your messages go to the list. > This is standard mailing list etiquette. It is important to preserve > threading for people who find this discussion later and so that we do > not waste our time re-answering the same questions that have already > been answered in private side-conversations. You'll likely get an > answer faster that way too. > > Jie Cheng writes: > > Hello Jed > > Thanks for your reply. Here is the log: (the first a couple of lines are > printed by my own code during a time step, ?iteration? means the number of > Newton?s iteration, ?GMRES_iteration? denotes the number of GMRES > iterations it takes in a particular Newton?s iteration, err1 and err2 are > my criteria of convergence) > > > Number of nonzeros allocated: 988540 > Step = 1, LoadFactor = 1.0000e-02 > REASON: 2 > Iteration = 1 GMRES_Iteration = 2 Err1 = 1.0000e+00, > Err2 = 1.8375e-11 > REASON: 2 > Iteration = 2 GMRES_Iteration = 4 Err1 = 4.1151e-02, > Err2 = 5.9467e-11 > REASON: 2 > Iteration = 3 GMRES_Iteration = 2 Err1 = 1.0265e-02, > Err2 = 2.1268e-12 > REASON: 2 > Iteration = 4 GMRES_Iteration = 2 Err1 = 8.9824e-04, > Err2 = 1.2622e-14 > REASON: 2 > Iteration = 5 GMRES_Iteration = 2 Err1 = 1.5741e-04, > Err2 = 8.1248e-16 > REASON: 2 > Iteration = 6 GMRES_Iteration = 2 Err1 = 7.1515e-06, > Err2 = 1.6605e-16 > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: > ---------------------------------------------- > > ./main on a arch-darwin-c-debug named jiecheng.local with 1 processor, by > Jie Fri Apr 22 18:06:33 2016 > Using Petsc Release Version 3.6.3, Dec, 03, 2015 > > Max Max/Min Avg Total > Time (sec): 4.119e+02 1.00000 4.119e+02 > Objects: 6.800e+01 1.00000 6.800e+01 > Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 > Flops/sec: 6.287e+08 1.00000 6.287e+08 6.287e+08 > Memory: 3.308e+07 1.00000 3.308e+07 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N > --> 2N flops > and VecAXPY() for complex vectors of length N > --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total > Avg %Total counts %Total > 0: Main Stage: 4.1191e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% > 0.000e+00 0.0% 0.000e+00 0.0% > > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and > PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this > phase > %M - percent messages in this phase %L - percent message lengths > in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over > all processors) > > ------------------------------------------------------------------------------------------------------------------------ > > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecMDot 49906 1.0 2.1200e+01 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 5 10 0 0 0 5 10 0 0 0 1231 > VecNorm 51599 1.0 1.5052e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 1 0 0 0 0 1 0 0 0 1159 > VecScale 51587 1.0 5.6397e+00 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 1 0 0 0 0 155 > VecCopy 1682 1.0 5.0184e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 1781 1.0 3.5445e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 3324 1.0 4.0115e-01 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 280 > VecAYPX 12 1.0 1.0931e-03 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 371 > VecMAXPY 51581 1.0 2.4876e+01 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 6 11 0 0 0 6 11 0 0 0 1117 > VecAssemblyBegin 18 1.0 1.6809e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 18 1.0 1.2112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNormalize 51581 1.0 8.1458e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 2 1 0 0 0 2 1 0 0 0 321 > MatMult 51555 1.0 1.5226e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 37 39 0 0 0 37 39 0 0 0 664 > MatSolve 51561 1.0 1.7415e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 42 39 0 0 0 42 39 0 0 0 580 > MatLUFactorNum 6 1.0 6.3280e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 353 > MatILUFactorSym 1 1.0 2.1535e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 6 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 6 1.0 1.7512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 1 1.0 6.8307e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 1 1.0 1.3239e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 6 1.0 9.4421e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPGMRESOrthog 49906 1.0 4.6091e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 11 20 0 0 0 11 20 0 0 0 1133 > KSPSetUp 7 1.0 3.3112e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 6 1.0 3.8880e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 94100 0 0 0 94100 0 0 0 666 > PCSetUp 6 1.0 6.8447e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 326 > PCApply 20 1.0 3.8806e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 > 0.0e+00 94100 0 0 0 94100 0 0 0 667 > > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Vector 56 56 7659456 0 > Matrix 2 2 24271340 0 > Krylov Solver 2 2 36720 0 > Preconditioner 2 2 1832 0 > Index Set 5 5 237080 0 > Viewer 1 0 0 0 > > ======================================================================================================================== > Average time to get PetscTime(): 9.53674e-08 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: --with-mpi-dir=/usr/local > ----------------------------------------- > Libraries compiled on Tue Mar 22 14:05:36 2016 on > calcium-68.dynamic2.rpi.edu > Machine characteristics: Darwin-15.4.0-x86_64-i386-64bit > Using PETSc directory: /usr/local/petsc/petsc-3.6.3 > Using PETSc arch: arch-darwin-c-debug > ----------------------------------------- > > Using C compiler: /usr/local/bin/mpicc -fPIC -Wall -Wwrite-strings > -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 ${COPTFLAGS} ${CFLAGS} > Using Fortran compiler: /usr/local/bin/mpif90 -fPIC -Wall > -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 > ${FOPTFLAGS} ${FFLAGS} > ----------------------------------------- > > Using include paths: > -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include > -I/usr/local/petsc/petsc-3.6.3/include > -I/usr/local/petsc/petsc-3.6.3/include > -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include > -I/usr/local/include > ----------------------------------------- > > Using C linker: /usr/local/bin/mpicc > Using Fortran linker: /usr/local/bin/mpif90 > Using libraries: > -Wl,-rpath,/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib > -L/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -lpetsc -llapack > -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib > -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin > -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin > -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran > -Wl,-rpath,/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 > -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 > -Wl,-rpath,/usr/local/gfortran/lib -L/usr/local/gfortran/lib -lgfortran > -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ > -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin > -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin > -lclang_rt.osx -Wl,-rpath,/usr/local/lib -L/usr/local/lib -ldl -lmpi > -lSystem > -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin > -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin > -lclang_rt.osx -ldl > ----------------------------------------- > > On Apr 22, 2016, at 5:12 PM, Jed Brown wrote: > > Always send the output with -log_summary when asking about performance. > > Jie Cheng writes: > > Hi > > I?m implementing finite element method on nonlinear solid mechanics. The > main portion of my code that involves PETSc is that in each step, the > tangent stiffness matrix A is formed and the increment of the nodal degrees > of freedom is solved. Standard Newton?s iteration. The problem is: when I > use Krylov methods to solve the linear system, the KSPsolve process takes > too long, although only 2 or 3 iterations are needed. > > The finite element formulation is the displacement/pressure mixed > formulation, which I believe is symmetric and positive-definite. However if > I pick conjugate gradient method with ICC preconditioned, PETSc gives me a > -8 converged reason, which indicates a non-positive-definite matrix. After > a couple of trials and errors, the only pair that works is GMRES plus > PCKSP. But as I said, the KSPsolve function takes too much time. > > A typical problem I?m trying has 16906 rows and 16906 cols. The message > printed out by ksp_view is as following: > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ksp > KSP and PC on KSP preconditioner follow > --------------------------------- > KSP Object: (ksp_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (ksp_) 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > package used to perform factorization: petsc > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > --------------------------------- > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16906, cols=16906 > total: nonzeros=988540, allocated nonzeros=988540 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 7582 nodes, limit used is 5 > > Could anyone give me any suggestion please? > > Thanks > Jie Cheng > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 22 23:26:27 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Apr 2016 23:26:27 -0500 Subject: [petsc-users] Krylov Method Takes Too Long to Solve In-Reply-To: <32396333-AB5A-4F7A-96A4-4EAF5B38B3D7@rpi.edu> References: <3B36860B-EB2E-425D-B638-39AF0A447B45@rpi.edu> <874mat4ac0.fsf@jedbrown.org> <4D9A6276-408E-4EA6-9E9B-F5104D99C8AF@rpi.edu> <87shyd2s53.fsf@jedbrown.org> <32396333-AB5A-4F7A-96A4-4EAF5B38B3D7@rpi.edu> Message-ID: Preconditioning for this type of discretization >>>> The finite element formulation is the displacement/pressure mixed formulation requires some thought. You cannot simply blindly apply some simple minded preconditioner like ICC or ILU, you will need to do some literature searches on preconditioner techniques for these discretitization. Based on the analogy of mixed methods for fluid problems I think the likely building block for efficient preconditioners will be so-called "block methods", what we call in PETSc PCFIELDSPLIT. Loosely speaking these "block methods" allow you to "eliminate" the displacement variables and iterate on the Schur complement, which is on the pressure variables. Likely you can use algebraic multigrid in the "elimination" of the displacement variables very effectively. Getting this right is a nontrivial project and requires you to understand your problem; you will never get good results without understanding it. So read the manual pages for PCFIELDSPLIT and then PCGAMG and google scholar for "preconditioners for finite element displacement/pressure mixed formulation". Note that some people writing about the displacement/pressure mixed formulation will emphasis is good performance against "locking" but fail to mention that it is non-trivial to solve the resulting linear systems. Often used more sophisticated discretizations resolves one set of numerical problems but introduces another set (like difficult to solve linear systems) that still must be solved to make the methods practical; there is no free lunch. Barry > On Apr 22, 2016, at 10:50 PM, Jie Cheng wrote: > > Hi Jed and Barry > > Thanks for your help. After I reconfigured PETSc without debugging, it did become much faster. But still not fast enough. The problem remains: what could be the factor that slows the solver down? In this example, only 4662 nodes and 2920 elements are used. Typically I use 3D hexahedral element, the number of degrees of freedom in the mixed formulation I use is 3*4662+2920 = 16906. At this point, my code runs only in serial. I am planning to parallelize it and then work with much finer mesh in the future. If as Barry said, the number of degrees of freedom is too small for iterative solvers, then which direct solver do you recommend to use? > > Here is the log when I ran without debugging. It is not fast enough because this is only 1 step, and there are 99 more steps. > > Thank you. > Jie > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > ./main on a arch-darwin-c-opt named jiecheng.local with 1 processor, by Jie Fri Apr 22 22:36:38 2016 > Using Petsc Release Version 3.6.3, Dec, 03, 2015 > > Max Max/Min Avg Total > Time (sec): 1.520e+02 1.00000 1.520e+02 > Objects: 6.800e+01 1.00000 6.800e+01 > Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 > Flops/sec: 1.704e+09 1.00000 1.704e+09 1.704e+09 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.5195e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecMDot 49906 1.0 8.4904e+00 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 3074 > VecNorm 51599 1.0 1.0644e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1639 > VecScale 51587 1.0 4.5893e-01 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1900 > VecCopy 1682 1.0 2.9097e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 1781 1.0 1.9067e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 3324 1.0 5.8797e-02 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1911 > VecAYPX 12 1.0 3.0708e-04 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1321 > VecMAXPY 51581 1.0 7.2805e+00 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 11 0 0 0 5 11 0 0 0 3817 > VecAssemblyBegin 18 1.0 4.0531e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 18 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNormalize 51581 1.0 1.5588e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1678 > MatMult 51555 1.0 6.2301e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 41 39 0 0 0 41 39 0 0 0 1622 > MatSolve 51561 1.0 5.8966e+01 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 39 39 0 0 0 39 39 0 0 0 1714 > MatLUFactorNum 6 1.0 1.4240e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1569 > MatILUFactorSym 1 1.0 1.0796e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 6 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 6 1.0 6.2988e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 1 1.0 5.9605e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 1 1.0 5.3310e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 6 1.0 5.8441e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPGMRESOrthog 49906 1.0 1.5348e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 0.0e+00 10 20 0 0 0 10 20 0 0 0 3401 > KSPSetUp 7 1.0 1.5509e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 6 1.0 1.3919e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 92100 0 0 0 92100 0 0 0 1861 > PCSetUp 6 1.0 1.5490e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1442 > PCApply 20 1.0 1.3901e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 91100 0 0 0 91100 0 0 0 1861 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Vector 56 56 7659456 0 > Matrix 2 2 24271340 0 > Krylov Solver 2 2 36720 0 > Preconditioner 2 2 1832 0 > Index Set 5 5 237080 0 > Viewer 1 0 0 0 > ======================================================================================================================== > Average time to get PetscTime(): 2.14577e-07 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: --with-debugging=no >> On Apr 22, 2016, at 6:30 PM, Jed Brown wrote: >> >> Allow me to call your attention to some fine print: >> >> ########################################################## >> # # >> # WARNING!!! # >> # # >> # This code was compiled with a debugging option, # >> # To get timing results run ./configure # >> # using --with-debugging=no, the performance will # >> # be generally two or three times faster. # >> # # >> ########################################################## >> >> >> Please always use "reply-all" so that your messages go to the list. >> This is standard mailing list etiquette. It is important to preserve >> threading for people who find this discussion later and so that we do >> not waste our time re-answering the same questions that have already >> been answered in private side-conversations. You'll likely get an >> answer faster that way too. >> >> Jie Cheng writes: >> >>> Hello Jed >>> >>> Thanks for your reply. Here is the log: (the first a couple of lines are printed by my own code during a time step, ?iteration? means the number of Newton?s iteration, ?GMRES_iteration? denotes the number of GMRES iterations it takes in a particular Newton?s iteration, err1 and err2 are my criteria of convergence) >>> >>> >>> Number of nonzeros allocated: 988540 >>> Step = 1, LoadFactor = 1.0000e-02 >>> REASON: 2 >>> Iteration = 1 GMRES_Iteration = 2 Err1 = 1.0000e+00, Err2 = 1.8375e-11 >>> REASON: 2 >>> Iteration = 2 GMRES_Iteration = 4 Err1 = 4.1151e-02, Err2 = 5.9467e-11 >>> REASON: 2 >>> Iteration = 3 GMRES_Iteration = 2 Err1 = 1.0265e-02, Err2 = 2.1268e-12 >>> REASON: 2 >>> Iteration = 4 GMRES_Iteration = 2 Err1 = 8.9824e-04, Err2 = 1.2622e-14 >>> REASON: 2 >>> Iteration = 5 GMRES_Iteration = 2 Err1 = 1.5741e-04, Err2 = 8.1248e-16 >>> REASON: 2 >>> Iteration = 6 GMRES_Iteration = 2 Err1 = 7.1515e-06, Err2 = 1.6605e-16 >>> ************************************************************************************************************************ >>> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** >>> ************************************************************************************************************************ >>> >>> ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- >>> >>> ./main on a arch-darwin-c-debug named jiecheng.local with 1 processor, by Jie Fri Apr 22 18:06:33 2016 >>> Using Petsc Release Version 3.6.3, Dec, 03, 2015 >>> >>> Max Max/Min Avg Total >>> Time (sec): 4.119e+02 1.00000 4.119e+02 >>> Objects: 6.800e+01 1.00000 6.800e+01 >>> Flops: 2.590e+11 1.00000 2.590e+11 2.590e+11 >>> Flops/sec: 6.287e+08 1.00000 6.287e+08 6.287e+08 >>> Memory: 3.308e+07 1.00000 3.308e+07 >>> MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 >>> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 >>> MPI Reductions: 0.000e+00 0.00000 >>> >>> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) >>> e.g., VecAXPY() for real vectors of length N --> 2N flops >>> and VecAXPY() for complex vectors of length N --> 8N flops >>> >>> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- >>> Avg %Total Avg %Total counts %Total Avg %Total counts %Total >>> 0: Main Stage: 4.1191e+02 100.0% 2.5896e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>> >>> ------------------------------------------------------------------------------------------------------------------------ >>> See the 'Profiling' chapter of the users' manual for details on interpreting output. >>> Phase summary info: >>> Count: number of times phase was executed >>> Time and Flops: Max - maximum over all processors >>> Ratio - ratio of maximum to minimum over all processors >>> Mess: number of messages sent >>> Avg. len: average message length (bytes) >>> Reduct: number of global reductions >>> Global: entire computation >>> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). >>> %T - percent time in this phase %F - percent flops in this phase >>> %M - percent messages in this phase %L - percent message lengths in this phase >>> %R - percent reductions in this phase >>> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> >>> ########################################################## >>> # # >>> # WARNING!!! # >>> # # >>> # This code was compiled with a debugging option, # >>> # To get timing results run ./configure # >>> # using --with-debugging=no, the performance will # >>> # be generally two or three times faster. # >>> # # >>> ########################################################## >>> >>> >>> Event Count Time (sec) Flops --- Global --- --- Stage --- Total >>> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> --- Event Stage 0: Main Stage >>> >>> VecMDot 49906 1.0 2.1200e+01 1.0 2.61e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 10 0 0 0 5 10 0 0 0 1231 >>> VecNorm 51599 1.0 1.5052e+00 1.0 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1159 >>> VecScale 51587 1.0 5.6397e+00 1.0 8.72e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 155 >>> VecCopy 1682 1.0 5.0184e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecSet 1781 1.0 3.5445e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAXPY 3324 1.0 4.0115e-01 1.0 1.12e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 280 >>> VecAYPX 12 1.0 1.0931e-03 1.0 4.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 371 >>> VecMAXPY 51581 1.0 2.4876e+01 1.0 2.78e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 11 0 0 0 6 11 0 0 0 1117 >>> VecAssemblyBegin 18 1.0 1.6809e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyEnd 18 1.0 1.2112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecNormalize 51581 1.0 8.1458e+00 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 321 >>> MatMult 51555 1.0 1.5226e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 37 39 0 0 0 37 39 0 0 0 664 >>> MatSolve 51561 1.0 1.7415e+02 1.0 1.01e+11 1.0 0.0e+00 0.0e+00 0.0e+00 42 39 0 0 0 42 39 0 0 0 580 >>> MatLUFactorNum 6 1.0 6.3280e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 353 >>> MatILUFactorSym 1 1.0 2.1535e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 6 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 6 1.0 1.7512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatGetRowIJ 1 1.0 6.8307e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatGetOrdering 1 1.0 1.3239e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatZeroEntries 6 1.0 9.4421e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> KSPGMRESOrthog 49906 1.0 4.6091e+01 1.0 5.22e+10 1.0 0.0e+00 0.0e+00 0.0e+00 11 20 0 0 0 11 20 0 0 0 1133 >>> KSPSetUp 7 1.0 3.3112e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> KSPSolve 6 1.0 3.8880e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 666 >>> PCSetUp 6 1.0 6.8447e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 326 >>> PCApply 20 1.0 3.8806e+02 1.0 2.59e+11 1.0 0.0e+00 0.0e+00 0.0e+00 94100 0 0 0 94100 0 0 0 667 >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> Memory usage is given in bytes: >>> >>> Object Type Creations Destructions Memory Descendants' Mem. >>> Reports information only for process 0. >>> >>> --- Event Stage 0: Main Stage >>> >>> Vector 56 56 7659456 0 >>> Matrix 2 2 24271340 0 >>> Krylov Solver 2 2 36720 0 >>> Preconditioner 2 2 1832 0 >>> Index Set 5 5 237080 0 >>> Viewer 1 0 0 0 >>> ======================================================================================================================== >>> Average time to get PetscTime(): 9.53674e-08 >>> #PETSc Option Table entries: >>> -log_summary >>> #End of PETSc Option Table entries >>> Compiled without FORTRAN kernels >>> Compiled with full precision matrices (default) >>> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >>> Configure options: --with-mpi-dir=/usr/local >>> ----------------------------------------- >>> Libraries compiled on Tue Mar 22 14:05:36 2016 on calcium-68.dynamic2.rpi.edu >>> Machine characteristics: Darwin-15.4.0-x86_64-i386-64bit >>> Using PETSc directory: /usr/local/petsc/petsc-3.6.3 >>> Using PETSc arch: arch-darwin-c-debug >>> ----------------------------------------- >>> >>> Using C compiler: /usr/local/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 ${COPTFLAGS} ${CFLAGS} >>> Using Fortran compiler: /usr/local/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 ${FOPTFLAGS} ${FFLAGS} >>> ----------------------------------------- >>> >>> Using include paths: -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/include -I/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/include -I/usr/local/include >>> ----------------------------------------- >>> >>> Using C linker: /usr/local/bin/mpicc >>> Using Fortran linker: /usr/local/bin/mpif90 >>> Using libraries: -Wl,-rpath,/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -L/usr/local/petsc/petsc-3.6.3/arch-darwin-c-debug/lib -lpetsc -llapack -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.3.0/lib/darwin -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -Wl,-rpath,/usr/local/gfortran/lib -L/usr/local/gfortran/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/usr/local/lib -L/usr/local/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/lib/darwin -lclang_rt.osx -ldl >>> ----------------------------------------- >>> >>>> On Apr 22, 2016, at 5:12 PM, Jed Brown wrote: >>>> >>>> Always send the output with -log_summary when asking about performance. >>>> >>>> Jie Cheng writes: >>>> >>>>> Hi >>>>> >>>>> I?m implementing finite element method on nonlinear solid mechanics. The main portion of my code that involves PETSc is that in each step, the tangent stiffness matrix A is formed and the increment of the nodal degrees of freedom is solved. Standard Newton?s iteration. The problem is: when I use Krylov methods to solve the linear system, the KSPsolve process takes too long, although only 2 or 3 iterations are needed. >>>>> >>>>> The finite element formulation is the displacement/pressure mixed formulation, which I believe is symmetric and positive-definite. However if I pick conjugate gradient method with ICC preconditioned, PETSc gives me a -8 converged reason, which indicates a non-positive-definite matrix. After a couple of trials and errors, the only pair that works is GMRES plus PCKSP. But as I said, the KSPsolve function takes too much time. >>>>> >>>>> A typical problem I?m trying has 16906 rows and 16906 cols. The message printed out by ksp_view is as following: >>>>> >>>>> KSP Object: 1 MPI processes >>>>> type: gmres >>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>>>> GMRES: happy breakdown tolerance 1e-30 >>>>> maximum iterations=10000, initial guess is zero >>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>>> left preconditioning >>>>> using PRECONDITIONED norm type for convergence test >>>>> PC Object: 1 MPI processes >>>>> type: ksp >>>>> KSP and PC on KSP preconditioner follow >>>>> --------------------------------- >>>>> KSP Object: (ksp_) 1 MPI processes >>>>> type: gmres >>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>>>> GMRES: happy breakdown tolerance 1e-30 >>>>> maximum iterations=10000, initial guess is zero >>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>>> left preconditioning >>>>> using PRECONDITIONED norm type for convergence test >>>>> PC Object: (ksp_) 1 MPI processes >>>>> type: ilu >>>>> ILU: out-of-place factorization >>>>> 0 levels of fill >>>>> tolerance for zero pivot 2.22045e-14 >>>>> matrix ordering: natural >>>>> factor fill ratio given 1, needed 1 >>>>> Factored matrix follows: >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=16906, cols=16906 >>>>> package used to perform factorization: petsc >>>>> total: nonzeros=988540, allocated nonzeros=988540 >>>>> total number of mallocs used during MatSetValues calls =0 >>>>> using I-node routines: found 7582 nodes, limit used is 5 >>>>> linear system matrix = precond matrix: >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=16906, cols=16906 >>>>> total: nonzeros=988540, allocated nonzeros=988540 >>>>> total number of mallocs used during MatSetValues calls =0 >>>>> using I-node routines: found 7582 nodes, limit used is 5 >>>>> --------------------------------- >>>>> linear system matrix = precond matrix: >>>>> Mat Object: 1 MPI processes >>>>> type: seqaij >>>>> rows=16906, cols=16906 >>>>> total: nonzeros=988540, allocated nonzeros=988540 >>>>> total number of mallocs used during MatSetValues calls =0 >>>>> using I-node routines: found 7582 nodes, limit used is 5 >>>>> >>>>> Could anyone give me any suggestion please? >>>>> >>>>> Thanks >>>>> Jie Cheng > From knepley at gmail.com Sat Apr 23 11:00:54 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 23 Apr 2016 11:00:54 -0500 Subject: [petsc-users] =?utf-8?b?562U5aSNOiAg562U5aSNOiDnrZTlpI06IEFib3V0?= =?utf-8?q?_the_MatGetOrdering_in_Petsc?= In-Reply-To: <003301d19cfb$fcc5a270$f650e750$@purdue.edu> References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> <003301d19cfb$fcc5a270$f650e750$@purdue.edu> Message-ID: On Fri, Apr 22, 2016 at 8:03 PM, Wei Deng wrote: > There is no bug, as the figure shows, rcm in matlab works well for > symmetric matrix, however, petsc rcm has some bad points, like (117,9) > Wei, We are not trying to do anything but call SPARSEPAK. It is possible that we have made an error here, which I will check. However, I believe that RCM depends on the starting row, and perhaps Matlab is doing some preprocessing here. Since we cannot see exactly what algorithm is being used in Matlab, there is no way for us to verify or match it. Perhaps you could send a small example which we can verify explicitly what ordering Matlab is using. That way we could see which row becomes 0. Thanks, Matt > > > *???**:* Matthew Knepley [mailto:knepley at gmail.com] > *????:* Friday, April 22, 2016 2:32 PM > *???:* Wei Deng > *??:* Barry Smith ; PETSc users list < > petsc-users at mcs.anl.gov> > *??:* Re: [petsc-users] ??: ??: About the MatGetOrdering in Petsc > > > > On Fri, Apr 22, 2016 at 12:18 PM, Wei Deng wrote: > > This is a symmetric matrix and use sysrcm in matlab transform this into > banded, however, call rcm in petsc still fails. > > > > Fails is not descriptive enough. Please send the screenshot. > > > > Matt > > > > filename = 'C:\Users\Wei\Desktop\bcspwr03.mat'; > load(filename) > A=Problem.A; > p = symrcm(A); > R = A(p,p); > % change to dense > %B = full(R); > bandwidth(R) > subplot(1,2,2),spy(A),title(name) > subplot(1,2,2),spy(R),title(name) > > name = 'Zd_Jac6.mat' > path = 'C:\Users\Wei\Documents\Courses\Parallelism Numerical Linear > Algebra\CS515\matlab\' > filename = strcat(path,name); > load(filename) > A=Problem.A; > p = symrcm(A); > R = A(p,p); > % change to dense > %B = full(R); > bandwidth(R) > subplot(1,2,2),spy(A),title(name) > subplot(1,2,2),spy(R),title(name) > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > ????: Friday, April 22, 2016 12:38 PM > ???: Wei Deng ; PETSc users list < > petsc-users at mcs.anl.gov> > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that > these algorithms are for symmetric matrices. > > Perhaps MATLAB detects a non-symmetric matrix and does something different > in that case. > > Barry > > > On Apr 22, 2016, at 11:32 AM, Wei Deng wrote: > > > > So that means that this RCM may be not sufficiently work for a > non-symmetric matrix? > > > > > > -----????----- > > ???: Barry Smith [mailto:bsmith at mcs.anl.gov] > > ????: Friday, April 22, 2016 12:30 PM > > ???: Wei Deng > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > > > > It can be applied to any graph but produces only a single permutation > (that is it returns the same permutation for the rows and the columns even > if the matrix is non-symmetric. From the source code > > > > /*****************************************************************/ > > /*****************************************************************/ > > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > > /*****************************************************************/ > > > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > > /* BY CALLING THE SUBROUTINE RCM.*/ > > > > /* INPUT PARAMETERS -*/ > > /* NEQNS - NUMBER OF EQUATIONS*/ > > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > > > /* OUTPUT PARAMETER -*/ > > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > > > /* WORKING PARAMETERS -*/ > > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > > /* IS NUMBERED.*/ > > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > > > /* PROGRAM SUBROUTINES -*/ > > /* FNROOT, RCM.*/ > > /*****************************************************************/ > > #undef __FUNCT__ > > > >> On Apr 22, 2016, at 8:36 AM, Wei Deng wrote: > >> > >> In other words, does the function MatGetOrdering only support symmetric > matrix? It can?t work well in other cases? > >> > >> ???: Wei Deng [mailto:deng106 at purdue.edu] > >> ????: Thursday, April 21, 2016 11:59 PM > >> ???: 'petsc-users at mcs.anl.gov' > >> ??: About the MatGetOrdering in Petsc > >> > >> Dear Sir/Madam, > >> > >> > >> When I was trying to change a sparse matrix into a banded one, I can > use RCM in matlab to change it perfectly. But this does not work in Petsc, > most points were moved to the diagonal position. However, there are still > some points left, leading the width unacceptable. > >> > >> Do you know why? > >> > >> The attachment is the code with which I have trouble. > >> > >> > >> Thanks a lot. > >> > >> > >> Wei Deng > >> Purdue Uni. > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From deng106 at purdue.edu Sat Apr 23 11:10:27 2016 From: deng106 at purdue.edu (Wei Deng) Date: Sat, 23 Apr 2016 12:10:27 -0400 Subject: [petsc-users] =?utf-8?b?562U5aSNOiDnrZTlpI06ICDnrZTlpI06IOetlA==?= =?utf-8?q?=E5=A4=8D=3A_About_the_MatGetOrdering_in_Petsc?= In-Reply-To: References: <007e01d19c9c$0c8159d0$25840d70$@purdue.edu> <001601d19cb4$909eaf20$b1dc0d60$@purdue.edu> <3E2F04F7-4A5D-4848-B812-4B9DE656C7CB@mcs.anl.gov> <002d01d19cbb$0f063ee0$2d12bca0$@purdue.edu> <003301d19cfb$fcc5a270$f650e750$@purdue.edu> Message-ID: <002401d19d7a$a93f1670$fbbd4350$@purdue.edu> Dear Matt, Really appreciate your help, the attachment are data for Matlab and c, and codes for them. Hope that would help. Best, Wei ???: Matthew Knepley [mailto:knepley at gmail.com] ????: Saturday, April 23, 2016 12:01 PM ???: Wei Deng ??: Barry Smith ; PETSc users list ??: Re: ??: [petsc-users] ??: ??: About the MatGetOrdering in Petsc On Fri, Apr 22, 2016 at 8:03 PM, Wei Deng > wrote: There is no bug, as the figure shows, rcm in matlab works well for symmetric matrix, however, petsc rcm has some bad points, like (117,9) Wei, We are not trying to do anything but call SPARSEPAK. It is possible that we have made an error here, which I will check. However, I believe that RCM depends on the starting row, and perhaps Matlab is doing some preprocessing here. Since we cannot see exactly what algorithm is being used in Matlab, there is no way for us to verify or match it. Perhaps you could send a small example which we can verify explicitly what ordering Matlab is using. That way we could see which row becomes 0. Thanks, Matt ???: Matthew Knepley [mailto:knepley at gmail.com ] ????: Friday, April 22, 2016 2:32 PM ???: Wei Deng > ??: Barry Smith >; PETSc users list > ??: Re: [petsc-users] ??: ??: About the MatGetOrdering in Petsc On Fri, Apr 22, 2016 at 12:18 PM, Wei Deng > wrote: This is a symmetric matrix and use sysrcm in matlab transform this into banded, however, call rcm in petsc still fails. Fails is not descriptive enough. Please send the screenshot. Matt filename = 'C:\Users\Wei\Desktop\bcspwr03.mat'; load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) name = 'Zd_Jac6.mat' path = 'C:\Users\Wei\Documents\Courses\Parallelism Numerical Linear Algebra\CS515\matlab\' filename = strcat(path,name); load(filename) A=Problem.A; p = symrcm(A); R = A(p,p); % change to dense %B = full(R); bandwidth(R) subplot(1,2,2),spy(A),title(name) subplot(1,2,2),spy(R),title(name) -----????----- ???: Barry Smith [mailto:bsmith at mcs.anl.gov ] ????: Friday, April 22, 2016 12:38 PM ???: Wei Deng >; PETSc users list > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc https://en.wikipedia.org/wiki/Cuthill?McKee_algorithm indicates that these algorithms are for symmetric matrices. Perhaps MATLAB detects a non-symmetric matrix and does something different in that case. Barry > On Apr 22, 2016, at 11:32 AM, Wei Deng > wrote: > > So that means that this RCM may be not sufficiently work for a non-symmetric matrix? > > > -----????----- > ???: Barry Smith [mailto:bsmith at mcs.anl.gov ] > ????: Friday, April 22, 2016 12:30 PM > ???: Wei Deng > > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] ??: About the MatGetOrdering in Petsc > > > It can be applied to any graph but produces only a single permutation (that is it returns the same permutation for the rows and the columns even if the matrix is non-symmetric. From the source code > > /*****************************************************************/ > /*****************************************************************/ > /********* GENRCM ..... GENERAL REVERSE CUTHILL MCKEE ********/ > /*****************************************************************/ > > /* PURPOSE - GENRCM FINDS THE REVERSE CUTHILL-MCKEE*/ > /* ORDERING FOR A GENERAL GRAPH. FOR EACH CONNECTED*/ > /* COMPONENT IN THE GRAPH, GENRCM OBTAINS THE ORDERING*/ > /* BY CALLING THE SUBROUTINE RCM.*/ > > /* INPUT PARAMETERS -*/ > /* NEQNS - NUMBER OF EQUATIONS*/ > /* (XADJ, ADJNCY) - ARRAY PAIR CONTAINING THE ADJACENCY*/ > /* STRUCTURE OF THE GRAPH OF THE MATRIX.*/ > > /* OUTPUT PARAMETER -*/ > /* PERM - VECTOR THAT CONTAINS THE RCM ORDERING.*/ > > /* WORKING PARAMETERS -*/ > /* MASK - IS USED TO MARK VARIABLES THAT HAVE BEEN*/ > /* NUMBERED DURING THE ORDERING PROCESS. IT IS*/ > /* INITIALIZED TO 1, AND SET TO ZERO AS EACH NODE*/ > /* IS NUMBERED.*/ > /* XLS - THE INDEX VECTOR FOR A LEVEL STRUCTURE. THE*/ > /* LEVEL STRUCTURE IS STORED IN THE CURRENTLY*/ > /* UNUSED SPACES IN THE PERMUTATION VECTOR PERM.*/ > > /* PROGRAM SUBROUTINES -*/ > /* FNROOT, RCM.*/ > /*****************************************************************/ > #undef __FUNCT__ > >> On Apr 22, 2016, at 8:36 AM, Wei Deng > wrote: >> >> In other words, does the function MatGetOrdering only support symmetric matrix? It can?t work well in other cases? >> >> ???: Wei Deng [mailto:deng106 at purdue.edu ] >> ????: Thursday, April 21, 2016 11:59 PM >> ???: 'petsc-users at mcs.anl.gov ' > >> ??: About the MatGetOrdering in Petsc >> >> Dear Sir/Madam, >> >> >> When I was trying to change a sparse matrix into a banded one, I can use RCM in matlab to change it perfectly. But this does not work in Petsc, most points were moved to the diagonal position. However, there are still some points left, leading the width unacceptable. >> >> Do you know why? >> >> The attachment is the code with which I have trouble. >> >> >> Thanks a lot. >> >> >> Wei Deng >> Purdue Uni. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bcspwr03.mat Type: application/octet-stream Size: 1314 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bcspwr03.mtx Type: application/octet-stream Size: 2446 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: matlab.m Type: application/octet-stream Size: 319 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: main.c URL: From balay at mcs.anl.gov Sun Apr 24 00:24:45 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 24 Apr 2016 00:24:45 -0500 Subject: [petsc-users] message for petsc git users - using 'maint' branch Message-ID: Dear PETSc 'maint' branch users, In anticipation of the next 3.7 release of petsc - I've updated 'maint' branch in petsc git repository to 3.7 pre-release changes. If you 'git pull' on maint branch - you would get these 3.7 pre-release changes. To stay on 3.6 - you can switch to 'maint-3.6' branch. git fetch git checkout maint-3.6 I apologize for any inconvenience this has caused. Satish From ztdepyahoo at 163.com Sun Apr 24 02:01:52 2016 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Sun, 24 Apr 2016 15:01:52 +0800 (CST) Subject: [petsc-users] error: AO was not declared in this scope. Message-ID: <1f7494f9.3bd7.154471310d6.Coremail.ztdepyahoo@163.com> Dear friends: I am adding the partition to my CFD code, but it give me error about " AO was not declared in this scope". the parallel code has been run suecesfuly with a off line mesh partition with ksp solver. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Apr 24 04:14:18 2016 From: zonexo at gmail.com (Wee-beng TAY) Date: Sun, 24 Apr 2016 17:14:18 +0800 Subject: [petsc-users] Modifying built files for use with Japan's K computer Message-ID: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> Hi, I was recently given access to Japan's K computer. I'm trying to compile PETSc on their sys, which uses Fujitsu's compiler. There's a help file but it's in Japanese. I guess I need to change some options to make PETSc compile with Fujitsu's compiler. Part of the help is below. So how should I use the diff or some other command to do the correction? I tried to run: diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org petsc-3.6.4/config/BuildSystem/config/package.py but got the err: diff: petsc-3.6.4/config/BuildSystem/config/package.py.org: No such file or directory Please advice. Thanks. /|$ diff -u petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py|/ /|--- petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org |//|2015|//|-|//|12|//|-|//|04|////|14|//|:|//|06|//|:|//|42.000000000|////|+|//|0900|/ /|+++ petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|09|//|:|//|37.000000000|////|+|//|0900|/ /|@@ -|//|174|//|,|//|7|////|+|//|174|//|,|//|7|////|@@|/ /||//|return|////|''|/ // /||//|def getSharedFlag(self,cflags):|/ /|- |//|for|////|flag in [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]:|/ /|+ |//|for|////|flag in [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]:|/ /||//|if|////|cflags.find(flag) >=|//|0|//|:|//|return|////|flag|/ /||//|return|////|''|/ /||/ /|$ diff -u petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py|/ /|--- petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org |//|2015|//|-|//|07|//|-|//|23|////|00|//|:|//|22|//|:|//|46.000000000|////|+|//|0900|/ /|+++ petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|10|//|:|//|05.000000000|////|+|//|0900|/ /|@@ -|//|1017|//|,|//|7|////|+|//|1017|//|,|//|7|////|@@|/ /||//|self.pushLanguage(language)|/ /||//|#different compilers are sensitive to the order of testing these flags. So separete out GCC test.|/ /||//|if|////|config.setCompilers.Configure.isGNU(self.getCompiler()): testFlags = [|//|'-fPIC'|//|]|/ /|- |//|else|//|: testFlags = [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]|/ /|+ |//|else|//|: testFlags = [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]|/ /||//|for|////|testFlag in testFlags:|/ /||//|try|//|:|/ /||//|self.logPrint(|//|'Trying '|//|+language+|//|' compiler flag '|//|+testFlag)|/ /|$ diff -u petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py|/ /|--- petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org |//|2016|//|-|//|01|//|-|//|25|////|15|//|:|//|42|//|:|//|23.000000000|//|+|//|0900|/ /|+++ petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py |//|2016|//|-|//|01|//|-|//|22|////|17|//|:|//|13|//|:|//|52.000000000|////|+|//|0900|/ /|@@ -|//|19|//|,|//|7|////|+|//|19|//|,|//|8|////|@@|/ /||//|self.found =|//|0|/ /||//|self.setCompilers.pushLanguage(|//|'C'|//|)|/ /||//|#|/ /|- |//|for|////|flag in [|//|"-fopenmp"|//|, # Gnu|/ /|+ |//|for|////|flag in [|//|"-Kopenmp"|//|, # Fujitsu|/ /|+ |//|"-fopenmp"|//|, # Gnu|/ /||//|"-qsmp=omp"|//|,# IBM XL C/C++|/ /||//|"-h omp"|//|, # Cray. Must come after XL because XL interprets|//|this|////|option as meaning|//|"-soname omp"|/ /||//|"-mp"|//|, # Portland Group|/ // /|$ diff -u ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py|/ /|--- ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org |//|2015|//|-|//|06|//|-|//|10|////|06|//|:|//|24|//|:|//|49.000000000|////|+|//|0900|/ /|+++ ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py |//|2016|//|-|//|02|//|-|//|19|////|11|//|:|//|56|//|:|//|12.000000000|////|+|//|0900|/ /|@@ -|//|164|//|,|//|7|////|+|//|164|//|,|//|7|////|@@|/ /||//|def checkCLibraries(self):|/ /||//|''|//|'Determines the libraries needed to link with C'|//|''|/ /||//|oldFlags = self.setCompilers.LDFLAGS|/ /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ /||//|self.pushLanguage(|//|'C'|//|)|/ /||//|(output, returnCode) = self.outputLink(|//|''|//|,|//|''|//|)|/ /||//|self.setCompilers.LDFLAGS = oldFlags|/ /|@@ -|//|413|//|,|//|7|////|+|//|413|//|,|//|7|////|@@|/ /||//|def checkCxxLibraries(self):|/ /||//|''|//|'Determines the libraries needed to link with C++'|//|''|/ /||//|oldFlags = self.setCompilers.LDFLAGS|/ /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ /||//|self.pushLanguage(|//|'Cxx'|//|)|/ /||//|(output, returnCode) = self.outputLink(|//|''|//|,|//|''|//|)|/ /||//|self.setCompilers.LDFLAGS = oldFlags|/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 24 09:52:30 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Apr 2016 09:52:30 -0500 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> Message-ID: On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY wrote: > Hi, > > I was recently given access to Japan's K computer. I'm trying to compile > PETSc on their sys, which uses Fujitsu's compiler. There's a help file but > it's in Japanese. I guess I need to change some options to make PETSc > compile with Fujitsu's compiler. > It looks like they want us to include some different options for the build. If you send what you want, I will make a branch for you to try out. Thanks, Matt > Part of the help is below. So how should I use the diff or some other > command to do the correction? I tried to run: > > diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org > petsc-3.6.4/config/BuildSystem/config/package.py > > but got the err: > > diff: petsc-3.6.4/config/BuildSystem/config/package.py.org: No such file > or directory > > Please advice. Thanks. > > > *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org > petsc-**3.6**.**3**/config/BuildSystem/config/**package* > *.py* > *--- petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org > **2015**-**12**-**04* *14**:**06**:**42.000000000* > *+**0900* > *+++ petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py * > *2016**-**01**-**22* *11**:**09**:**37.000000000* *+**0900* > *@@ -**174**,**7* *+**174**,**7* *@@* > *return* *''* > > *def getSharedFlag(self,cflags):* > *- **for* *flag in [**'-PIC'**, **'-fPIC'**, **'-KPIC'**, **'-qpic'* > *]:* > *+ **for* *flag in [**'-KPIC'**, **'-fPIC'**, **'-PIC'**, **'-qpic'* > *]:* > *if* *cflags.find(flag) >=**0**: **return* *flag* > *return* *''* > > *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org > petsc-**3.6**.**3* > */config/BuildSystem/config/setCompilers.py* > *--- petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org > **2015**-**07**-**23* *00**:**22**:* > *46.000000000* *+**0900* > *+++ petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py * > *2016**-**01**-**22* *11**:**10**:**05.000000000* *+**0900* > *@@ -**1017**,**7* *+**1017**,**7* *@@* > *self.pushLanguage(language)* > *#different compilers are sensitive to the order of testing these > flags. So separete out GCC test.* > *if* *config.setCompilers.Configure.isGNU(self.getCompiler()): > testFlags = [**'-fPIC'**]* > *- **else**: testFlags = [**'-PIC'**, **'-fPIC'**, **'-KPIC'**,* > *'-qpic'**]* > *+ **else**: testFlags = [**'-KPIC'**, **'-fPIC'**, **'-PIC'**,* > *'-qpic'**]* > *for* *testFlag in testFlags:* > *try**:* > *self.logPrint(**'Trying '**+language+**' compiler flag '* > *+testFlag)* > *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org > petsc-**3.6**.**3* > */config/BuildSystem/config/packages/openmp.py* > *--- petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org > **2016**-**01**-**25* *15**:**42**:* > *23.000000000**+**0900* > *+++ petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py * > *2016**-**01**-**22* *17**:**13**:**52.000000000* *+**0900* > *@@ -**19**,**7* *+**19**,**8* *@@* > *self.found = **0* > *self.setCompilers.pushLanguage(**'C'**)* > *#* > *- **for* *flag in [**"-fopenmp"**, # Gnu* > *+ **for* *flag in [**"-Kopenmp"**, # Fujitsu* > *+ **"-fopenmp"**, # Gnu* > *"-qsmp=omp"**,# IBM XL C/C++* > *"-h omp"**, # Cray. Must come after XL because XL > interprets **this* *option as meaning**"-soname omp"* > *"-mp"**, # Portland Group* > > *$ diff -u ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org > ./petsc-**3.6**.**3* > */config/BuildSystem/config/compilers.py* > *--- ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org > **2015**-**06**-**10* *06**:**24**:* > *49.000000000* *+**0900* > *+++ ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py * > *2016**-**02**-**19* *11**:**56**:**12.000000000* *+**0900* > *@@ -**164**,**7* *+**164**,**7* *@@* > *def checkCLibraries(self):* > *''**'Determines the libraries needed to link with C'**''* > *oldFlags = self.setCompilers.LDFLAGS* > *- self.setCompilers.LDFLAGS += **' -v'* > *+ self.setCompilers.LDFLAGS += **' -###'* > *self.pushLanguage(**'C'**)* > *(output, returnCode) = self.outputLink(**''**, **''**)* > *self.setCompilers.LDFLAGS = oldFlags* > *@@ -**413**,**7* *+**413**,**7* *@@* > *def checkCxxLibraries(self):* > *''**'Determines the libraries needed to link with C++'**''* > *oldFlags = self.setCompilers.LDFLAGS* > *- self.setCompilers.LDFLAGS += **' -v'* > *+ self.setCompilers.LDFLAGS += **' -###'* > *self.pushLanguage(**'Cxx'**)* > *(output, returnCode) = self.outputLink(**''**, **''**)* > *self.setCompilers.LDFLAGS = oldFlags* > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 24 09:53:24 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Apr 2016 09:53:24 -0500 Subject: [petsc-users] error: AO was not declared in this scope. In-Reply-To: <1f7494f9.3bd7.154471310d6.Coremail.ztdepyahoo@163.com> References: <1f7494f9.3bd7.154471310d6.Coremail.ztdepyahoo@163.com> Message-ID: On Sun, Apr 24, 2016 at 2:01 AM, ??? wrote: > Dear friends: > I am adding the partition to my CFD code, but it give me error about > " AO was not declared in this scope". > the parallel code has been run suecesfuly with a off line mesh partition > with ksp solver. > When reporting an error, we need to see the entire error message and stack trace. Thanks, Matt > Regards. > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Apr 24 10:12:20 2016 From: zonexo at gmail.com (Wee-beng TAY) Date: Sun, 24 Apr 2016 23:12:20 +0800 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> Message-ID: Hi Matt, I have attached the page which explains what to modify to use PETSc with the K computer. Thanks On 24/4/2016 10:52 PM, Matthew Knepley wrote: > On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY > wrote: > > Hi, > > I was recently given access to Japan's K computer. I'm trying to > compile PETSc on their sys, which uses Fujitsu's compiler. There's > a help file but it's in Japanese. I guess I need to change some > options to make PETSc compile with Fujitsu's compiler. > > It looks like they want us to include some different options for the > build. If you send what you want, > I will make a branch for you to try out. > > Thanks, > > Matt > > Part of the help is below. So how should I use the diff or some > other command to do the correction? I tried to run: > > diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org > > petsc-3.6.4/config/BuildSystem/config/package.py > > but got the err: > > diff: petsc-3.6.4/config/BuildSystem/config/package.py.org > : No such file or directory > > Please advice. Thanks. > > > /|$ diff -u > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org > > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py|/ > /|--- > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org > > |//|2015|//|-|//|12|//|-|//|04|////|14|//|:|//|06|//|:|//|42.000000000|////|+|//|0900|/ > /|+++ > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py > |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|09|//|:|//|37.000000000|////|+|//|0900|/ > /|@@ -|//|174|//|,|//|7|////|+|//|174|//|,|//|7|////|@@|/ > /||//|return|////|''|/ > // > /||//|def getSharedFlag(self,cflags):|/ > /|- |//|for|////|flag in > [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]:|/ > /|+ |//|for|////|flag in > [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]:|/ > /||//|if|////|cflags.find(flag) >=|//|0|//|:|//|return|////|flag|/ > /||//|return|////|''|/ > /||/ > /|$ diff -u > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org > > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py|/ > /|--- > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org > > |//|2015|//|-|//|07|//|-|//|23|////|00|//|:|//|22|//|:|//|46.000000000|////|+|//|0900|/ > /|+++ > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py > |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|10|//|:|//|05.000000000|////|+|//|0900|/ > /|@@ -|//|1017|//|,|//|7|////|+|//|1017|//|,|//|7|////|@@|/ > /||//|self.pushLanguage(language)|/ > /||//|#different compilers are sensitive to the order of testing > these flags. So separete out GCC test.|/ > /||//|if|////|config.setCompilers.Configure.isGNU(self.getCompiler()): > testFlags = [|//|'-fPIC'|//|]|/ > /|- |//|else|//|: testFlags = > [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]|/ > /|+ |//|else|//|: testFlags = > [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]|/ > /||//|for|////|testFlag in testFlags:|/ > /||//|try|//|:|/ > /||//|self.logPrint(|//|'Trying '|//|+language+|//|' compiler flag > '|//|+testFlag)|/ > /|$ diff -u > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org > > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py|/ > /|--- > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org > > |//|2016|//|-|//|01|//|-|//|25|////|15|//|:|//|42|//|:|//|23.000000000|//|+|//|0900|/ > /|+++ > petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py > |//|2016|//|-|//|01|//|-|//|22|////|17|//|:|//|13|//|:|//|52.000000000|////|+|//|0900|/ > /|@@ -|//|19|//|,|//|7|////|+|//|19|//|,|//|8|////|@@|/ > /||//|self.found =|//|0|/ > /||//|self.setCompilers.pushLanguage(|//|'C'|//|)|/ > /||//|#|/ > /|- |//|for|////|flag in [|//|"-fopenmp"|//|, # Gnu|/ > /|+ |//|for|////|flag in [|//|"-Kopenmp"|//|, # Fujitsu|/ > /|+ |//|"-fopenmp"|//|, # Gnu|/ > /||//|"-qsmp=omp"|//|,# IBM XL C/C++|/ > /||//|"-h omp"|//|, # Cray. Must come after XL because XL > interprets|//|this|////|option as meaning|//|"-soname omp"|/ > /||//|"-mp"|//|, # Portland Group|/ > // > /|$ diff -u > ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org > > ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py|/ > /|--- > ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org > > |//|2015|//|-|//|06|//|-|//|10|////|06|//|:|//|24|//|:|//|49.000000000|////|+|//|0900|/ > /|+++ > ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py > |//|2016|//|-|//|02|//|-|//|19|////|11|//|:|//|56|//|:|//|12.000000000|////|+|//|0900|/ > /|@@ -|//|164|//|,|//|7|////|+|//|164|//|,|//|7|////|@@|/ > /||//|def checkCLibraries(self):|/ > /||//|''|//|'Determines the libraries needed to link with C'|//|''|/ > /||//|oldFlags = self.setCompilers.LDFLAGS|/ > /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ > /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ > /||//|self.pushLanguage(|//|'C'|//|)|/ > /||//|(output, returnCode) = self.outputLink(|//|''|//|,|//|''|//|)|/ > /||//|self.setCompilers.LDFLAGS = oldFlags|/ > /|@@ -|//|413|//|,|//|7|////|+|//|413|//|,|//|7|////|@@|/ > /||//|def checkCxxLibraries(self):|/ > /||//|''|//|'Determines the libraries needed to link with C++'|//|''|/ > /||//|oldFlags = self.setCompilers.LDFLAGS|/ > /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ > /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ > /||//|self.pushLanguage(|//|'Cxx'|//|)|/ > /||//|(output, returnCode) = self.outputLink(|//|''|//|,|//|''|//|)|/ > /||//|self.setCompilers.LDFLAGS = oldFlags|/ > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- PETSc???? Skip to end of metadata Created by Yoshihiko Suzuki, last modified on 2016/03/08 Go to start of metadata PETSc???? ?????????? ?????? ???????? ?????? ???????????? ????????????????? ?????????? (1)??????? ?Portable, Extensible Toolkit for Scientific Computation (2)???????? ????????????????????????????????????????????? (3)??????? ?3.6.3 (4)?? ?Fortran,C?C++,MPI,OpenMP (5)???url ?http://www.mcs.anl.gov/petsc/ (6)????? ?BSD 2-clause license (7)??????? ?BLAS/LAPACK,SCALAPACK (8)?????? ? - ?????? ?????????????????????????????????? (1)??????????? ???url???????????????????????????????????????????? ????petsc-3.6.3????????????????????????????????? (2) ???????? ?configure?????????????????????????????????????????? $ diff -u petsc-3.6.3/config/BuildSystem/config/package.py.org petsc-3.6.3/config/BuildSystem/config/package.py --- petsc-3.6.3/config/BuildSystem/config/package.py.org 2015-12-04 14:06:42.000000000 +0900 +++ petsc-3.6.3/config/BuildSystem/config/package.py 2016-01-22 11:09:37.000000000 +0900 @@ -174,7 +174,7 @@ return '' def getSharedFlag(self,cflags): - for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: + for flag in ['-KPIC', '-fPIC', '-PIC', '-qpic']: if cflags.find(flag) >=0: return flag return '' $ diff -u petsc-3.6.3/config/BuildSystem/config/setCompilers.py.org petsc-3.6.3/config/BuildSystem/config/setCompilers.py --- petsc-3.6.3/config/BuildSystem/config/setCompilers.py.org 2015-07-23 00:22:46.000000000 +0900 +++ petsc-3.6.3/config/BuildSystem/config/setCompilers.py 2016-01-22 11:10:05.000000000 +0900 @@ -1017,7 +1017,7 @@ self.pushLanguage(language) #different compilers are sensitive to the order of testing these flags. So separete out GCC test. if config.setCompilers.Configure.isGNU(self.getCompiler()): testFlags = ['-fPIC'] - else: testFlags = ['-PIC', '-fPIC', '-KPIC','-qpic'] + else: testFlags = ['-KPIC', '-fPIC', '-PIC','-qpic'] for testFlag in testFlags: try: self.logPrint('Trying '+language+' compiler flag '+testFlag) $ diff -u petsc-3.6.3/config/BuildSystem/config/packages/openmp.py.org petsc-3.6.3/config/BuildSystem/config/packages/openmp.py --- petsc-3.6.3/config/BuildSystem/config/packages/openmp.py.org 2016-01-25 15:42:23.000000000+0900 +++ petsc-3.6.3/config/BuildSystem/config/packages/openmp.py 2016-01-22 17:13:52.000000000 +0900 @@ -19,7 +19,8 @@ self.found = 0 self.setCompilers.pushLanguage('C') # - for flag in ["-fopenmp", # Gnu + for flag in ["-Kopenmp", # Fujitsu + "-fopenmp", # Gnu "-qsmp=omp",# IBM XL C/C++ "-h omp", # Cray. Must come after XL because XL interprets this option as meaning"-soname omp" "-mp", # Portland Group $ diff -u ./petsc-3.6.3/config/BuildSystem/config/compilers.py.org ./petsc-3.6.3/config/BuildSystem/config/compilers.py --- ./petsc-3.6.3/config/BuildSystem/config/compilers.py.org 2015-06-10 06:24:49.000000000 +0900 +++ ./petsc-3.6.3/config/BuildSystem/config/compilers.py 2016-02-19 11:56:12.000000000 +0900 @@ -164,7 +164,7 @@ def checkCLibraries(self): '''Determines the libraries needed to link with C''' oldFlags = self.setCompilers.LDFLAGS - self.setCompilers.LDFLAGS += ' -v' + self.setCompilers.LDFLAGS += ' -###' self.pushLanguage('C') (output, returnCode) = self.outputLink('', '') self.setCompilers.LDFLAGS = oldFlags @@ -413,7 +413,7 @@ def checkCxxLibraries(self): '''Determines the libraries needed to link with C++''' oldFlags = self.setCompilers.LDFLAGS - self.setCompilers.LDFLAGS += ' -v' + self.setCompilers.LDFLAGS += ' -###' self.pushLanguage('Cxx') (output, returnCode) = self.outputLink('', '') self.setCompilers.LDFLAGS = oldFlags (3)configure??? ?????????????????????????????????????????????????????? ?1???1???????????????????????configure?????? ?????????$HOME/apps/petsc-3.6.3???? $ export PETSC_ARCH=sparc64-unknown-linux-gnu $ export PETSC_DIR=`pwd` $ ./configure --with-cc=mpifcc --with-cxx=mpiFCC --with-fc=mpifrt --with-64-bit-pointers=1 \ --CC="mpifcc" --CFLAGS="-Xg -O2" --CXX="mpiFCC" --CXXFLAGS="-Xg -O2" --FC="mpifrt" --FFLAGS="-X9 -O2" \ --LD_SHARED="" --LDDFLAGS="" --with-openmp=1 --with-mpiexec=mpiexec --known-endian=big --with-shared-libraries=0 \ --with-blas-lapack-lib="-SSL2" --with-scalapack-lib="-SCALAPACK" \ --prefix=$HOME/apps/petsc-3.6.3 ?configure???????????????????????????? ???make PETSC_DIR=$PWD PETSC_ARCH=sparc64-unknown-linux-gnu all ?1 ???($PWD???????????) ????????1?????? $ make PETSC_DIR=$PWD PETSC_ARCH=sparc64-unknown-linux-gnu all ??1????????????????????????????????????????? ???make PETSC_DIR=$PWD PETSC_ARCH=sparc64-unknown-linux-gnu install ?2 ???($PWD???????????) ???????? ????????2?????? $ make PETSC_DIR=$PWD PETSC_ARCH=sparc64-unknown-linux-gnu install ??2??????configure??prefix?????????????????PETSc?????????? ??????????????????????? ???make PETSC_DIR=$HOME/apps/petsc-3.6.3 PETSC_ARCH="" test ?3 ?? ?($HOME???????????) ?????? ??3?2??????????????????????????????? $ make PETSC_DIR=$HOME/apps/petsc-3.6.3 PETSC_ARCH="" test ????NPROC???????????????????????????????????????????????????? ???make PETSC_DIR=$HOME/apps/petsc-3.6.3 PETSC_ARCH="" stream NPMAX=NPROC ???????????? ?configure?????????"--with-shared-libraries"?1?????????????????????? ????????????????? ???????????????????????????????? $ cd $HOME/apps/petsc-3.6.3 $ ls -R .: bin include lib share ./bin: FASTMathInstaller.py configVars.py petsc_gen_xdmf.py popup uncrustify.cfg PetscBinaryIO.py julia petsclogformat.py portabilitycheck.py update.py PetscBinaryIO_tests.py parseargs.py petscmpiexec saws urlget TOPSGenerator.py petsc-mpiexec.uni petscnagupgrade.py sendToJenkins urlget.py adiforfix.py petsc_conf.py petscnagupgrade.pyc taucc.py win32fe ./bin/julia: PETSc.jl examples ./bin/julia/examples: tutorials ./bin/julia/examples/tutorials: exKSP.jl ./bin/saws: SAWs.py getSAWs.bash getStack.bash ./bin/win32fe: win32fe.exe win32feutils.dll ./include: index.html petscdmplex.h.html petscoptions.h makefile.html petscdmredundant.h petscoptions.h.html petsc petscdmredundant.h.html petscpc.h petsc.h petscdmshell.h petscpc.h.html petsc.h.html petscdmshell.h.html petscpc.mod petsc.mod petscdmsliced.h petscpcdef.mod petscao.h petscdmsliced.h.html petscpctypes.h petscao.h.html petscdmtypes.h petscpctypes.h.html petscao.mod petscdmtypes.h.html petscpf.h petscaodef.mod petscdraw.h petscpf.h.html petscbag.h petscdraw.h.html petscsf.h petscbag.h.html petscdrawtypes.h petscsf.h.html petscblaslapack.h petscdrawtypes.h.html petscsftypes.h petscblaslapack.h.html petscds.h petscsftypes.h.html petscblaslapack_c.h petscds.h.html petscsnes.h petscblaslapack_c.h.html petscdstypes.h petscsnes.h.html petscblaslapack_caps.h petscdstypes.h.html petscsnes.mod petscblaslapack_caps.h.html petscdt.h petscsnesdef.mod petscblaslapack_stdcall.h petscdt.h.html petscsys.h petscblaslapack_stdcall.h.html petscerror.h petscsys.h.html petscblaslapack_uscore.h petscerror.h.html petscsys.mod petscblaslapack_uscore.h.html petscfe.h petscsysdef.mod petscbt.h petscfe.h.html petsctao.h petscbt.h.html petscfetypes.h petsctao.h.html petsccharacteristic.h petscfetypes.h.html petsctaolinesearch.h petsccharacteristic.h.html petscfix.h petsctaolinesearch.h.html petscconf.h petscfv.h petsctime.h petscconfiginfo.h petscfv.h.html petsctime.h.html petscctable.h petscfvtypes.h petscts.h petscctable.h.html petscfvtypes.h.html petscts.h.html petsccusp.h petscis.h petscts.mod petsccusp.h.html petscis.h.html petsctsdef.mod petscdef.mod petscis.mod petscvalgrind.h petscdm.h petscisdef.mod petscvalgrind.h.html petscdm.h.html petscistypes.h petscvec.h petscdm.mod petscistypes.h.html petscvec.h.html petscdmcomposite.h petscksp.h petscvec.mod petscdmcomposite.h.html petscksp.h.html petscvecdef.mod petscdmcomposite.mod petscksp.mod petscversion.h petscdmda.h petsckspdef.mod petscversion.h.html petscdmda.h.html petsclog.h petscviennacl.h petscdmda.mod petsclog.h.html petscviennacl.h.html petscdmdadef.mod petscmachineinfo.h petscviewer.h petscdmdatypes.h petscmat.h petscviewer.h.html petscdmdatypes.h.html petscmat.h.html petscviewerhdf5.h petscdmdef.mod petscmat.mod petscviewerhdf5.h.html petscdmmoab.h petscmatdef.mod petscviewersaws.h petscdmmoab.h.html petscmatelemental.h petscviewersaws.h.html petscdmnetwork.h petscmatelemental.h.html petscviewertypes.h petscdmnetwork.h.html petscmath.h petscviewertypes.h.html petscdmpatch.h petscmath.h.html petscwebclient.h petscdmpatch.h.html petscmatlab.h petscwebclient.h.html petscdmplex.h petscmatlab.h.html ./include/petsc: finclude mpiuni private ./include/petsc/finclude: ftn-auto petscdmdadef.h petscksp.h90 petscsnesfas.h ftn-custom petscdmdadef.h.html petsckspdef.h petscsnesfasdef.h index.html petscdmdef.h petsckspdef.h.html petscsys.h makefile.html petscdmdef.h.html petsclog.h petscsys.h.html petsc.h petscdmplex.h petsclog.h.html petscsysdef.h petsc.h.html petscdmplex.h90 petsclogdef.h petscsysdef.h.html petsc.h90 petscdmplexdef.h petsclogdef.h.html petsctao.h petscao.h petscdmredundant.h90 petscmat.h petsctao.h.html petscao.h.html petscdmredundant.h90.html petscmat.h.html petsctaodef.h petscaodef.h petscdmshell.h90 petscmat.h90 petscts.h petscaodef.h.html petscdraw.h petscmat.h90.html petscts.h.html petscbag.h petscdraw.h.html petscmatdef.h petscts.h90 petscbagdef.h petscdrawdef.h petscmatdef.h.html petsctsdef.h petscdef.h petscdrawdef.h.html petscmatfactorinfosize.h petsctsdef.h.html petscdef.h.html petscdt.h90 petscmatinfosize.h petscvec.h petscdm.h petscerror.h petscpc.h petscvec.h.html petscdm.h.html petscerrordef.h petscpc.h.html petscvec.h90 petscdm.h90 petscis.h petscpc.h90 petscvec.h90.html petscdm.h90.html petscis.h.html petscpcdef.h petscvecdef.h petscdmcomposite.h90 petscis.h90 petscpcdef.h.html petscvecdef.h.html petscdmcomposite.h90.html petscis.h90.html petscsnes.h petscviewer.h petscdmda.h petscisdef.h petscsnes.h.html petscviewer.h.html petscdmda.h.html petscisdef.h.html petscsnes.h90 petscviewer.h90 petscdmda.h90 petscksp.h petscsnesdef.h petscviewerdef.h petscdmda.h90.html petscksp.h.html petscsnesdef.h.html petscviewerdef.h.html ./include/petsc/finclude/ftn-auto: petsc.h90 petscdmshell.h90 petscpcexotic.h90 petscsnesfas.h90 petscao.h90 petscdraw.h90 petscpcgasm.h90 petscsys.h90 petscdm.h90 petscis.h90 petscpcmg.h90 petsctao.h90 petscdmcomposite.h90 petscksp.h90 petscpetscsf.h90 petsctaolinesearch.h90 petscdmda.h90 petscmat.h90 petscpf.h90 petscts.h90 petscdmpatch.h90 petscmatfd.h90 petscprofiling.h90 petscvec.h90 petscdmplex.h90 petscmatorderings.h90 petscsemilagrange.h90 petscviewer.h90 petscdmredundant.h90 petscpc.h90 petscsnes.h90 ./include/petsc/finclude/ftn-custom: petscdm.h90 petscdmshell.h90 petsckspdef.h90 petscsnes.h90 petscvechide.h90 petscdmcomposite.h90 petscdt.h90 petscksphide.h90 petscsnesdef.h90 petscviewer.h90 petscdmda.h90 petscdtdef.h90 petscmat.h90 petscsneshide.h90 petscviewerdef.h90 petscdmdef.h90 petscdthide.h90 petscmatdef.h90 petscts.h90 petscviewerhide.h90 petscdmhide.h90 petscis.h90 petscmathide.h90 petsctsdef.h90 petscdmplex.h90 petscisdef.h90 petscpc.h90 petsctshide.h90 petscdmplexdef.h90 petscishide.h90 petscpcdef.h90 petscvec.h90 petscdmredundant.h90 petscksp.h90 petscpchide.h90 petscvecdef.h90 ./include/petsc/mpiuni: mpi.h mpi.h.html mpif.h mpif.h.html ./include/petsc/private: bagimpl.h drawimpl.h.html matorderimpl.h petscimpl.h.html bagimpl.h.html dtimpl.h matorderimpl.h.html sfimpl.h characteristicimpl.h dtimpl.h.html pcgamgimpl.h sfimpl.h.html characteristicimpl.h.html fortranimpl.h pcgamgimpl.h.html snesimpl.h dmdaimpl.h index.html pcimpl.h snesimpl.h.html dmdaimpl.h.html isimpl.h pcimpl.h.html taoimpl.h dmimpl.h isimpl.h.html pcmgimpl.h taoimpl.h.html dmimpl.h.html kernels pcmgimpl.h.html taolinesearchimpl.h dmmbimpl.h kspimpl.h petscdsimpl.h taolinesearchimpl.h.html dmmbimpl.h.html kspimpl.h.html petscdsimpl.h.html tsimpl.h dmnetworkimpl.h linesearchimpl.h petscfeimpl.h tsimpl.h.html dmnetworkimpl.h.html linesearchimpl.h.html petscfeimpl.h.html vecimpl.h dmpatchimpl.h logimpl.h petscfptimpl.h vecimpl.h.html dmpatchimpl.h.html logimpl.h.html petscfptimpl.h.html viewerimpl.h dmpleximpl.h makefile.html petscfvimpl.h viewerimpl.h.html dmpleximpl.h.html matimpl.h petscfvimpl.h.html drawimpl.h matimpl.h.html petscimpl.h ./include/petsc/private/kernels: blockinvert.h blockmatmult.h blocktranspose.h petscaxpy.h blockinvert.h.html blockmatmult.h.html blocktranspose.h.html petscaxpy.h.html ./lib: libpetsc.a petsc pkgconfig ./lib/petsc: conf ./lib/petsc/conf: PETScConfig.cmake adicmfb.init error.log petscrules uninstall.py RDict.db bfort-base.txt files petscvariables variables adic.init bfort-mpi.txt gmake.log reconfigure-sparc64-unknown-linux-gnu.py adicmastercontrol bfort-petsc.txt make.log rules adicmf.init configure.log modules test ./lib/petsc/conf/modules: petsc ./lib/petsc/conf/modules/petsc: 3.6.3 ./lib/pkgconfig: PETSc.pc ./share: petsc ./share/petsc: datafiles matlab saws ./share/petsc/datafiles: matrices meshes ./share/petsc/datafiles/matrices: amesos2_test_mat0.mtx nh-complex-int32-float32 ns-real-int32-float64 spd-real-int64-float32 hpd-complex-int32-float32 nh-complex-int32-float64 ns-real-int64-float32 spd-real-int64-float64 hpd-complex-int32-float64 nh-complex-int64-float32 ns-real-int64-float64 hpd-complex-int64-float32 nh-complex-int64-float64 spd-real-int32-float32 hpd-complex-int64-float64 ns-real-int32-float32 spd-real-int32-float64 ./share/petsc/datafiles/meshes: Rect-tri3.exo doublet-tet.msh sevenside-quad.exo square.cas squaremotor-30.exo annulus-20.exo grid_c.cgns sevenside.exo square.msh blockcylinder-50.exo sevenside-quad-15.exo simpleblock-100.exo square_bin.msh ./share/petsc/matlab: @PetscOpenFile PetscBinaryWrite.m.html generatehtml.m @PetscOpenSocket PetscReadBinaryMatlab.m generatehtml.m.html PetscBagRead.m PetscReadBinaryMatlab.m.html laplacian.m PetscBagRead.m.html UFgetPetscMat.m laplacian.m.html PetscBinaryRead.m UFgetPetscMat.m.html laplacianlicense.txt PetscBinaryRead.m.html generatePetscTestFiles.m launch.m PetscBinaryWrite.m generatePetscTestFiles.m.html launch.m.html ./share/petsc/matlab/@PetscOpenFile: PetscOpenFile.m close.m read.m write.m ./share/petsc/matlab/@PetscOpenSocket: PetscOpenSocket.m close.m read.m write.m ./share/petsc/saws: documentation images js linearsolveroptions.html readme ./share/petsc/saws/images: arrow.png transition.bmp ./share/petsc/saws/js: PETSc.js drawDiagrams.js listLogic.js parsePrefix.js tree.js boxTree.js events.js main.js populateLists.js treeInterface.js defaults.js getCmdOptions.js matrixTex.js recordSawsData.js utils.js $ ?? ?? ? 2016/03/9??HPCI???????????????? From knepley at gmail.com Sun Apr 24 13:52:35 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Apr 2016 13:52:35 -0500 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> Message-ID: On Sun, Apr 24, 2016 at 10:12 AM, Wee-beng TAY wrote: > Hi Matt, > > I have attached the page which explains what to modify to use PETSc with > the K computer. > These changes are now in knepley/feature-configure-kcomputer, so you cd $PETSC_DIR git fetch git checkout knepley/feature-configure-kcomputer Thanks, Matt > Thanks > > > > On 24/4/2016 10:52 PM, Matthew Knepley wrote: > > On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY wrote: > >> Hi, >> >> I was recently given access to Japan's K computer. I'm trying to compile >> PETSc on their sys, which uses Fujitsu's compiler. There's a help file but >> it's in Japanese. I guess I need to change some options to make PETSc >> compile with Fujitsu's compiler. >> > It looks like they want us to include some different options for the > build. If you send what you want, > I will make a branch for you to try out. > > Thanks, > > Matt > >> Part of the help is below. So how should I use the diff or some other >> command to do the correction? I tried to run: >> >> diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org >> petsc-3.6.4/config/BuildSystem/config/package.py >> >> but got the err: >> >> diff: petsc-3.6.4/config/BuildSystem/config/package.py.org: No such file >> or directory >> >> Please advice. Thanks. >> >> >> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org >> petsc-**3.6**.**3**/config/BuildSystem/config/**package* >> *.py* >> *--- petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org >> **2015**-**12**-**04* *14**:**06**:**42.000000000* >> *+**0900* >> *+++ petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py * >> *2016**-**01**-**22* *11**:**09**:**37.000000000* *+**0900* >> *@@ -**174**,**7* *+**174**,**7* *@@* >> *return* *''* >> >> *def getSharedFlag(self,cflags):* >> *- **for* *flag in [**'-PIC'**, **'-fPIC'**, **'-KPIC'**, **'-qpic'* >> *]:* >> *+ **for* *flag in [**'-KPIC'**, **'-fPIC'**, **'-PIC'**, **'-qpic'* >> *]:* >> *if* *cflags.find(flag) >=**0**: **return* *flag* >> *return* *''* >> >> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org >> petsc-**3.6**.**3* >> */config/BuildSystem/config/setCompilers.py* >> *--- petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org >> **2015**-**07**-**23* *00**:**22**:* >> *46.000000000* *+**0900* >> *+++ petsc-**3.6**.**3* >> */config/BuildSystem/config/setCompilers.py **2016**-**01**-**22* >> *11**:**10**:**05.000000000* *+**0900* >> *@@ -**1017**,**7* *+**1017**,**7* *@@* >> *self.pushLanguage(language)* >> *#different compilers are sensitive to the order of testing these >> flags. So separete out GCC test.* >> *if* *config.setCompilers.Configure.isGNU(self.getCompiler()): >> testFlags = [**'-fPIC'**]* >> *- **else**: testFlags = [**'-PIC'**, **'-fPIC'**, **'-KPIC'**,* >> *'-qpic'**]* >> *+ **else**: testFlags = [**'-KPIC'**, **'-fPIC'**, **'-PIC'**,* >> *'-qpic'**]* >> *for* *testFlag in testFlags:* >> *try**:* >> *self.logPrint(**'Trying '**+language+**' compiler flag '* >> *+testFlag)* >> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org >> petsc-**3.6**.**3* >> */config/BuildSystem/config/packages/openmp.py* >> *--- petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org >> **2016**-**01**-**25* *15**:**42**:* >> *23.000000000**+**0900* >> *+++ petsc-**3.6**.**3* >> */config/BuildSystem/config/packages/openmp.py **2016**-**01**-**22* >> *17**:**13**:**52.000000000* *+**0900* >> *@@ -**19**,**7* *+**19**,**8* *@@* >> *self.found = **0* >> *self.setCompilers.pushLanguage(**'C'**)* >> *#* >> *- **for* *flag in [**"-fopenmp"**, # Gnu* >> *+ **for* *flag in [**"-Kopenmp"**, # Fujitsu* >> *+ **"-fopenmp"**, # Gnu* >> *"-qsmp=omp"**,# IBM XL C/C++* >> *"-h omp"**, # Cray. Must come after XL because XL >> interprets **this* *option as meaning**"-soname omp"* >> *"-mp"**, # Portland Group* >> >> *$ diff -u ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org >> ./petsc-**3.6**.**3* >> */config/BuildSystem/config/compilers.py* >> *--- ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org >> **2015**-**06**-**10* *06**:**24**:* >> *49.000000000* *+**0900* >> *+++ ./petsc-**3.6**.**3* >> */config/BuildSystem/config/compilers.py **2016**-**02**-**19* >> *11**:**56**:**12.000000000* *+**0900* >> *@@ -**164**,**7* *+**164**,**7* *@@* >> *def checkCLibraries(self):* >> *''**'Determines the libraries needed to link with C'**''* >> *oldFlags = self.setCompilers.LDFLAGS* >> *- self.setCompilers.LDFLAGS += **' -v'* >> *+ self.setCompilers.LDFLAGS += **' -###'* >> *self.pushLanguage(**'C'**)* >> *(output, returnCode) = self.outputLink(**''**, **''**)* >> *self.setCompilers.LDFLAGS = oldFlags* >> *@@ -**413**,**7* *+**413**,**7* *@@* >> *def checkCxxLibraries(self):* >> *''**'Determines the libraries needed to link with C++'**''* >> *oldFlags = self.setCompilers.LDFLAGS* >> *- self.setCompilers.LDFLAGS += **' -v'* >> *+ self.setCompilers.LDFLAGS += **' -###'* >> *self.pushLanguage(**'Cxx'**)* >> *(output, returnCode) = self.outputLink(**''**, **''**)* >> *self.setCompilers.LDFLAGS = oldFlags* >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Apr 24 21:31:46 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 25 Apr 2016 10:31:46 +0800 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> Message-ID: <571D8192.2020807@gmail.com> On 25/4/2016 2:52 AM, Matthew Knepley wrote: > On Sun, Apr 24, 2016 at 10:12 AM, Wee-beng TAY > wrote: > > Hi Matt, > > I have attached the page which explains what to modify to use > PETSc with the K computer. > > These changes are now in knepley/feature-configure-kcomputer, so you > > cd $PETSC_DIR > git fetch > git checkout knepley/feature-configure-kcomputer > Hi, I got the warning and error: " --LD_SHARED="" --LDDFLAGS="" --with-openmp=1 --with-mpiexec=mpiexec --known-endian=big --with-shared-libraries=0 --with-blas-lapack-lib="-SSL2" --with-scalapack-lib="-SCALAPACK" --prefix=/home/hp150306/t00196/lib/petsc-3.6.4_rel --with-fortran-interfaces=1 --download-hypre=1 --with-debugging=0 --with-batch=1 =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== *****WARNING: PETSC_DIR appears to be a Git clone - but git is not found in PATH******** =============================================================================== TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:553) Exception AttributeError: AttributeError("Popen3 instance has no attribute 'pid'",) in > ignored ******************************************************************************* UNABLE to EXECUTE BINARIES for ./configure ------------------------------------------------------------------------------- [Errno 12] Cannot allocate memory ******************************************************************************* Any other solution? Thanks! > > Thanks, > > Matt > > Thanks > > > > On 24/4/2016 10:52 PM, Matthew Knepley wrote: >> On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY > > wrote: >> >> Hi, >> >> I was recently given access to Japan's K computer. I'm trying >> to compile PETSc on their sys, which uses Fujitsu's compiler. >> There's a help file but it's in Japanese. I guess I need to >> change some options to make PETSc compile with Fujitsu's >> compiler. >> >> It looks like they want us to include some different options for >> the build. If you send what you want, >> I will make a branch for you to try out. >> >> Thanks, >> >> Matt >> >> Part of the help is below. So how should I use the diff or >> some other command to do the correction? I tried to run: >> >> diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org >> >> petsc-3.6.4/config/BuildSystem/config/package.py >> >> but got the err: >> >> diff: petsc-3.6.4/config/BuildSystem/config/package.py.org >> : No such file or directory >> >> Please advice. Thanks. >> >> >> /|$ diff -u >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org >> >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py|/ >> /|--- >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py.org >> >> |//|2015|//|-|//|12|//|-|//|04|////|14|//|:|//|06|//|:|//|42.000000000|////|+|//|0900|/ >> /|+++ >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/|//|package|//|.py >> |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|09|//|:|//|37.000000000|////|+|//|0900|/ >> /|@@ -|//|174|//|,|//|7|////|+|//|174|//|,|//|7|////|@@|/ >> /||//|return|////|''|/ >> // >> /||//|def getSharedFlag(self,cflags):|/ >> /|- |//|for|////|flag in >> [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]:|/ >> /|+ |//|for|////|flag in >> [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]:|/ >> /||//|if|////|cflags.find(flag) >> >=|//|0|//|:|//|return|////|flag|/ >> /||//|return|////|''|/ >> /||/ >> /|$ diff -u >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org >> >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py|/ >> /|--- >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py.org >> >> |//|2015|//|-|//|07|//|-|//|23|////|00|//|:|//|22|//|:|//|46.000000000|////|+|//|0900|/ >> /|+++ >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/setCompilers.py >> |//|2016|//|-|//|01|//|-|//|22|////|11|//|:|//|10|//|:|//|05.000000000|////|+|//|0900|/ >> /|@@ -|//|1017|//|,|//|7|////|+|//|1017|//|,|//|7|////|@@|/ >> /||//|self.pushLanguage(language)|/ >> /||//|#different compilers are sensitive to the order of >> testing these flags. So separete out GCC test.|/ >> /||//|if|////|config.setCompilers.Configure.isGNU(self.getCompiler()): >> testFlags = [|//|'-fPIC'|//|]|/ >> /|- |//|else|//|: testFlags = >> [|//|'-PIC'|//|,|//|'-fPIC'|//|,|//|'-KPIC'|//|,|//|'-qpic'|//|]|/ >> /|+ |//|else|//|: testFlags = >> [|//|'-KPIC'|//|,|//|'-fPIC'|//|,|//|'-PIC'|//|,|//|'-qpic'|//|]|/ >> /||//|for|////|testFlag in testFlags:|/ >> /||//|try|//|:|/ >> /||//|self.logPrint(|//|'Trying '|//|+language+|//|' compiler >> flag '|//|+testFlag)|/ >> /|$ diff -u >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org >> >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py|/ >> /|--- >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py.org >> >> |//|2016|//|-|//|01|//|-|//|25|////|15|//|:|//|42|//|:|//|23.000000000|//|+|//|0900|/ >> /|+++ >> petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/packages/openmp.py >> |//|2016|//|-|//|01|//|-|//|22|////|17|//|:|//|13|//|:|//|52.000000000|////|+|//|0900|/ >> /|@@ -|//|19|//|,|//|7|////|+|//|19|//|,|//|8|////|@@|/ >> /||//|self.found =|//|0|/ >> /||//|self.setCompilers.pushLanguage(|//|'C'|//|)|/ >> /||//|#|/ >> /|- |//|for|////|flag in [|//|"-fopenmp"|//|, # Gnu|/ >> /|+ |//|for|////|flag in [|//|"-Kopenmp"|//|, # Fujitsu|/ >> /|+ |//|"-fopenmp"|//|, # Gnu|/ >> /||//|"-qsmp=omp"|//|,# IBM XL C/C++|/ >> /||//|"-h omp"|//|, # Cray. Must come after XL because XL >> interprets|//|this|////|option as meaning|//|"-soname omp"|/ >> /||//|"-mp"|//|, # Portland Group|/ >> // >> /|$ diff -u >> ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org >> >> ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py|/ >> /|--- >> ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py.org >> >> |//|2015|//|-|//|06|//|-|//|10|////|06|//|:|//|24|//|:|//|49.000000000|////|+|//|0900|/ >> /|+++ >> ./petsc-|//|3.6|//|.|//|3|//|/config/BuildSystem/config/compilers.py >> |//|2016|//|-|//|02|//|-|//|19|////|11|//|:|//|56|//|:|//|12.000000000|////|+|//|0900|/ >> /|@@ -|//|164|//|,|//|7|////|+|//|164|//|,|//|7|////|@@|/ >> /||//|def checkCLibraries(self):|/ >> /||//|''|//|'Determines the libraries needed to link with >> C'|//|''|/ >> /||//|oldFlags = self.setCompilers.LDFLAGS|/ >> /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ >> /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ >> /||//|self.pushLanguage(|//|'C'|//|)|/ >> /||//|(output, returnCode) = >> self.outputLink(|//|''|//|,|//|''|//|)|/ >> /||//|self.setCompilers.LDFLAGS = oldFlags|/ >> /|@@ -|//|413|//|,|//|7|////|+|//|413|//|,|//|7|////|@@|/ >> /||//|def checkCxxLibraries(self):|/ >> /||//|''|//|'Determines the libraries needed to link with >> C++'|//|''|/ >> /||//|oldFlags = self.setCompilers.LDFLAGS|/ >> /|- self.setCompilers.LDFLAGS +=|//|' -v'|/ >> /|+ self.setCompilers.LDFLAGS +=|//|' -###'|/ >> /||//|self.pushLanguage(|//|'Cxx'|//|)|/ >> /||//|(output, returnCode) = >> self.outputLink(|//|''|//|,|//|''|//|)|/ >> /||//|self.setCompilers.LDFLAGS = oldFlags|/ >> >> >> >> >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 24 21:36:30 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Apr 2016 21:36:30 -0500 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: <571D8192.2020807@gmail.com> References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> <571D8192.2020807@gmail.com> Message-ID: On Sun, Apr 24, 2016 at 9:31 PM, TAY wee-beng wrote: > > On 25/4/2016 2:52 AM, Matthew Knepley wrote: > > On Sun, Apr 24, 2016 at 10:12 AM, Wee-beng TAY wrote: > >> Hi Matt, >> >> I have attached the page which explains what to modify to use PETSc with >> the K computer. >> > These changes are now in knepley/feature-configure-kcomputer, so you > > cd $PETSC_DIR > git fetch > git checkout knepley/feature-configure-kcomputer > > > > Hi, > > I got the warning and error: > > " --LD_SHARED="" --LDDFLAGS="" --with-openmp=1 --with-mpiexec=mpiexec > --known-endian=big --with-shared-libraries=0 --with-blas-lapack-lib="-SSL2" > --with-scalapack-lib="-SCALAPACK" > --prefix=/home/hp150306/t00196/lib/petsc-3.6.4_rel > --with-fortran-interfaces=1 --download-hypre=1 --with-debugging=0 > --with-batch=1 > > =============================================================================== > Configuring PETSc to compile on your system > > =============================================================================== > =============================================================================== > *****WARNING: PETSC_DIR appears to be a Git clone - but git is not found in > PATH******** > =============================================================================== > TESTING: checkCCompiler from > config.setCompilers(config/BuildSystem/config/setCompilers.py:553) > Exception AttributeError: AttributeError("Popen3 instance has no attribute > 'pid'",) in 0x107bbf38>> ignored > > ******************************************************************************* > UNABLE to EXECUTE BINARIES for ./configure > > ------------------------------------------------------------------------------- > [Errno 12] Cannot allocate memory > > ******************************************************************************* > For configure problems, you must send configure.log Thanks, Matt > > Any other solution? > > Thanks! > > > Thanks, > > Matt > > >> Thanks >> >> >> >> On 24/4/2016 10:52 PM, Matthew Knepley wrote: >> >> On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY < >> zonexo at gmail.com> wrote: >> >>> Hi, >>> >>> I was recently given access to Japan's K computer. I'm trying to compile >>> PETSc on their sys, which uses Fujitsu's compiler. There's a help file but >>> it's in Japanese. I guess I need to change some options to make PETSc >>> compile with Fujitsu's compiler. >>> >> It looks like they want us to include some different options for the >> build. If you send what you want, >> I will make a branch for you to try out. >> >> Thanks, >> >> Matt >> >>> Part of the help is below. So how should I use the diff or some other >>> command to do the correction? I tried to run: >>> >>> diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org >>> petsc-3.6.4/config/BuildSystem/config/package.py >>> >>> but got the err: >>> >>> diff: petsc-3.6.4/config/BuildSystem/config/package.py.org: No such >>> file or directory >>> >>> Please advice. Thanks. >>> >>> >>> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org >>> petsc-**3.6**.**3**/config/BuildSystem/config/**package* >>> *.py* >>> *--- petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py.org >>> **2015**-**12**-**04* *14**:**06**:* >>> *42.000000000* *+**0900* >>> *+++ petsc-**3.6**.**3**/config/BuildSystem/config/**package**.py * >>> *2016**-**01**-**22* *11**:**09**:**37.000000000* *+**0900* >>> *@@ -**174**,**7* *+**174**,**7* *@@* >>> *return* *''* >>> >>> *def getSharedFlag(self,cflags):* >>> *- **for* *flag in [**'-PIC'**, **'-fPIC'**, **'-KPIC'**, **'-qpic'* >>> *]:* >>> *+ **for* *flag in [**'-KPIC'**, **'-fPIC'**, **'-PIC'**, **'-qpic'* >>> *]:* >>> *if* *cflags.find(flag) >=**0**: **return* *flag* >>> *return* *''* >>> >>> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org >>> petsc-**3.6**.**3* >>> */config/BuildSystem/config/setCompilers.py* >>> *--- petsc-**3.6**.**3**/config/BuildSystem/config/setCompilers.py.org >>> **2015**-**07**-**23* *00**:**22**:* >>> *46.000000000* *+**0900* >>> *+++ petsc-**3.6**.**3* >>> */config/BuildSystem/config/setCompilers.py **2016**-**01**-**22* >>> *11**:**10**:**05.000000000* *+**0900* >>> *@@ -**1017**,**7* *+**1017**,**7* *@@* >>> *self.pushLanguage(language)* >>> *#different compilers are sensitive to the order of testing >>> these flags. So separete out GCC test.* >>> *if* *config.setCompilers.Configure.isGNU(self.getCompiler()): >>> testFlags = [**'-fPIC'**]* >>> *- **else**: testFlags = [**'-PIC'**, **'-fPIC'**, **'-KPIC'**,* >>> *'-qpic'**]* >>> *+ **else**: testFlags = [**'-KPIC'**, **'-fPIC'**, **'-PIC'**,* >>> *'-qpic'**]* >>> *for* *testFlag in testFlags:* >>> *try**:* >>> *self.logPrint(**'Trying '**+language+**' compiler flag '* >>> *+testFlag)* >>> *$ diff -u petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org >>> petsc-**3.6**.**3* >>> */config/BuildSystem/config/packages/openmp.py* >>> *--- petsc-**3.6**.**3**/config/BuildSystem/config/packages/openmp.py.org >>> **2016**-**01**-**25* *15**:**42**:* >>> *23.000000000**+**0900* >>> *+++ petsc-**3.6**.**3* >>> */config/BuildSystem/config/packages/openmp.py **2016**-**01**-**22* >>> *17**:**13**:**52.000000000* *+**0900* >>> *@@ -**19**,**7* *+**19**,**8* *@@* >>> *self.found = **0* >>> *self.setCompilers.pushLanguage(**'C'**)* >>> *#* >>> *- **for* *flag in [**"-fopenmp"**, # Gnu* >>> *+ **for* *flag in [**"-Kopenmp"**, # Fujitsu* >>> *+ **"-fopenmp"**, # Gnu* >>> *"-qsmp=omp"**,# IBM XL C/C++* >>> *"-h omp"**, # Cray. Must come after XL because XL >>> interprets **this* *option as meaning**"-soname omp"* >>> *"-mp"**, # Portland Group* >>> >>> *$ diff -u ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org >>> ./petsc-**3.6**.**3* >>> */config/BuildSystem/config/compilers.py* >>> *--- ./petsc-**3.6**.**3**/config/BuildSystem/config/compilers.py.org >>> **2015**-**06**-**10* *06**:**24**:* >>> *49.000000000* *+**0900* >>> *+++ ./petsc-**3.6**.**3* >>> */config/BuildSystem/config/compilers.py **2016**-**02**-**19* >>> *11**:**56**:**12.000000000* *+**0900* >>> *@@ -**164**,**7* *+**164**,**7* *@@* >>> *def checkCLibraries(self):* >>> *''**'Determines the libraries needed to link with C'**''* >>> *oldFlags = self.setCompilers.LDFLAGS* >>> *- self.setCompilers.LDFLAGS += **' -v'* >>> *+ self.setCompilers.LDFLAGS += **' -###'* >>> *self.pushLanguage(**'C'**)* >>> *(output, returnCode) = self.outputLink(**''**, **''**)* >>> *self.setCompilers.LDFLAGS = oldFlags* >>> *@@ -**413**,**7* *+**413**,**7* *@@* >>> *def checkCxxLibraries(self):* >>> *''**'Determines the libraries needed to link with C++'**''* >>> *oldFlags = self.setCompilers.LDFLAGS* >>> *- self.setCompilers.LDFLAGS += **' -v'* >>> *+ self.setCompilers.LDFLAGS += **' -###'* >>> *self.pushLanguage(**'Cxx'**)* >>> *(output, returnCode) = self.outputLink(**''**, **''**)* >>> *self.setCompilers.LDFLAGS = oldFlags* >>> >>> >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 24 21:36:58 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 24 Apr 2016 21:36:58 -0500 Subject: [petsc-users] Modifying built files for use with Japan's K computer In-Reply-To: <571D8192.2020807@gmail.com> References: <42f3c825-6e16-8451-d4d8-cef992dbb69a@gmail.com> <571D8192.2020807@gmail.com> Message-ID: <97A740EF-D92D-4D67-84F5-07976DDFB33E@mcs.anl.gov> For configure questions always always send the complete configure.log to petsc-maint at mcs.anl.gov Barry petsc-maint at mcs.anl.gov is the only address that can handle the large files configure.log > On Apr 24, 2016, at 9:31 PM, TAY wee-beng wrote: > > > On 25/4/2016 2:52 AM, Matthew Knepley wrote: >> On Sun, Apr 24, 2016 at 10:12 AM, Wee-beng TAY wrote: >> Hi Matt, >> >> I have attached the page which explains what to modify to use PETSc with the K computer. >> >> These changes are now in knepley/feature-configure-kcomputer, so you >> >> cd $PETSC_DIR >> git fetch >> git checkout knepley/feature-configure-kcomputer >> > > Hi, > > I got the warning and error: > > " --LD_SHARED="" --LDDFLAGS="" --with-openmp=1 --with-mpiexec=mpiexec --known-endian=big --with-shared-libraries=0 --with-blas-lapack-lib="-SSL2" --with-scalapack-lib="-SCALAPACK" --prefix=/home/hp150306/t00196/lib/petsc-3.6.4_rel --with-fortran-interfaces=1 --download-hypre=1 --with-debugging=0 --with-batch=1 > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > =============================================================================== *****WARNING: PETSC_DIR appears to be a Git clone - but git is not found in PATH******** =============================================================================== TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:553) Exception AttributeError: AttributeError("Popen3 instance has no attribute 'pid'",) in > ignored > ******************************************************************************* > UNABLE to EXECUTE BINARIES for ./configure > ------------------------------------------------------------------------------- > [Errno 12] Cannot allocate memory > ******************************************************************************* > > > Any other solution? > > Thanks! >> >> Thanks, >> >> Matt >> >> Thanks >> >> >> On 24/4/2016 10:52 PM, Matthew Knepley wrote: >>> On Sun, Apr 24, 2016 at 4:14 AM, Wee-beng TAY wrote: >>> Hi, >>> >>> I was recently given access to Japan's K computer. I'm trying to compile PETSc on their sys, which uses Fujitsu's compiler. There's a help file but it's in Japanese. I guess I need to change some options to make PETSc compile with Fujitsu's compiler. >>> >>> It looks like they want us to include some different options for the build. If you send what you want, >>> I will make a branch for you to try out. >>> >>> Thanks, >>> >>> Matt >>> Part of the help is below. So how should I use the diff or some other command to do the correction? I tried to run: >>> >>> diff -u petsc-3.6.4/config/BuildSystem/config/package.py.org petsc-3.6.4/config/BuildSystem/config/package.py >>> >>> but got the err: >>> >>> diff: petsc-3.6.4/config/BuildSystem/config/package.py.org: No such file or directory >>> >>> Please advice. Thanks. >>> >>> $ diff -u petsc-3.6.3/config/BuildSystem/config/package.py.org petsc-3.6.3/config/BuildSystem/config/package.py >>> --- petsc-3.6.3/config/BuildSystem/config/package.py.org 2015-12-04 14:06:42.000000000 +0900 >>> +++ petsc-3.6.3/config/BuildSystem/config/package.py 2016-01-22 11:09:37.000000000 +0900 >>> @@ -174,7 +174,7 @@ >>> return '' >>> >>> def getSharedFlag(self,cflags): >>> - for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: >>> + for flag in ['-KPIC', '-fPIC', '-PIC', '-qpic']: >>> if cflags.find(flag) >=0: return flag >>> return '' >>> >>> $ diff -u petsc-3.6.3/config/BuildSystem/config/setCompilers.py.org petsc-3.6.3/config/BuildSystem/config/setCompilers.py >>> --- petsc-3.6.3/config/BuildSystem/config/setCompilers.py.org 2015-07-23 00:22:46.000000000 +0900 >>> +++ petsc-3.6.3/config/BuildSystem/config/setCompilers.py 2016-01-22 11:10:05.000000000 +0900 >>> @@ -1017,7 +1017,7 @@ >>> self.pushLanguage(language) >>> #different compilers are sensitive to the order of testing these flags. So separete out GCC test. >>> if config.setCompilers.Configure.isGNU(self.getCompiler()): testFlags = ['-fPIC'] >>> - else: testFlags = ['-PIC', '-fPIC', '-KPIC','-qpic'] >>> + else: testFlags = ['-KPIC', '-fPIC', '-PIC','-qpic'] >>> for testFlag in testFlags: >>> try: >>> self.logPrint('Trying '+language+' compiler flag '+testFlag) >>> $ diff -u petsc-3.6.3/config/BuildSystem/config/packages/openmp.py.org petsc-3.6.3/config/BuildSystem/config/packages/openmp.py >>> --- petsc-3.6.3/config/BuildSystem/config/packages/openmp.py.org 2016-01-25 15:42:23.000000000+0900 >>> +++ petsc-3.6.3/config/BuildSystem/config/packages/openmp.py 2016-01-22 17:13:52.000000000 +0900 >>> @@ -19,7 +19,8 @@ >>> self.found = 0 >>> self.setCompilers.pushLanguage('C') >>> # >>> - for flag in ["-fopenmp", # Gnu >>> + for flag in ["-Kopenmp", # Fujitsu >>> + "-fopenmp", # Gnu >>> "-qsmp=omp",# IBM XL C/C++ >>> "-h omp", # Cray. Must come after XL because XL interprets this option as meaning"-soname omp" >>> "-mp", # Portland Group >>> >>> $ diff -u ./petsc-3.6.3/config/BuildSystem/config/compilers.py.org ./petsc-3.6.3/config/BuildSystem/config/compilers.py >>> --- ./petsc-3.6.3/config/BuildSystem/config/compilers.py.org 2015-06-10 06:24:49.000000000 +0900 >>> +++ ./petsc-3.6.3/config/BuildSystem/config/compilers.py 2016-02-19 11:56:12.000000000 +0900 >>> @@ -164,7 +164,7 @@ >>> def checkCLibraries(self): >>> '''Determines the libraries needed to link with C''' >>> oldFlags = self.setCompilers.LDFLAGS >>> - self.setCompilers.LDFLAGS += ' -v' >>> + self.setCompilers.LDFLAGS += ' -###' >>> self.pushLanguage('C') >>> (output, returnCode) = self.outputLink('', '') >>> self.setCompilers.LDFLAGS = oldFlags >>> @@ -413,7 +413,7 @@ >>> def checkCxxLibraries(self): >>> '''Determines the libraries needed to link with C++''' >>> oldFlags = self.setCompilers.LDFLAGS >>> - self.setCompilers.LDFLAGS += ' -v' >>> + self.setCompilers.LDFLAGS += ' -###' >>> self.pushLanguage('Cxx') >>> (output, returnCode) = self.outputLink('', '') >>> self.setCompilers.LDFLAGS = oldFlags >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From gbisht at lbl.gov Tue Apr 26 09:50:48 2016 From: gbisht at lbl.gov (Gautam Bisht) Date: Tue, 26 Apr 2016 07:50:48 -0700 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: <87mw9g3eto.fsf@jedbrown.org> References: <87mw9g3eto.fsf@jedbrown.org> Message-ID: I want to follow up on this old thread. If a user knows the exact fill pattern of the off-diagonal block (i.e. d_nz+o_nz or d_nnz+o_nnz ), can one still not preallocate memory for the off-diagonal matrix when using DMComposite? -Gautam. On Tue, Sep 30, 2014 at 3:48 PM, Jed Brown wrote: > Gautam Bisht writes: > > > Hi, > > > > The comment on line 419 of SNES ex 28.c > > < > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex28.c.html > > > > says > > that the approach used in this example is not the best way to allocate > > off-diagonal blocks. Is there an example that shows a better way off > > allocating memory for off-diagonal matrix blocks when using DMComposite? > > The problem is that the allocation interfaces specialize on the matrix > type. Barry wrote a DMCompositeSetCoupling(), but there are no > examples. This is something that PETSc needs to improve. I have been > unsuccessful at conceiving a _simple_ yet flexible/extensible solution. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chengj5 at rpi.edu Tue Apr 26 13:03:03 2016 From: chengj5 at rpi.edu (Jie Cheng) Date: Tue, 26 Apr 2016 14:03:03 -0400 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method Message-ID: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> Hello everyone I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: 223 224 298 297 1 2 76 75 224 225 299 298 2 3 77 76 ? ? 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: 0 0.0011 3.9e-5 2.3677e-5 0.001.9975 3.9e-5 ? ? 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! Best Jie Cheng From bsmith at mcs.anl.gov Tue Apr 26 13:18:16 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 26 Apr 2016 13:18:16 -0500 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> Message-ID: The "trick" is that first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements and then you "renumber" the elements and vertices so that the elements on the first process are numbered first, followed by the elements on the second process etc and similarly the vertices on the first process are numbered before the vertices on the second processes etc. Now each process just needs to loop over its elements compute the element stiffness/load and call MatSetValues/VecSetValues() the the "new" numbering of the vertices. The "old" numbering that was on the disk is simply not used in communicating with PETSc, you only use the new PETSc numbering. Barry > On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: > > Hello everyone > > I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: > > 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. > 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: > 223 224 298 297 1 2 76 75 > 224 225 299 298 2 3 77 76 > ? ? > 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: > 0 0.0011 3.9e-5 > 2.3677e-5 0.001.9975 3.9e-5 > ? ? > > 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. > 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. > 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. > > The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? > > This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! > > Best > Jie Cheng From antoine.deblois at aero.bombardier.com Tue Apr 26 14:25:44 2016 From: antoine.deblois at aero.bombardier.com (Antoine De Blois) Date: Tue, 26 Apr 2016 19:25:44 +0000 Subject: [petsc-users] MatAXPY: appearance of memory leak with DIFFERENT_NONZERO_PATTERN Message-ID: <22B78B7D747CBF4FA36FCD2C6EC7AF5E0101AF7BC8@MTLWAEXCH005.ca.aero.bombardier.net> Hi everyone, I am using petsc-3.6.2 and I get an Out-of-memory error, from what I believe to be a leak from petsc. I am doing successive calls to MatAXPY during an optimization loop. I use MatAXPY to blend 2 matrices together; a linear combination of the my operator and its first-order PC B = w*A + (1-w)*B The matrix operator and PC does not have the same non-zeros patterns, hence the DIFFERENT_NONZERO_PATTERN option. In fact, the non-zeros of the PC are a subset of the non-zeros of A, but I found out about the SUBSET_NONZERO_PATTERN too late... My matrices are pretty big (about 90 million unknown, with 33 non-zeros per row). So after the second call to MatAXPY, I already blow up the memory (on 256 cores). My code snippet would look like: PETSC_Mat_create(&( IMP_Mat_dR2dW) , 5, 0, 1 ,33,22); PETSC_Mat_create(&( IMP_Mat_dR1dW_PC), 5, 0, 1, 19,5); // notice the different non-zeros patterns while(1) { MatZeroEntries( IMP_Mat_dR2dW); MatZeroEntries( IMP_Mat_dR1dW_PC); // fill the dR2dW matrix .... // ... ierr = MatSetValuesBlocked( IMP_Mat_dR2dW, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); // the dRdW_AD_values already contain the w factor // .... MatAssemblyBegin( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY); MatAssemblyEnd( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY); // fill the dR1dW_PC matrix .... // ... ierr = MatSetValuesBlocked( IMP_Mat_dR1dW_PC, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); // ... MatAssemblyBegin( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY); MatAssemblyEnd( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY); // blend the matrices // Out-of-memory appears here MatAXPY( IMP_Mat_dR1dW_PC, 1-w, IMP_Mat_dR2dW, SAME_NONZERO_PATTERN); } // KSPSetOperators // KSPsolve } I looked up the .../petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c at line 2120. It appears that the temporary matrix B is allocated, but never freed. Note that I have not run Valgrind to confirm this leak. For now, I will make the non-zeros of the PC the same as the operator and use SAME_NONZERO_PATTERN. Thank you for your time and efforts, Antoine PETSC error below: [117]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [117]PETSC ERROR: Out of memory. This could be due to allocating [117]PETSC ERROR: too large an object or bleeding by not properly [117]PETSC ERROR: destroying unneeded objects. [117]PETSC ERROR: Memory allocated 0 Memory used by process 8423399424 [117]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [117]PETSC ERROR: Memory requested 704476168 [117]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [117]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [117]PETSC ERROR: /home/ad007804/CODES/FANSC_REPO_after_merge/fansc13.2.1_argus on a ARGUS_impi_opt named node1064 by ad007804 Tue Apr 26 09:34:52 2016 [117]PETSC ERROR: Configure options --CFLAGS=-axAVX --CXXFLAGS="-std=c++11 -DMPICH_IGNORE_CXX_SEEK -DMPICH_IGNORE_CXX_SEEK" --FC=mpiifort --FFLAGS=-axAVX --download-hypre=/home/b0528796/libs/petsc_extpacks/hypre-2.10.1.tar.gz --FC=mpiifort --FFLAGS=-axAVX --download-metis=/home/b0528796/libs/petsc_extpacks/metis-5.1.0-p1.tar.gz --download-ml=/home/b0528796/libs/petsc_extpacks/ml-6.2-win.tar.gz --download-parmetis=/home/b0528796/libs/petsc_extpacks/parmetis-4.0.3-p1.tar.gz --download-suitesparse=/home/b0528796/libs/petsc_extpacks/SuiteSparse-4.4.3.tar.gz --download-superlu_dist --with-blas-lapack-lib=-mkl --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=no --download-mumps=/home/b0528796/libs/petsc_extpacks/MUMPS_5.0.0-p1.tar.gz --with-scalapack-lib="-L/gpfs/fs1/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/gpfs/fs1/intel/mkl/include [117]PETSC ERROR: #1 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c [117]PETSC ERROR: #2 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c [117]PETSC ERROR: #3 MatAXPY_MPIBAIJ() line 2120 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c [117]PETSC ERROR: #4 MatAXPY() line 39 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c Antoine DeBlois Sp?cialiste ing?nierie, MDO lead / Engineering Specialist, MDO lead A?ronautique / Aerospace 514-855-5001, x 50862 antoine.deblois at aero.bombardier.com 2351 Blvd Alfred-Nobel Montreal, Qc H4S 1A9 [Description?: http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg] CONFIDENTIALITY NOTICE - This communication may contain privileged or confidential information. If you are not the intended recipient or received this communication by error, please notify the sender and delete the message without copying -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 4648 bytes Desc: image001.jpg URL: From bsmith at mcs.anl.gov Tue Apr 26 14:32:03 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 26 Apr 2016 14:32:03 -0500 Subject: [petsc-users] MatAXPY: appearance of memory leak with DIFFERENT_NONZERO_PATTERN In-Reply-To: <22B78B7D747CBF4FA36FCD2C6EC7AF5E0101AF7BC8@MTLWAEXCH005.ca.aero.bombardier.net> References: <22B78B7D747CBF4FA36FCD2C6EC7AF5E0101AF7BC8@MTLWAEXCH005.ca.aero.bombardier.net> Message-ID: <73FE5894-F7FF-4D8C-94E1-C7A519F9D06B@mcs.anl.gov> The code ierr = MatHeaderReplace(Y,&B);CHKERRQ(ierr); is suppose to handling freeing up that memory. So I don't think this is directly the problem. You'll need to run with valgrind for example to determine if there is lost memory and where. Is it possible that the matrices just require more memory than you have available? Barry > On Apr 26, 2016, at 2:25 PM, Antoine De Blois wrote: > > Hi everyone, > > I am using petsc-3.6.2 and I get an Out-of-memory error, from what I believe to be a leak from petsc. I am doing successive calls to MatAXPY during an optimization loop. I use MatAXPY to blend 2 matrices together; a linear combination of the my operator and its first-order PC > B = w*A + (1-w)*B > > The matrix operator and PC does not have the same non-zeros patterns, hence the DIFFERENT_NONZERO_PATTERN option. In fact, the non-zeros of the PC are a subset of the non-zeros of A, but I found out about the SUBSET_NONZERO_PATTERN too late? > > My matrices are pretty big (about 90 million unknown, with 33 non-zeros per row). So after the second call to MatAXPY, I already blow up the memory (on 256 cores). My code snippet would look like: > > PETSC_Mat_create(&( IMP_Mat_dR2dW) , 5, 0, 1 ,33,22); > PETSC_Mat_create(&( IMP_Mat_dR1dW_PC), 5, 0, 1, 19,5); // notice the different non-zeros patterns > > while(1) > { > MatZeroEntries( IMP_Mat_dR2dW); > MatZeroEntries( IMP_Mat_dR1dW_PC); > // fill the dR2dW matrix ?. > // ? > ierr = MatSetValuesBlocked( IMP_Mat_dR2dW, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); // the dRdW_AD_values already contain the w factor > // ?. > MatAssemblyBegin( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY); > > // fill the dR1dW_PC matrix ?. > // ? > ierr = MatSetValuesBlocked( IMP_Mat_dR1dW_PC, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); > // ? > MatAssemblyBegin( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY); > > // blend the matrices > // Out-of-memory appears here > MatAXPY( IMP_Mat_dR1dW_PC, > 1-w, > IMP_Mat_dR2dW, > SAME_NONZERO_PATTERN); > } > // KSPSetOperators > // KSPsolve > } > > I looked up the ?/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c at line 2120. It appears that the temporary matrix B is allocated, but never freed. Note that I have not run Valgrind to confirm this leak. > For now, I will make the non-zeros of the PC the same as the operator and use SAME_NONZERO_PATTERN. > > Thank you for your time and efforts, > Antoine > > PETSC error below: > [117]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [117]PETSC ERROR: Out of memory. This could be due to allocating > [117]PETSC ERROR: too large an object or bleeding by not properly > [117]PETSC ERROR: destroying unneeded objects. > [117]PETSC ERROR: Memory allocated 0 Memory used by process 8423399424 > [117]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [117]PETSC ERROR: Memory requested 704476168 > [117]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [117]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 > [117]PETSC ERROR: /home/ad007804/CODES/FANSC_REPO_after_merge/fansc13.2.1_argus on a ARGUS_impi_opt named node1064 by ad007804 Tue Apr 26 09:34:52 2016 > [117]PETSC ERROR: Configure options --CFLAGS=-axAVX --CXXFLAGS="-std=c++11 -DMPICH_IGNORE_CXX_SEEK -DMPICH_IGNORE_CXX_SEEK" --FC=mpiifort --FFLAGS=-axAVX --download-hypre=/home/b0528796/libs/petsc_extpacks/hypre-2.10.1.tar.gz --FC=mpiifort --FFLAGS=-axAVX --download-metis=/home/b0528796/libs/petsc_extpacks/metis-5.1.0-p1.tar.gz --download-ml=/home/b0528796/libs/petsc_extpacks/ml-6.2-win.tar.gz --download-parmetis=/home/b0528796/libs/petsc_extpacks/parmetis-4.0.3-p1.tar.gz --download-suitesparse=/home/b0528796/libs/petsc_extpacks/SuiteSparse-4.4.3.tar.gz --download-superlu_dist --with-blas-lapack-lib=-mkl --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=no --download-mumps=/home/b0528796/libs/petsc_extpacks/MUMPS_5.0.0-p1.tar.gz --with-scalapack-lib="-L/gpfs/fs1/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/gpfs/fs1/intel/mkl/include > [117]PETSC ERROR: #1 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c > [117]PETSC ERROR: #2 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c > [117]PETSC ERROR: #3 MatAXPY_MPIBAIJ() line 2120 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c > [117]PETSC ERROR: #4 MatAXPY() line 39 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c > > Antoine DeBlois > Sp?cialiste ing?nierie, MDO lead / Engineering Specialist, MDO lead > A?ronautique / Aerospace > 514-855-5001, x 50862 > antoine.deblois at aero.bombardier.com > > 2351 Blvd Alfred-Nobel > Montreal, Qc > H4S 1A9 > > > CONFIDENTIALITY NOTICE - This communication may contain privileged or confidential information. > If you are not the intended recipient or received this communication by error, please notify the sender > and delete the message without copying From dave.mayhem23 at gmail.com Tue Apr 26 16:00:45 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 26 Apr 2016 23:00:45 +0200 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: References: <87mw9g3eto.fsf@jedbrown.org> Message-ID: On 26 April 2016 at 16:50, Gautam Bisht wrote: > I want to follow up on this old thread. If a user knows the exact fill > pattern of the off-diagonal block (i.e. d_nz+o_nz or d_nnz+o_nnz ), can > one still not preallocate memory for the off-diagonal matrix when using > DMComposite? > You are always free to over-ride the method dm->ops->creatematrix with your own custom code to create and preallocate the matrix. You will need to add #include into your source file to expose the contents of the DM object. > > -Gautam. > > On Tue, Sep 30, 2014 at 3:48 PM, Jed Brown wrote: > >> Gautam Bisht writes: >> >> > Hi, >> > >> > The comment on line 419 of SNES ex 28.c >> > < >> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex28.c.html >> > >> > says >> > that the approach used in this example is not the best way to allocate >> > off-diagonal blocks. Is there an example that shows a better way off >> > allocating memory for off-diagonal matrix blocks when using DMComposite? >> >> The problem is that the allocation interfaces specialize on the matrix >> type. Barry wrote a DMCompositeSetCoupling(), but there are no >> examples. This is something that PETSc needs to improve. I have been >> unsuccessful at conceiving a _simple_ yet flexible/extensible solution. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 26 16:58:57 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 26 Apr 2016 15:58:57 -0600 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: References: <87mw9g3eto.fsf@jedbrown.org> Message-ID: <87h9eovxpa.fsf@jedbrown.org> Dave May writes: > You are always free to over-ride the method > dm->ops->creatematrix > with your own custom code to create > and preallocate the matrix. DMShellSetCreateMatrix() No need to include the private header. I know this isn't great. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From dave.mayhem23 at gmail.com Tue Apr 26 17:18:11 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 27 Apr 2016 00:18:11 +0200 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: <87h9eovxpa.fsf@jedbrown.org> References: <87mw9g3eto.fsf@jedbrown.org> <87h9eovxpa.fsf@jedbrown.org> Message-ID: On 26 April 2016 at 23:58, Jed Brown wrote: > Dave May writes: > > You are always free to over-ride the method > > dm->ops->creatematrix > > with your own custom code to create > > and preallocate the matrix. > > DMShellSetCreateMatrix() > > No need to include the private header. I know this isn't great. > This always bugged me. I prefer to access the pointer as at least it's clear what I am doing and when reading the code later, I am not required to ask myself whether the DM is actually a shell or not. Why doesn't there exist a generic setter for each object which allows one to set a method for a particular operation? The implementation for Mat defines typedef enum { } MatOperation. Using this, we could have PetscErrorCode MatSetOperation(Mat mat,MatOperation op,zzzzz) If there was a similar typedef enum for all other objects, an XXXSetOperation() would be viable. Is there a good reason to not have such a setter in the library? Thanks, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From chengj5 at rpi.edu Tue Apr 26 18:50:06 2016 From: chengj5 at rpi.edu (Jie Cheng) Date: Tue, 26 Apr 2016 19:50:06 -0400 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> Message-ID: <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> Hi Barry Thanks for your answer. But given the mesh file I have, how do I partition the elements and vertices? MatCreateMPIAdj requires the graph information on adjacent elements (to set up ia and ja). Do I have to use more sophisticated meshing software to create mesh files that have graph information in it? Jie > On Apr 26, 2016, at 2:18 PM, Barry Smith wrote: > > > The "trick" is that first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements and then you "renumber" the elements and vertices so that the elements on the first process are numbered first, followed by the elements on the second process etc and similarly the vertices on the first process are numbered before the vertices on the second processes etc. > > Now each process just needs to loop over its elements compute the element stiffness/load and call MatSetValues/VecSetValues() the the "new" numbering of the vertices. The "old" numbering that was on the disk is simply not used in communicating with PETSc, you only use the new PETSc numbering. > > Barry > >> On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: >> >> Hello everyone >> >> I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: >> >> 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. >> 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: >> 223 224 298 297 1 2 76 75 >> 224 225 299 298 2 3 77 76 >> ? ? >> 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: >> 0 0.0011 3.9e-5 >> 2.3677e-5 0.001.9975 3.9e-5 >> ? ? >> >> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. >> 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. >> 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. >> >> The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? >> >> This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! >> >> Best >> Jie Cheng > From bsmith at mcs.anl.gov Tue Apr 26 18:56:22 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 26 Apr 2016 18:56:22 -0500 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> Message-ID: > On Apr 26, 2016, at 6:50 PM, Jie Cheng wrote: > > Hi Barry > > Thanks for your answer. But given the mesh file I have, how do I partition the elements and vertices? MatCreateMPIAdj requires the graph information on adjacent elements (to set up ia and ja). Do I have to use more sophisticated meshing software to create mesh files that have graph information in it? If you don't have any neighbor information then you cannot do any partitioning. So you need a way to get or create the neighbor information. Barry > > Jie > >> On Apr 26, 2016, at 2:18 PM, Barry Smith wrote: >> >> >> The "trick" is that first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements and then you "renumber" the elements and vertices so that the elements on the first process are numbered first, followed by the elements on the second process etc and similarly the vertices on the first process are numbered before the vertices on the second processes etc. >> >> Now each process just needs to loop over its elements compute the element stiffness/load and call MatSetValues/VecSetValues() the the "new" numbering of the vertices. The "old" numbering that was on the disk is simply not used in communicating with PETSc, you only use the new PETSc numbering. >> >> Barry >> >>> On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: >>> >>> Hello everyone >>> >>> I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: >>> >>> 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. >>> 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: >>> 223 224 298 297 1 2 76 75 >>> 224 225 299 298 2 3 77 76 >>> ? ? >>> 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: >>> 0 0.0011 3.9e-5 >>> 2.3677e-5 0.001.9975 3.9e-5 >>> ? ? >>> >>> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. >>> 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. >>> 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. >>> >>> The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? >>> >>> This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! >>> >>> Best >>> Jie Cheng >> > From stali at geology.wisc.edu Tue Apr 26 19:32:51 2016 From: stali at geology.wisc.edu (Tabrez Ali) Date: Tue, 26 Apr 2016 19:32:51 -0500 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> Message-ID: <572008B3.5090907@geology.wisc.edu> I do something similar, e.g., see lines 41-138 in https://bitbucket.org/stali/defmod/src/default/src/main.F90 which you can just copy/paste. Using DMPlex might be a better option in the long run. Tabrez On 04/26/2016 06:50 PM, Jie Cheng wrote: > Hi Barry > > Thanks for your answer. But given the mesh file I have, how do I partition the elements and vertices? MatCreateMPIAdj requires the graph information on adjacent elements (to set up ia and ja). Do I have to use more sophisticated meshing software to create mesh files that have graph information in it? > > Jie > >> On Apr 26, 2016, at 2:18 PM, Barry Smith wrote: >> >> >> The "trick" is that first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements and then you "renumber" the elements and vertices so that the elements on the first process are numbered first, followed by the elements on the second process etc and similarly the vertices on the first process are numbered before the vertices on the second processes etc. >> >> Now each process just needs to loop over its elements compute the element stiffness/load and call MatSetValues/VecSetValues() the the "new" numbering of the vertices. The "old" numbering that was on the disk is simply not used in communicating with PETSc, you only use the new PETSc numbering. >> >> Barry >> >>> On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: >>> >>> Hello everyone >>> >>> I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: >>> >>> 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. >>> 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: >>> 223 224 298 297 1 2 76 75 >>> 224 225 299 298 2 3 77 76 >>> ? ? >>> 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: >>> 0 0.0011 3.9e-5 >>> 2.3677e-5 0.001.9975 3.9e-5 >>> ? ? >>> >>> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. >>> 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. >>> 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. >>> >>> The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? >>> >>> This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! >>> >>> Best >>> Jie Cheng From chengj5 at rpi.edu Tue Apr 26 23:00:42 2016 From: chengj5 at rpi.edu (Jie Cheng) Date: Wed, 27 Apr 2016 00:00:42 -0400 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> Message-ID: <3BA76B8D-93F1-4814-A1F9-9DAEFFA42445@rpi.edu> Hello Barry It turns out METIS can easily generate the necessary graph information from the type of connectivity file I described with mesh2dual and mesh2nodal commands. But I do not fully understand the trick you talked about: Suppose I have 10 elements and I am running my program with 2 MPI ranks. 1) I should let rank 0 take care of the first to the 5th element, and rank 1 take care of the 6th to the 10th element and let PETSc do the real partitioning? 2) I have the necessary graph information, but how do I partition the elements and the vertices? Call MatCreateMPIAdj twice to create two isgs? I do not understand what you meant by ?partition the vertices (nodal values) subservient to the partitioning of the elements?. 3) To set values in the global stiffness matrix I need to know the global element number and global vertex number, how to get them? I have to ask these in detail because the manual seems to be too brief about the approach. Thanks for your patience. Best Jie > On Apr 26, 2016, at 7:56 PM, Barry Smith wrote: > > >> On Apr 26, 2016, at 6:50 PM, Jie Cheng wrote: >> >> Hi Barry >> >> Thanks for your answer. But given the mesh file I have, how do I partition the elements and vertices? MatCreateMPIAdj requires the graph information on adjacent elements (to set up ia and ja). Do I have to use more sophisticated meshing software to create mesh files that have graph information in it? > > If you don't have any neighbor information then you cannot do any partitioning. So you need a way to get or create the neighbor information. > > Barry > >> >> Jie >> >>> On Apr 26, 2016, at 2:18 PM, Barry Smith wrote: >>> >>> >>> The "trick" is that first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements and then you "renumber" the elements and vertices so that the elements on the first process are numbered first, followed by the elements on the second process etc and similarly the vertices on the first process are numbered before the vertices on the second processes etc. >>> >>> Now each process just needs to loop over its elements compute the element stiffness/load and call MatSetValues/VecSetValues() the the "new" numbering of the vertices. The "old" numbering that was on the disk is simply not used in communicating with PETSc, you only use the new PETSc numbering. >>> >>> Barry >>> >>>> On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: >>>> >>>> Hello everyone >>>> >>>> I have a finite element code to solve nonlinear solid mechanics (using Newton-Raphson iteration) implemented with PETSc. The code is serial, organized as following: >>>> >>>> 1) Read the connectivity of the unstructured mesh, coordinates of nodes from individual txt files. >>>> 1.1) Connectivity file contains [# of elements] rows, and each row lists the global number of nodes on that element. Take 3d hexahedral elements for instance: >>>> 223 224 298 297 1 2 76 75 >>>> 224 225 299 298 2 3 77 76 >>>> ? ? >>>> 1.2) Coordinates file contains [# of nodes] rows, and each row lists the coordinates of a node, for example: >>>> 0 0.0011 3.9e-5 >>>> 2.3677e-5 0.001.9975 3.9e-5 >>>> ? ? >>>> >>>> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since the dimensions and nonzero pattern are known from the connectivity. >>>> 3) Loop over the element to compute the element stiffness matrix and right hand side. Then assemble the global stiffness matrix and right hand side. >>>> 4) Solve the linear equation with KSPsolve for the displacement increment, then go back to Step 3. >>>> >>>> The code works fine in serial, now I?m trying to parallelize it. To partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. Either way I will find a way to assign different elements and nodes to different ranks. My question is: since PETSc does not allow us to control the memory distribution of the parallel matrix/vector, how do I make sure the rank happens to have all/most memory it needs for the specific elements? For example, rank 0 is in charged of element n, and needs to insert values to A[ i ][ j ], how do I make sure the i-th row is assigned to rank 0? >>>> >>>> This question is fundamental for people work with finite element methods. I checked the tutorial codes but did not find an example to follow. Section 3.5 of the manual talks about partition, but it does not say anything about linking the partition with the memory distribution. Could anyone give me some instructions on this issue? Thank you in advance! >>>> >>>> Best >>>> Jie Cheng >>> >> > From norihiro.w at gmail.com Tue Apr 26 23:08:44 2016 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Wed, 27 Apr 2016 04:08:44 +0000 Subject: [petsc-users] SAMG support Message-ID: Hi, Is SAMG, a commercial multigrid solver, not supported in PETSc anymore? It seems an interface to the library was somehow implemented in PETSc v2. but I cannot find it in the latest version. I guess its support was dropped out already but want to make it sure. Best, Nori -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 26 23:23:44 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 26 Apr 2016 23:23:44 -0500 Subject: [petsc-users] SAMG support In-Reply-To: References: Message-ID: Yes its removed.. https://bitbucket.org/petsc/petsc/commits/758d9a8da9b91e4064de3b63c5dd04fe789a8049 Satish On Tue, 26 Apr 2016, Norihiro Watanabe wrote: > Hi, > > Is SAMG, a commercial multigrid solver, not supported in PETSc anymore? It > seems an interface to the library was somehow implemented in PETSc v2. but > I cannot find it in the latest version. I guess its support was dropped out > already but want to make it sure. > > > Best, > Nori > From norihiro.w at gmail.com Tue Apr 26 23:40:17 2016 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Wed, 27 Apr 2016 06:40:17 +0200 Subject: [petsc-users] SAMG support In-Reply-To: References: Message-ID: Thanks! On Wed, Apr 27, 2016 at 6:23 AM, Satish Balay wrote: > Yes its removed.. > > https://bitbucket.org/petsc/petsc/commits/758d9a8da9b91e4064de3b63c5dd04fe789a8049 > > Satish > > On Tue, 26 Apr 2016, Norihiro Watanabe wrote: > >> Hi, >> >> Is SAMG, a commercial multigrid solver, not supported in PETSc anymore? It >> seems an interface to the library was somehow implemented in PETSc v2. but >> I cannot find it in the latest version. I guess its support was dropped out >> already but want to make it sure. >> >> >> Best, >> Nori >> > -- Norihiro Watanabe From knepley at gmail.com Wed Apr 27 06:43:20 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Apr 2016 06:43:20 -0500 Subject: [petsc-users] Memory Distribution and Mesh Partition in Finite Element Method In-Reply-To: <3BA76B8D-93F1-4814-A1F9-9DAEFFA42445@rpi.edu> References: <4F96BABC-1902-4059-94F5-D65D8DEB4BEE@rpi.edu> <06AF1F47-6D5F-4024-945E-6429ED51E22F@rpi.edu> <3BA76B8D-93F1-4814-A1F9-9DAEFFA42445@rpi.edu> Message-ID: On Tue, Apr 26, 2016 at 11:00 PM, Jie Cheng wrote: > Hello Barry > > It turns out METIS can easily generate the necessary graph information > from the type of connectivity file I described with mesh2dual and > mesh2nodal commands. But I do not fully understand the trick you talked > about: > > Suppose I have 10 elements and I am running my program with 2 MPI ranks. > > 1) I should let rank 0 take care of the first to the 5th element, and rank > 1 take care of the 6th to the 10th element and let PETSc do the real > partitioning? > No, Barry says "first partition the element across processes, then partition the vertices (nodal values) subservient to the partitioning of the elements" meaning only give a vertex to a process if it already owns the element containing this vertex. > 2) I have the necessary graph information, but how do I partition the > elements and the vertices? Call MatCreateMPIAdj twice to create two isgs? I > do not understand what you meant by ?partition the vertices (nodal values) > subservient to the partitioning of the elements?. > If you used PETSc to do it, you would create an MPIAdj which the element adjacency information (which I assume you got from mesh2dual). We would partition the elements by calling parmetis underneath. Then it would be up to you to partition vertices, just as it is when calling parmetis by hand. > 3) To set values in the global stiffness matrix I need to know the global > element number and global vertex number, how to get them? > You, of course, start with these numbers. As Barry says, in order to make the dofs contiguous you should renumber the cells and vertices so that each process owns a contiguous block. Thanks, Matt > I have to ask these in detail because the manual seems to be too brief > about the approach. Thanks for your patience. > > Best > Jie > > > > On Apr 26, 2016, at 7:56 PM, Barry Smith wrote: > > > > > >> On Apr 26, 2016, at 6:50 PM, Jie Cheng wrote: > >> > >> Hi Barry > >> > >> Thanks for your answer. But given the mesh file I have, how do I > partition the elements and vertices? MatCreateMPIAdj requires the graph > information on adjacent elements (to set up ia and ja). Do I have to use > more sophisticated meshing software to create mesh files that have graph > information in it? > > > > If you don't have any neighbor information then you cannot do any > partitioning. So you need a way to get or create the neighbor information. > > > > Barry > > > >> > >> Jie > >> > >>> On Apr 26, 2016, at 2:18 PM, Barry Smith wrote: > >>> > >>> > >>> The "trick" is that first partition the element across processes, then > partition the vertices (nodal values) subservient to the partitioning of > the elements and then you "renumber" the elements and vertices so that the > elements on the first process are numbered first, followed by the elements > on the second process etc and similarly the vertices on the first process > are numbered before the vertices on the second processes etc. > >>> > >>> Now each process just needs to loop over its elements compute the > element stiffness/load and call MatSetValues/VecSetValues() the the "new" > numbering of the vertices. The "old" numbering that was on the disk is > simply not used in communicating with PETSc, you only use the new PETSc > numbering. > >>> > >>> Barry > >>> > >>>> On Apr 26, 2016, at 1:03 PM, Jie Cheng wrote: > >>>> > >>>> Hello everyone > >>>> > >>>> I have a finite element code to solve nonlinear solid mechanics > (using Newton-Raphson iteration) implemented with PETSc. The code is > serial, organized as following: > >>>> > >>>> 1) Read the connectivity of the unstructured mesh, coordinates of > nodes from individual txt files. > >>>> 1.1) Connectivity file contains [# of elements] rows, and each row > lists the global number of nodes on that element. Take 3d hexahedral > elements for instance: > >>>> 223 224 298 297 1 2 76 75 > >>>> 224 225 299 298 2 3 77 76 > >>>> ? ? > >>>> 1.2) Coordinates file contains [# of nodes] rows, and each row lists > the coordinates of a node, for example: > >>>> 0 0.0011 3.9e-5 > >>>> 2.3677e-5 0.001.9975 3.9e-5 > >>>> ? ? > >>>> > >>>> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since > the dimensions and nonzero pattern are known from the connectivity. > >>>> 3) Loop over the element to compute the element stiffness matrix and > right hand side. Then assemble the global stiffness matrix and right hand > side. > >>>> 4) Solve the linear equation with KSPsolve for the displacement > increment, then go back to Step 3. > >>>> > >>>> The code works fine in serial, now I?m trying to parallelize it. To > partition the mesh, I can use partdmesh from METIS, or let PETSc calls it. > Either way I will find a way to assign different elements and nodes to > different ranks. My question is: since PETSc does not allow us to control > the memory distribution of the parallel matrix/vector, how do I make sure > the rank happens to have all/most memory it needs for the specific > elements? For example, rank 0 is in charged of element n, and needs to > insert values to A[ i ][ j ], how do I make sure the i-th row is assigned > to rank 0? > >>>> > >>>> This question is fundamental for people work with finite element > methods. I checked the tutorial codes but did not find an example to > follow. Section 3.5 of the manual talks about partition, but it does not > say anything about linking the partition with the memory distribution. > Could anyone give me some instructions on this issue? Thank you in advance! > >>>> > >>>> Best > >>>> Jie Cheng > >>> > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Wed Apr 27 10:17:34 2016 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Wed, 27 Apr 2016 17:17:34 +0200 Subject: [petsc-users] DMDACreateSection Message-ID: <5720D80E.90909@math.u-psud.fr> Hi, I would like to know if it is possible to use DMDACreateSection to create a grid for Stokes problem for example using Q2-Q1 finite element discretization. I read old threads that tell that the implementation was not yet finished. And there is some comments in the code like "this is wrong" that tell me that it may be not available. It could be great to use DMDA like DMPlex. I use dmcomposite to do two grids but sections seems to be more easy to use and probably more efficient for the creation of the non zero structure of the matrix that solves the problem. Best, Loic From knepley at gmail.com Wed Apr 27 10:21:30 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Apr 2016 10:21:30 -0500 Subject: [petsc-users] DMDACreateSection In-Reply-To: <5720D80E.90909@math.u-psud.fr> References: <5720D80E.90909@math.u-psud.fr> Message-ID: On Wed, Apr 27, 2016 at 10:17 AM, gouarin wrote: > Hi, > > I would like to know if it is possible to use DMDACreateSection to create > a grid for Stokes problem for example using Q2-Q1 finite element > discretization. I read old threads that tell that the implementation was > not yet finished. And there is some comments in the code like "this is > wrong" that tell me that it may be not available. > > It could be great to use DMDA like DMPlex. > I agree, but it was a lot of work to keep the implementation up, and I am way behind. > I use dmcomposite to do two grids but sections seems to be more easy to > use and probably more efficient for the creation of the non zero structure > of the matrix that solves the problem. > I have switched to just using Plex for these things, since the data in the mesh is pretty small compared to the solution, even if I represent the entire Cartesian grid. I will also note that I think this is correct for the future as well since the DMForest implementation (the wrapper Toby Isaac wrote for p4est) uses PetscSection as well, is memory efficient, and does structured AMR. I am switching everything I had on DMDA to use this. If you want help in this direction, of course I am at your service. I am currently getting my magma dynamics code to work in this way. Thanks, Matt > Best, > Loic > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Wed Apr 27 13:15:49 2016 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Wed, 27 Apr 2016 20:15:49 +0200 Subject: [petsc-users] DMDACreateSection In-Reply-To: References: <5720D80E.90909@math.u-psud.fr> Message-ID: <572101D5.2010208@math.u-psud.fr> Le 27/04/2016 17:21, Matthew Knepley a ?crit : > On Wed, Apr 27, 2016 at 10:17 AM, gouarin > wrote: > > Hi, > > I would like to know if it is possible to use DMDACreateSection to > create a grid for Stokes problem for example using Q2-Q1 finite > element discretization. I read old threads that tell that the > implementation was not yet finished. And there is some comments in > the code like "this is wrong" that tell me that it may be not available. > > It could be great to use DMDA like DMPlex. > > > I agree, but it was a lot of work to keep the implementation up, and I > am way behind. > > I use dmcomposite to do two grids but sections seems to be more easy > to use and probably more efficient for the creation of the non zero > structure of the matrix that solves the problem. > > > I have switched to just using Plex for these things, since the data in > the mesh is pretty small compared > to the solution, even if I represent the entire Cartesian grid. > > I will also note that I think this is correct for the future as well > since the DMForest implementation (the > wrapper Toby Isaac wrote for p4est) uses PetscSection as well, is memory > efficient, and does structured > AMR. I am switching everything I had on DMDA to use this. If you want > help in this direction, of course > I am at your service. I am currently getting my magma dynamics code to > work in this way. > Thanks Matt for your answer. I will switch to Plex then. Loic > Thanks, > > Matt > > Best, > Loic > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Loic Gouarin Laboratoire de Math?matiques Universit? Paris-Sud B?timent 425 91405 Orsay Cedex France Tel: (+33) 1 69 15 60 14 Fax: (+33) 1 69 15 67 18 From jed at jedbrown.org Wed Apr 27 15:49:40 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 27 Apr 2016 14:49:40 -0600 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: References: <87mw9g3eto.fsf@jedbrown.org> <87h9eovxpa.fsf@jedbrown.org> Message-ID: <874mamvkt7.fsf@jedbrown.org> Dave May writes: > This always bugged me. > I prefer to access the pointer as at least it's clear what I am doing and > when reading the code later, I am not required to ask myself whether the DM > is actually a shell or not. > > Why doesn't there exist a generic setter for each object which allows one > to set a method for a particular operation? You're just objecting to the function being named DMShellSetCreateMatrix instead of DMSetCreateMatrix? > The implementation for Mat defines typedef enum { } MatOperation. > Using this, we could have > PetscErrorCode MatSetOperation(Mat mat,MatOperation op,zzzzz) > > If there was a similar typedef enum for all other objects, an > XXXSetOperation() would be viable. The MatOperation enums are a bit of a maintenance burden and don't offer any type checking. We have them for Mat and Vec (sort of) because there are so many methods. We could add them for other objects, but would give up type checking relative to the existing specialized functions. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From doougsini at gmail.com Wed Apr 27 17:27:27 2016 From: doougsini at gmail.com (Seungbum Koo) Date: Wed, 27 Apr 2016 17:27:27 -0500 Subject: [petsc-users] MatAssemblyBegin does not return. Message-ID: Is there anyone who is experiencing the same problem as mine? My code worked fine with PETSc-3.4.5 but the same code does not work after upgrading to 3.6.3. Call to MatAssemblyBegin does not return.I waited 10+ hours and quit it. It used to return within a minute with 3.4.5. When I downgrade to 3.4.5, everything works fine again. Is there anyone had the same problem and resolved? Thanks in advance. Seungbum -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 27 17:29:29 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Apr 2016 17:29:29 -0500 Subject: [petsc-users] MatAssemblyBegin does not return. In-Reply-To: References: Message-ID: On Wed, Apr 27, 2016 at 5:27 PM, Seungbum Koo wrote: > Is there anyone who is experiencing the same problem as mine? > > My code worked fine with PETSc-3.4.5 but the same code does not work after > upgrading to 3.6.3. Call to MatAssemblyBegin does not return.I waited 10+ > hours and quit it. It used to return within a minute with 3.4.5. When I > downgrade to 3.4.5, everything works fine again. > Can you run in the debugger and get a stack trace, so that we can see where it might be hanging? We have not had any other reports of this, and 3.6 has been out for a year. Thanks, Matt > Is there anyone had the same problem and resolved? > > Thanks in advance. > Seungbum > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zocca.marco at gmail.com Wed Apr 27 20:16:47 2016 From: zocca.marco at gmail.com (Marco Zocca) Date: Thu, 28 Apr 2016 03:16:47 +0200 Subject: [petsc-users] petsc 3.7 and slepc 3.6.1 incompatibility Message-ID: Dear All, first, let me congratulate and thank the PETSc team and all contributors for release 3.7 . I just noticed that the changes in Options handling introduced in 3.7 broke the build of the latest SLEPc (3.6.1, since 3.6.3 is flagged as incompatible with 3.7). Is there already a time estimate for a compatible release of SLEPc ? Thank you and kind regards, Marco Zocca From hzhang at mcs.anl.gov Wed Apr 27 21:00:11 2016 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 27 Apr 2016 21:00:11 -0500 Subject: [petsc-users] petsc 3.7 and slepc 3.6.1 incompatibility In-Reply-To: References: Message-ID: Marco: You may use petsc-master and slepc-master. Hong Dear All, > > first, let me congratulate and thank the PETSc team and all > contributors for release 3.7 . > > I just noticed that the changes in Options handling introduced in 3.7 > broke the build of the latest SLEPc (3.6.1, since 3.6.3 is flagged as > incompatible with 3.7). > > Is there already a time estimate for a compatible release of SLEPc ? > > Thank you and kind regards, > > Marco Zocca > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Apr 28 00:14:37 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 28 Apr 2016 07:14:37 +0200 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: <874mamvkt7.fsf@jedbrown.org> References: <87mw9g3eto.fsf@jedbrown.org> <87h9eovxpa.fsf@jedbrown.org> <874mamvkt7.fsf@jedbrown.org> Message-ID: On 27 April 2016 at 22:49, Jed Brown wrote: > Dave May writes: > > This always bugged me. > > I prefer to access the pointer as at least it's clear what I am doing and > > when reading the code later, I am not required to ask myself whether the > DM > > is actually a shell or not. > > > > Why doesn't there exist a generic setter for each object which allows one > > to set a method for a particular operation? > > You're just objecting to the function being named DMShellSetCreateMatrix > instead of DMSetCreateMatrix? > Absolutely! Its inconsistent with nearly all other PETSc methods. With all other methods, this function would only take affect if the DM was of type shell, e.g through using either PetscTryMethod(), or by using PetscObjectTpeCompare(). This is way it is done in PCShell. It is also inconsistent with some of the other DMShellSetXXX() as some type check the type (e.g. DMShellSetCreateMatrix), whilst some do, e.g. DMShellSetCoarsen(). > > > The implementation for Mat defines typedef enum { } MatOperation. > > Using this, we could have > > PetscErrorCode MatSetOperation(Mat mat,MatOperation op,zzzzz) > > > > If there was a similar typedef enum for all other objects, an > > XXXSetOperation() would be viable. > > The MatOperation enums are a bit of a maintenance burden and don't offer > any type checking. We have them for Mat and Vec (sort of) because there > are so many methods. We could add them for other objects, but would > give up type checking relative to the existing specialized functions. > Agreed, we shouldn't give up on type checking. But type checking is ignored with MatShellSetOperation()... This function seems to do approximately what I proposed. I'm I could autogenerate the functions from the op structure so it wouldn't be a huge burden (just a lot of code). Could we at least make the (i) XXXShell objects APIs consistent with each other in terms of how methods/operations are set, (ii) change all the shell setter to only take affect if the type was "shell" and (iii) add type checked setters for Mat, PC and DM (which appear to be the only methods with shell implementations). I'd be happy to do this is everyone approves. Cheers Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Apr 28 00:17:57 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 28 Apr 2016 07:17:57 +0200 Subject: [petsc-users] Allocating memory for off-diagonal matrix blocks when using DMComposite In-Reply-To: References: <87mw9g3eto.fsf@jedbrown.org> <87h9eovxpa.fsf@jedbrown.org> <874mamvkt7.fsf@jedbrown.org> Message-ID: > It is also inconsistent with some of the other DMShellSetXXX() as some > type check the type (e.g. DMShellSetCreateMatrix), whilst some do, e.g. > DMShellSetCoarsen(). > > ooops - I meant some setters DO check types and some DON'T DMShellSetCreateMatrix() --> doesn't check the type DMShellSetCoarsen() --> DOES check the type -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Apr 28 03:24:11 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 28 Apr 2016 10:24:11 +0200 Subject: [petsc-users] petsc 3.7 and slepc 3.6.1 incompatibility In-Reply-To: References: Message-ID: > El 28 abr 2016, a las 4:00, Hong escribi?: > > Marco: > You may use petsc-master and slepc-master. > Hong > > Dear All, > > first, let me congratulate and thank the PETSc team and all > contributors for release 3.7 . > > I just noticed that the changes in Options handling introduced in 3.7 > broke the build of the latest SLEPc (3.6.1, since 3.6.3 is flagged as > incompatible with 3.7). > > Is there already a time estimate for a compatible release of SLEPc ? > > Thank you and kind regards, > > Marco Zocca > Well, all SLEPc 3.6.x (also 3.6.1) are incompatible with 3.7, but only 3.6.3 checks this explicitly. SLEPc 3.7 will be released in a couple of weeks. Stay tuned via the mailing list. Meanwhile you can use the master branch as suggested by Hong. Jose From ilyascfd at gmail.com Fri Apr 29 07:12:50 2016 From: ilyascfd at gmail.com (Ilyas YILMAZ) Date: Fri, 29 Apr 2016 15:12:50 +0300 Subject: [petsc-users] Averaging 3D DA global vector Message-ID: Hello, Somewhere in my code, I need to take spatial average of a global vector created by 3D DA context and store it into another global vector. It is like X3D(streamwise,normalwise,spanwise) --> Y3D(0,normalwise,0). It is very easy to do serially in Fortran, for example, tot = (ni-2)*(nk-2) do k= 2,nkm1 do j= 2,njm1 do i= 2,nim1 su(2,j,2)=su(2,j,2)+cs(i,j,k)/float(tot) end do end do end do What could be the most efficient way of doing this in PETSC? or Is it possible to do as follows using ADD_VALUES? mxmz = mx*mz do k=zs,zs+zm-1 do j=ys,ys+ym-1 do i=xs,xs+xm-1 row = i - gxs + (j - gys)*gxm + (k - gzs)*gxm*gym + 1 rowj = xs - gxs + (j - gys)*gxm + (zs - gzs)*gxm*gym + 1 scaX3DAVE(idX3DAVE+*rowj*) = scaX3DAVE(idX3DAVE+*rowj*) + scaX3D(idX3D+*row*)/float(mx*mz) end do end do end do call VecRestoreArray() call DALocaltoGlobal(da,localX3DAVE,*ADD_VALUES*,X3DAVE,ierr) Thanks. IY -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlc at illinois.edu Fri Apr 29 07:34:43 2016 From: vlc at illinois.edu (Le Chenadec, Vincent) Date: Fri, 29 Apr 2016 12:34:43 +0000 Subject: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL off-diagonal blocks Message-ID: <3664C6DEB0AB9C429B69017A6483BACD45B7B07B@CHIMBX6.ad.uillinois.edu> Dear all, I'm trying to use PCFIELDSPLIT as a preconditioner on a block matrix A, [A_{00} A_{01}] [A_{10} A_{11}] where the diagonal blocks (A_{00} and A_{11}) are diagonally dominant. I would like to use either one of the block preconditioners (not even based on the Schur complement, simply either one of the Jacobi/Gauss-Seidel ones). The issue I'm running into concerns the formats I would like to use, so I'll start with that: 1. A_{00} and A_{11} are both in MATMPIAIJ format (created using DMCreateMatrix on 2D DMDAs of different sizes), 2. A_{10} and A_{01} are both in MATSHELL format (created using DMCreateShell). The reason why I used this format is that it seemed convenient, since the 2D DMDAs are of different sizes and I assumed I had to provide the MATOP_MULT operation only, 3. Finally the blocks are stored in MATNEST format (MatCreateNest). With this choice, matrix-vector multiplications work fine (the DMDAs being packed in a DMComposite), which makes me think that so far the code is ok. Also, when I attempt to invert the matrix without preconditioner, I get the correct result. Now here is where things do not work: if I try to use a preconditioner, I get the error pasted below (for a simple PCJACOBI) [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Mat type shell [0]PETSC ERROR: #1 MatGetDiagonal() line 4306 in petsc/3.6.3_dbg/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_Jacobi() line 180 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #3 PCSetUp_Jacobi_NonSymmetric() line 261 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #4 PCApply_Jacobi() line 286 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #5 PCApply() line 483 in petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSP_PCApply() line 242 in petsc/3.6.3_dbg/include/petsc/private/kspimpl.h [0]PETSC ERROR: #7 KSPSolve_CG() line 139 in petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: #8 KSPSolve() line 604 in petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 PCApply_FieldSplit() line 988 in petsc/3.6.3_dbg/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #10 PCApply() line 483 in petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #11 KSP_PCApply() line 242 in petsc/3.6.3_dbg/include/petsc/private/kspimpl.h [0]PETSC ERROR: #12 KSPSolve_CG() line 139 in petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: #13 KSPSolve() line 604 in petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c I may not be calling the appropriate routines, here is what I'm basically doing: 1. KSP type set to KSPCG (A is symmetric), PC type to PCFIELDSPLIT, 2. I then use MatNestGetISs on A to supply the ISs to PCFieldSplitSetIS, 3. Finally I use PCFieldSplitGetSubKSP to use set the subKSP for each diagonal block. As I mentioned, as long as the subKSPs don't have any preconditioning (PCNONE), KSPSolve does not complain, only when I try to using one (even PCJACOBI) do I get similar errors). That makes me suspect that I'm doing something wrong, but I could not find an example that was doing the same thing... Any pointers would be much appreciated! Thanks, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 29 07:42:16 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 29 Apr 2016 07:42:16 -0500 Subject: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL off-diagonal blocks In-Reply-To: <3664C6DEB0AB9C429B69017A6483BACD45B7B07B@CHIMBX6.ad.uillinois.edu> References: <3664C6DEB0AB9C429B69017A6483BACD45B7B07B@CHIMBX6.ad.uillinois.edu> Message-ID: On Fri, Apr 29, 2016 at 7:34 AM, Le Chenadec, Vincent wrote: > Dear all, > > I'm trying to use PCFIELDSPLIT as a preconditioner on a block matrix A, > > [A_{00} A_{01}] > [A_{10} A_{11}] > > where the diagonal blocks (A_{00} and A_{11}) are diagonally dominant. I > would like to use either one of the block preconditioners (not even based > on the Schur complement, simply either one of the Jacobi/Gauss-Seidel ones). > For any questions about solvers, we need to see the output of -ksp_view. Below, it seems clear that either a) One of your diagonal blocks is a MatShell, or b) you are trying to invert an off-diagonal block. Matt > The issue I'm running into concerns the formats I would like to use, so > I'll start with that: > > 1. A_{00} and A_{11} are both in MATMPIAIJ format (created using > DMCreateMatrix on 2D DMDAs of different sizes), > 2. A_{10} and A_{01} are both in MATSHELL format (created using > DMCreateShell). The reason why I used this format is that it seemed > convenient, since the 2D DMDAs are of different sizes and I assumed I had > to provide the MATOP_MULT operation only, > 3. Finally the blocks are stored in MATNEST format (MatCreateNest). > > With this choice, matrix-vector multiplications work fine (the DMDAs being > packed in a DMComposite), which makes me think that so far the code is ok. > Also, when I attempt to invert the matrix without preconditioner, I get > the correct result. > > Now here is where things do not work: if I try to use a preconditioner, I > get the error pasted below (for a simple PCJACOBI) > > [0]PETSC ERROR: No support for this operation for this object type > > [0]PETSC ERROR: Mat type shell > > [0]PETSC ERROR: #1 MatGetDiagonal() line 4306 in > petsc/3.6.3_dbg/src/mat/interface/matrix.c > > [0]PETSC ERROR: #2 PCSetUp_Jacobi() line 180 in > petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c > > [0]PETSC ERROR: #3 PCSetUp_Jacobi_NonSymmetric() line 261 in > petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c > > [0]PETSC ERROR: #4 PCApply_Jacobi() line 286 in > petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c > > [0]PETSC ERROR: #5 PCApply() line 483 in > petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #6 KSP_PCApply() line 242 in > petsc/3.6.3_dbg/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #7 KSPSolve_CG() line 139 in > petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c > > [0]PETSC ERROR: #8 KSPSolve() line 604 in > petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #9 PCApply_FieldSplit() line 988 in > petsc/3.6.3_dbg/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #10 PCApply() line 483 in > petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #11 KSP_PCApply() line 242 in > petsc/3.6.3_dbg/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #12 KSPSolve_CG() line 139 in > petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c > > [0]PETSC ERROR: #13 KSPSolve() line 604 in > petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c > > I may not be calling the appropriate routines, here is what I'm basically > doing: > > 1. KSP type set to KSPCG (A is symmetric), PC type to PCFIELDSPLIT, > 2. I then use MatNestGetISs on A to supply the ISs to > PCFieldSplitSetIS, > 3. Finally I use PCFieldSplitGetSubKSP to use set the subKSP for each > diagonal block. > > As I mentioned, as long as the subKSPs don't have any preconditioning > (PCNONE), KSPSolve does not complain, only when I try to using one (even > PCJACOBI) do I get similar errors). That makes me suspect that I'm doing > something wrong, but I could not find an example that was doing the same > thing... > > Any pointers would be much appreciated! > > Thanks, > Vincent > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlc at illinois.edu Fri Apr 29 08:32:06 2016 From: vlc at illinois.edu (Le Chenadec, Vincent) Date: Fri, 29 Apr 2016 13:32:06 +0000 Subject: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL off-diagonal blocks In-Reply-To: References: <3664C6DEB0AB9C429B69017A6483BACD45B7B07B@CHIMBX6.ad.uillinois.edu>, Message-ID: <3664C6DEB0AB9C429B69017A6483BACD45B7B0E1@CHIMBX6.ad.uillinois.edu> Dear Matt, thanks for your prompt response. You are were right (KSPView output below): I was trying to precondition the off-diagonal blocks (I messed up the ordering of the blocks, stored as ((00), (01), (11, 10)) where I should have simply used row-major ordering). One question: is PCFieldSplitSetIS is most effective way to go about this? Or is there a way to pass the DMDAs directly rather than extracting the ISs from A? Thanks a lot, Vincent KSP Object: 8 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 8 MPI processes type: fieldsplit FieldSplit with MULTIPLICATIVE composition: total splits = 3 Solver info for each split is in the following KSP objects: Split number 0 Defined by IS KSP Object: (fieldsplit_0_) 8 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 8 MPI processes type: jacobi PC has not been set up so information may be incomplete linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 8 MPI processes type: mpiaij rows=262144, cols=262144 total: nonzeros=7.07789e+06, allocated nonzeros=7.07789e+06 total number of mallocs used during MatSetValues calls =0 Split number 1 Defined by IS KSP Object: (fieldsplit_1_) 8 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_1_) 8 MPI processes type: jacobi PC has not been set up so information may be incomplete linear system matrix = precond matrix: Mat Object: (fieldsplit_1_) 8 MPI processes type: shell rows=262144, cols=262144 Split number 2 Defined by IS KSP Object: (fieldsplit_2_) 8 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_2_) 8 MPI processes type: jacobi PC has not been set up so information may be incomplete linear system matrix = precond matrix: Mat Object: (fieldsplit_2_) 8 MPI processes type: shell rows=262144, cols=262144 linear system matrix = precond matrix: Mat Object: 8 MPI processes type: nest rows=786432, cols=786432 Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=262144, cols=262144 (0,1) : type=shell, rows=262144, cols=262144 (0,2) : type=shell, rows=262144, cols=262144 (1,0) : type=mpiaij, rows=262144, cols=262144 (1,1) : prefix="fieldsplit_1_", type=shell, rows=262144, cols=262144 (1,2) : type=shell, rows=262144, cols=262144 (2,0) : type=mpiaij, rows=262144, cols=262144 (2,1) : type=shell, rows=262144, cols=262144 (2,2) : prefix="fieldsplit_2_", type=shell, rows=262144, cols=262144 ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Friday, April 29, 2016 7:42 AM To: Le Chenadec, Vincent Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL off-diagonal blocks On Fri, Apr 29, 2016 at 7:34 AM, Le Chenadec, Vincent > wrote: Dear all, I'm trying to use PCFIELDSPLIT as a preconditioner on a block matrix A, [A_{00} A_{01}] [A_{10} A_{11}] where the diagonal blocks (A_{00} and A_{11}) are diagonally dominant. I would like to use either one of the block preconditioners (not even based on the Schur complement, simply either one of the Jacobi/Gauss-Seidel ones). For any questions about solvers, we need to see the output of -ksp_view. Below, it seems clear that either a) One of your diagonal blocks is a MatShell, or b) you are trying to invert an off-diagonal block. Matt The issue I'm running into concerns the formats I would like to use, so I'll start with that: 1. A_{00} and A_{11} are both in MATMPIAIJ format (created using DMCreateMatrix on 2D DMDAs of different sizes), 2. A_{10} and A_{01} are both in MATSHELL format (created using DMCreateShell). The reason why I used this format is that it seemed convenient, since the 2D DMDAs are of different sizes and I assumed I had to provide the MATOP_MULT operation only, 3. Finally the blocks are stored in MATNEST format (MatCreateNest). With this choice, matrix-vector multiplications work fine (the DMDAs being packed in a DMComposite), which makes me think that so far the code is ok. Also, when I attempt to invert the matrix without preconditioner, I get the correct result. Now here is where things do not work: if I try to use a preconditioner, I get the error pasted below (for a simple PCJACOBI) [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Mat type shell [0]PETSC ERROR: #1 MatGetDiagonal() line 4306 in petsc/3.6.3_dbg/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_Jacobi() line 180 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #3 PCSetUp_Jacobi_NonSymmetric() line 261 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #4 PCApply_Jacobi() line 286 in petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #5 PCApply() line 483 in petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSP_PCApply() line 242 in petsc/3.6.3_dbg/include/petsc/private/kspimpl.h [0]PETSC ERROR: #7 KSPSolve_CG() line 139 in petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: #8 KSPSolve() line 604 in petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 PCApply_FieldSplit() line 988 in petsc/3.6.3_dbg/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #10 PCApply() line 483 in petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #11 KSP_PCApply() line 242 in petsc/3.6.3_dbg/include/petsc/private/kspimpl.h [0]PETSC ERROR: #12 KSPSolve_CG() line 139 in petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: #13 KSPSolve() line 604 in petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c I may not be calling the appropriate routines, here is what I'm basically doing: 1. KSP type set to KSPCG (A is symmetric), PC type to PCFIELDSPLIT, 2. I then use MatNestGetISs on A to supply the ISs to PCFieldSplitSetIS, 3. Finally I use PCFieldSplitGetSubKSP to use set the subKSP for each diagonal block. As I mentioned, as long as the subKSPs don't have any preconditioning (PCNONE), KSPSolve does not complain, only when I try to using one (even PCJACOBI) do I get similar errors). That makes me suspect that I'm doing something wrong, but I could not find an example that was doing the same thing... Any pointers would be much appreciated! Thanks, Vincent -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 29 09:01:34 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 29 Apr 2016 09:01:34 -0500 Subject: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL off-diagonal blocks In-Reply-To: <3664C6DEB0AB9C429B69017A6483BACD45B7B0E1@CHIMBX6.ad.uillinois.edu> References: <3664C6DEB0AB9C429B69017A6483BACD45B7B07B@CHIMBX6.ad.uillinois.edu> <3664C6DEB0AB9C429B69017A6483BACD45B7B0E1@CHIMBX6.ad.uillinois.edu> Message-ID: On Fri, Apr 29, 2016 at 8:32 AM, Le Chenadec, Vincent wrote: > Dear Matt, > > thanks for your prompt response. > > You are were right (KSPView output below): I was trying to precondition > the off-diagonal blocks (I messed up the ordering of the blocks, stored as > ((00), (01), (11, 10)) where I should have simply used row-major ordering). > > One question: is PCFieldSplitSetIS is most effective way to go about this? > Or is there a way to pass the DMDAs directly rather than extracting the ISs > from A? > This is the best way when you are using a bunch of DMs. I have a more flexible dof specification in DMPlex, but it has not migrated to DMDA yet. Thanks, Matt > Thanks a lot, > Vincent > > KSP Object: 8 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 8 MPI processes > type: fieldsplit > FieldSplit with MULTIPLICATIVE composition: total splits = 3 > Solver info for each split is in the following KSP objects: > Split number 0 Defined by IS > KSP Object: (fieldsplit_0_) 8 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 8 MPI processes > type: jacobi > PC has not been set up so information may be incomplete > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 8 MPI processes > type: mpiaij > rows=262144, cols=262144 > total: nonzeros=7.07789e+06, allocated nonzeros=7.07789e+06 > total number of mallocs used during MatSetValues calls =0 > Split number 1 Defined by IS > KSP Object: (fieldsplit_1_) 8 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_1_) 8 MPI processes > type: jacobi > PC has not been set up so information may be incomplete > linear system matrix = precond matrix: > Mat Object: (fieldsplit_1_) 8 MPI processes > type: shell > rows=262144, cols=262144 > Split number 2 Defined by IS > KSP Object: (fieldsplit_2_) 8 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_2_) 8 MPI processes > type: jacobi > PC has not been set up so information may be incomplete > linear system matrix = precond matrix: > Mat Object: (fieldsplit_2_) 8 MPI processes > type: shell > rows=262144, cols=262144 > linear system matrix = precond matrix: > Mat Object: 8 MPI processes > type: nest > rows=786432, cols=786432 > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=262144, > cols=262144 > (0,1) : type=shell, rows=262144, cols=262144 > (0,2) : type=shell, rows=262144, cols=262144 > (1,0) : type=mpiaij, rows=262144, cols=262144 > (1,1) : prefix="fieldsplit_1_", type=shell, rows=262144, > cols=262144 > (1,2) : type=shell, rows=262144, cols=262144 > (2,0) : type=mpiaij, rows=262144, cols=262144 > (2,1) : type=shell, rows=262144, cols=262144 > (2,2) : prefix="fieldsplit_2_", type=shell, rows=262144, > cols=262144 > > > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com] > *Sent:* Friday, April 29, 2016 7:42 AM > *To:* Le Chenadec, Vincent > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] PCFIELDSPLIT on a MATNEST with MATSHELL > off-diagonal blocks > > On Fri, Apr 29, 2016 at 7:34 AM, Le Chenadec, Vincent > wrote: > >> Dear all, >> >> I'm trying to use PCFIELDSPLIT as a preconditioner on a block matrix A, >> >> [A_{00} A_{01}] >> [A_{10} A_{11}] >> >> where the diagonal blocks (A_{00} and A_{11}) are diagonally dominant. I >> would like to use either one of the block preconditioners (not even based >> on the Schur complement, simply either one of the Jacobi/Gauss-Seidel ones). >> > > For any questions about solvers, we need to see the output of -ksp_view. > > Below, it seems clear that either a) One of your diagonal blocks is a > MatShell, or b) you are trying to invert > an off-diagonal block. > > Matt > > >> The issue I'm running into concerns the formats I would like to use, so >> I'll start with that: >> >> 1. A_{00} and A_{11} are both in MATMPIAIJ format (created using >> DMCreateMatrix on 2D DMDAs of different sizes), >> 2. A_{10} and A_{01} are both in MATSHELL format (created using >> DMCreateShell). The reason why I used this format is that it seemed >> convenient, since the 2D DMDAs are of different sizes and I assumed I had >> to provide the MATOP_MULT operation only, >> 3. Finally the blocks are stored in MATNEST format (MatCreateNest). >> >> With this choice, matrix-vector multiplications work fine (the DMDAs >> being packed in a DMComposite), which makes me think that so far the code >> is ok. Also, when I attempt to invert the matrix without preconditioner, >> I get the correct result. >> >> Now here is where things do not work: if I try to use a preconditioner, >> I get the error pasted below (for a simple PCJACOBI) >> >> [0]PETSC ERROR: No support for this operation for this object type >> >> [0]PETSC ERROR: Mat type shell >> >> [0]PETSC ERROR: #1 MatGetDiagonal() line 4306 in >> petsc/3.6.3_dbg/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #2 PCSetUp_Jacobi() line 180 in >> petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c >> >> [0]PETSC ERROR: #3 PCSetUp_Jacobi_NonSymmetric() line 261 in >> petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c >> >> [0]PETSC ERROR: #4 PCApply_Jacobi() line 286 in >> petsc/3.6.3_dbg/src/ksp/pc/impls/jacobi/jacobi.c >> >> [0]PETSC ERROR: #5 PCApply() line 483 in >> petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #6 KSP_PCApply() line 242 in >> petsc/3.6.3_dbg/include/petsc/private/kspimpl.h >> >> [0]PETSC ERROR: #7 KSPSolve_CG() line 139 in >> petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c >> >> [0]PETSC ERROR: #8 KSPSolve() line 604 in >> petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c >> >> [0]PETSC ERROR: #9 PCApply_FieldSplit() line 988 in >> petsc/3.6.3_dbg/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #10 PCApply() line 483 in >> petsc/3.6.3_dbg/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #11 KSP_PCApply() line 242 in >> petsc/3.6.3_dbg/include/petsc/private/kspimpl.h >> >> [0]PETSC ERROR: #12 KSPSolve_CG() line 139 in >> petsc/3.6.3_dbg/src/ksp/ksp/impls/cg/cg.c >> >> [0]PETSC ERROR: #13 KSPSolve() line 604 in >> petsc/3.6.3_dbg/src/ksp/ksp/interface/itfunc.c >> >> I may not be calling the appropriate routines, here is what I'm >> basically doing: >> >> 1. KSP type set to KSPCG (A is symmetric), PC type to PCFIELDSPLIT, >> 2. I then use MatNestGetISs on A to supply the ISs to >> PCFieldSplitSetIS, >> 3. Finally I use PCFieldSplitGetSubKSP to use set the subKSP for each >> diagonal block. >> >> As I mentioned, as long as the subKSPs don't have any preconditioning >> (PCNONE), KSPSolve does not complain, only when I try to using one (even >> PCJACOBI) do I get similar errors). That makes me suspect that I'm doing >> something wrong, but I could not find an example that was doing the same >> thing... >> >> Any pointers would be much appreciated! >> >> Thanks, >> Vincent >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 29 09:04:04 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 29 Apr 2016 09:04:04 -0500 Subject: [petsc-users] Averaging 3D DA global vector In-Reply-To: References: Message-ID: On Fri, Apr 29, 2016 at 7:12 AM, Ilyas YILMAZ wrote: > Hello, > > Somewhere in my code, I need to take spatial average of a global vector > created by 3D DA context and store it into another global vector. > It is like X3D(streamwise,normalwise,spanwise) --> Y3D(0,normalwise,0). It > is very easy to do serially in Fortran, for example, > You can write the same code with DMDAVecGetArray(). Thanks, Matt > tot = (ni-2)*(nk-2) > > do k= 2,nkm1 > do j= 2,njm1 > do i= 2,nim1 > > su(2,j,2)=su(2,j,2)+cs(i,j,k)/float(tot) > > end do > end do > end do > > What could be the most efficient way of doing this in PETSC? > > or Is it possible to do as follows using ADD_VALUES? > > mxmz = mx*mz > > do k=zs,zs+zm-1 > do j=ys,ys+ym-1 > do i=xs,xs+xm-1 > > row = i - gxs + (j - gys)*gxm + (k - gzs)*gxm*gym + 1 > rowj = xs - gxs + (j - gys)*gxm + (zs - gzs)*gxm*gym + 1 > > scaX3DAVE(idX3DAVE+*rowj*) = scaX3DAVE(idX3DAVE+*rowj*) > + scaX3D(idX3D+*row*)/float(mx*mz) > > end do > end do > end do > > call VecRestoreArray() > call DALocaltoGlobal(da,localX3DAVE,*ADD_VALUES*,X3DAVE,ierr) > > > Thanks. > > IY > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilyascfd at gmail.com Sat Apr 30 10:04:48 2016 From: ilyascfd at gmail.com (Ilyas YILMAZ) Date: Sat, 30 Apr 2016 18:04:48 +0300 Subject: [petsc-users] a crash due to memory issue Message-ID: Hello, The code segment I wrote based on "src/dm/da/examples/tutorials/ex2.c" crashes when destroying things / freeing memory as given below. I can't figure out what I'm missing? Any comments are welcome. (Petsc 3.1.p8, intel fortran compiler). The code and error are below. !...Create scatter from global DA parallel vector to local vector that contains all entries call DAGlobalToNaturalAllCreate(da,tolocalall,ierr) call DAGlobalToNaturalAllCreate(da,tolocalall2,ierr) call DAGlobalToNaturalAllCreate(da,tolocalall3,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSRENOMlocalall,ierr) call VecScatterBegin(tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSDENOMlocalall,ierr) call VecScatterBegin(tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSlocalall,ierr) call VecScatterBegin(tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecGetArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) call VecGetArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) call VecGetArray( CSlocalall,scaCSlocalall ,idCSlocalall ,ierr) SOME WORK HERE call VecRestoreArray(CSlocalall ,scaCSlocalall ,idCSlocalall,ierr) call VecRestoreArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) call VecRestoreArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) !...scatter back to global vector call DANaturalAllToGlobalCreate(da,fromlocalall,ierr) call DANaturalAllToGlobalCreate(da,fromlocalall2,ierr) call DANaturalAllToGlobalCreate(da,fromlocalall3,ierr) call VecScatterBegin(fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterBegin(fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterBegin(fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) !..free memory call VecScatterDestroy(fromlocalall,ierr) call VecScatterDestroy(fromlocalall2,ierr) call VecScatterDestroy(fromlocalall3,ierr) call VecScatterDestroy(tolocalall,ierr) call VecScatterDestroy(tolocalall2,ierr) call VecScatterDestroy(tolocalall3,ierr) call VecDestroy(CSDENOMlocalall,ierr) call VecDestroy(CSRENOMlocalall,ierr) call VecDestroy(CSlocalall,ierr) . . . . . . ...................... SGS model is called MUT computed MUT computed MUT computed MUT computed [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. . . . . . Thank you Ilyas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 30 10:08:55 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 30 Apr 2016 10:08:55 -0500 Subject: [petsc-users] a crash due to memory issue In-Reply-To: References: Message-ID: On Sat, Apr 30, 2016 at 10:04 AM, Ilyas YILMAZ wrote: > Hello, > > The code segment I wrote based on "src/dm/da/examples/tutorials/ex2.c" > crashes when destroying things / freeing memory as given below. > I can't figure out what I'm missing? Any comments are welcome. (Petsc > 3.1.p8, intel fortran compiler). The code and error are below. > > !...Create scatter from global DA parallel vector to local vector > that contains all entries > call DAGlobalToNaturalAllCreate(da,tolocalall,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall2,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall3,ierr) > 1) Why are you doing this? There does not seem to be any need for it. 2) Always run with debugging until your code works 3) If you get an SEGV, run with valgrind (http://www.valgrind.org) Matt > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSRENOMlocalall,ierr) > call > VecScatterBegin(tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSDENOMlocalall,ierr) > call > VecScatterBegin(tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSlocalall,ierr) > call > VecScatterBegin(tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > > call > VecGetArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecGetArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > call VecGetArray( CSlocalall,scaCSlocalall > ,idCSlocalall ,ierr) > > > SOME WORK HERE > > > call VecRestoreArray(CSlocalall ,scaCSlocalall > ,idCSlocalall,ierr) > call > VecRestoreArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecRestoreArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > > > !...scatter back to global vector > call DANaturalAllToGlobalCreate(da,fromlocalall,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall2,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall3,ierr) > > call > VecScatterBegin(fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > !..free memory > call VecScatterDestroy(fromlocalall,ierr) > call VecScatterDestroy(fromlocalall2,ierr) > call VecScatterDestroy(fromlocalall3,ierr) > > call VecScatterDestroy(tolocalall,ierr) > call VecScatterDestroy(tolocalall2,ierr) > call VecScatterDestroy(tolocalall3,ierr) > > call VecDestroy(CSDENOMlocalall,ierr) > call VecDestroy(CSRENOMlocalall,ierr) > call VecDestroy(CSlocalall,ierr) > > > . > . > . > . > . > . > ...................... SGS model is called > MUT computed > MUT computed > MUT computed > MUT computed > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find > memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > . > . > . > . > . > > Thank you > > Ilyas. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sat Apr 30 10:12:41 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Sat, 30 Apr 2016 16:12:41 +0100 Subject: [petsc-users] a crash due to memory issue In-Reply-To: References: Message-ID: On 30 April 2016 at 16:04, Ilyas YILMAZ wrote: > Hello, > > The code segment I wrote based on "src/dm/da/examples/tutorials/ex2.c" > crashes when destroying things / freeing memory as given below. > I can't figure out what I'm missing? Any comments are welcome. (Petsc > 3.1.p8, > I'd strongly suggest updating to version 3.7 before proceeding any further. > intel fortran compiler). The code and error are below. > > !...Create scatter from global DA parallel vector to local vector > that contains all entries > call DAGlobalToNaturalAllCreate(da,tolocalall,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall2,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall3,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSRENOMlocalall,ierr) > call > VecScatterBegin(tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSDENOMlocalall,ierr) > call > VecScatterBegin(tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSlocalall,ierr) > call > VecScatterBegin(tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > > call > VecGetArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecGetArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > call VecGetArray( CSlocalall,scaCSlocalall > ,idCSlocalall ,ierr) > > > SOME WORK HERE > > > call VecRestoreArray(CSlocalall ,scaCSlocalall > ,idCSlocalall,ierr) > call > VecRestoreArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecRestoreArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > > > !...scatter back to global vector > call DANaturalAllToGlobalCreate(da,fromlocalall,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall2,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall3,ierr) > > call > VecScatterBegin(fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > !..free memory > call VecScatterDestroy(fromlocalall,ierr) > call VecScatterDestroy(fromlocalall2,ierr) > call VecScatterDestroy(fromlocalall3,ierr) > > call VecScatterDestroy(tolocalall,ierr) > call VecScatterDestroy(tolocalall2,ierr) > call VecScatterDestroy(tolocalall3,ierr) > > call VecDestroy(CSDENOMlocalall,ierr) > call VecDestroy(CSRENOMlocalall,ierr) > call VecDestroy(CSlocalall,ierr) > > > . > . > . > . > . > . > ...................... SGS model is called > MUT computed > MUT computed > MUT computed > MUT computed > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find > memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > . > . > . > . > . > > Thank you > > Ilyas. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Apr 30 10:12:51 2016 From: balay at mcs.anl.gov (Balay, Satish) Date: Sat, 30 Apr 2016 15:12:51 +0000 Subject: [petsc-users] a crash due to memory issue In-Reply-To: References: Message-ID: <8F84BD8FD72DC64AA206CF6E3D7FC474970C00AA@BUTKUS.anl.gov> Also suggest using latest version 3.7. Satish ________________________________ From: Ilyas YILMAZ Sent: ?4/?30/?2016 10:04 AM To: PETSc users list Subject: [petsc-users] a crash due to memory issue Hello, The code segment I wrote based on "src/dm/da/examples/tutorials/ex2.c" crashes when destroying things / freeing memory as given below. I can't figure out what I'm missing? Any comments are welcome. (Petsc 3.1.p8, intel fortran compiler). The code and error are below. !...Create scatter from global DA parallel vector to local vector that contains all entries call DAGlobalToNaturalAllCreate(da,tolocalall,ierr) call DAGlobalToNaturalAllCreate(da,tolocalall2,ierr) call DAGlobalToNaturalAllCreate(da,tolocalall3,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSRENOMlocalall,ierr) call VecScatterBegin(tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSDENOMlocalall,ierr) call VecScatterBegin(tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSlocalall,ierr) call VecScatterBegin(tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecGetArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) call VecGetArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) call VecGetArray( CSlocalall,scaCSlocalall ,idCSlocalall ,ierr) SOME WORK HERE call VecRestoreArray(CSlocalall ,scaCSlocalall ,idCSlocalall,ierr) call VecRestoreArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) call VecRestoreArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) !...scatter back to global vector call DANaturalAllToGlobalCreate(da,fromlocalall,ierr) call DANaturalAllToGlobalCreate(da,fromlocalall2,ierr) call DANaturalAllToGlobalCreate(da,fromlocalall3,ierr) call VecScatterBegin(fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterBegin(fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterBegin(fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd( fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) !..free memory call VecScatterDestroy(fromlocalall,ierr) call VecScatterDestroy(fromlocalall2,ierr) call VecScatterDestroy(fromlocalall3,ierr) call VecScatterDestroy(tolocalall,ierr) call VecScatterDestroy(tolocalall2,ierr) call VecScatterDestroy(tolocalall3,ierr) call VecDestroy(CSDENOMlocalall,ierr) call VecDestroy(CSRENOMlocalall,ierr) call VecDestroy(CSlocalall,ierr) . . . . . . ...................... SGS model is called MUT computed MUT computed MUT computed MUT computed [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. . . . . . Thank you Ilyas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilyascfd at gmail.com Sat Apr 30 14:28:39 2016 From: ilyascfd at gmail.com (Ilyas YILMAZ) Date: Sat, 30 Apr 2016 22:28:39 +0300 Subject: [petsc-users] a crash due to memory issue In-Reply-To: <8F84BD8FD72DC64AA206CF6E3D7FC474970C00AA@BUTKUS.anl.gov> References: <8F84BD8FD72DC64AA206CF6E3D7FC474970C00AA@BUTKUS.anl.gov> Message-ID: Thank you all. Time to upgrade. IY 2016-04-30 18:12 GMT+03:00 Balay, Satish : > Also suggest using latest version 3.7. > > Satish > ------------------------------ > From: Ilyas YILMAZ > Sent: ?4/?30/?2016 10:04 AM > To: PETSc users list > Subject: [petsc-users] a crash due to memory issue > > Hello, > > The code segment I wrote based on "src/dm/da/examples/tutorials/ex2.c" > crashes when destroying things / freeing memory as given below. > I can't figure out what I'm missing? Any comments are welcome. (Petsc > 3.1.p8, intel fortran compiler). The code and error are below. > > !...Create scatter from global DA parallel vector to local vector > that contains all entries > call DAGlobalToNaturalAllCreate(da,tolocalall,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall2,ierr) > call DAGlobalToNaturalAllCreate(da,tolocalall3,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSRENOMlocalall,ierr) > call > VecScatterBegin(tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall,CSRENOM,CSRENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSDENOMlocalall,ierr) > call > VecScatterBegin(tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall2,CSDENOM,CSDENOMlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,im*jm*km,CSlocalall,ierr) > call > VecScatterBegin(tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > tolocalall3,CS,CSlocalall,INSERT_VALUES,SCATTER_FORWARD,ierr) > > > call > VecGetArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecGetArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > call VecGetArray( CSlocalall,scaCSlocalall > ,idCSlocalall ,ierr) > > > SOME WORK HERE > > > call VecRestoreArray(CSlocalall ,scaCSlocalall > ,idCSlocalall,ierr) > call > VecRestoreArray(CSRENOMlocalall,scaCSRENOMlocalall,idCSRENOMlocalall,ierr) > call > VecRestoreArray(CSDENOMlocalall,scaCSDENOMlocalall,idCSDENOMlocalall,ierr) > > > !...scatter back to global vector > call DANaturalAllToGlobalCreate(da,fromlocalall,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall2,ierr) > call DANaturalAllToGlobalCreate(da,fromlocalall3,ierr) > > call > VecScatterBegin(fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall,CSRENOMlocalall,CSRENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall2,CSDENOMlocalall,CSDENOM,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call > VecScatterBegin(fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > call VecScatterEnd( > fromlocalall3,CSlocalall,CS,INSERT_VALUES,SCATTER_FORWARD,ierr) > > !..free memory > call VecScatterDestroy(fromlocalall,ierr) > call VecScatterDestroy(fromlocalall2,ierr) > call VecScatterDestroy(fromlocalall3,ierr) > > call VecScatterDestroy(tolocalall,ierr) > call VecScatterDestroy(tolocalall2,ierr) > call VecScatterDestroy(tolocalall3,ierr) > > call VecDestroy(CSDENOMlocalall,ierr) > call VecDestroy(CSRENOMlocalall,ierr) > call VecDestroy(CSlocalall,ierr) > > > . > . > . > . > . > . > ...................... SGS model is called > MUT computed > MUT computed > MUT computed > MUT computed > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find > memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > . > . > . > . > . > > Thank you > > Ilyas. > -------------- next part -------------- An HTML attachment was scrubbed... URL: