[petsc-users] PCCOMPOSITE with PCBJACOBI
Barry Smith
bsmith at mcs.anl.gov
Wed Jun 28 11:31:34 CDT 2017
> On Jun 28, 2017, at 2:07 AM, Robert Annewandter <robert.annewandter at opengosim.com> wrote:
>
> Thank you Barry!
>
> We like to hard wire it into PFLOTRAN with CPR-AMG Block Jacobi Two-Stage Preconditioning potentially becoming the standard solver strategy.
Understood. Note that you can embed the options into the program with PetscOptionsSetValue() so they don't need to be on the command line.
Barry
> Using the options database is a great start to reverse engineer the issue!
>
> Thanks!
> Robert
>
>
>
>
> On 27/06/17 23:45, Barry Smith wrote:
>> It is difficult, if not impossible at times to get all the options where you want them to be using the function call interface. On the other hand it is generally easy (if there are no inner PCSHELLS) to do this via the options database
>>
>> -pc_type composite
>> -pc_composite_type multiplicative
>> -pc_composite_pcs galerkin,bjacobi
>>
>> -sub_0_galerkin_ksp_type preonly
>> -sub_0_galerkin_pc_type none
>>
>> -sub_1_sub_pc_factor_shift_type inblocks
>> -sub_1_sub_pc_factor_zero_pivot zpiv
>>
>>
>>
>>
>>> On Jun 27, 2017, at 11:24 AM, Robert Annewandter <robert.annewandter at opengosim.com>
>>> wrote:
>>>
>>> Dear PETSc folks,
>>>
>>>
>>> I want a Block Jacobi PC to be the second PC in a two-stage preconditioning scheme implemented via multiplicative PCCOMPOSITE, with the outermost KSP an FGMRES.
>>>
>>>
>>> However, PCBJacobiGetSubKSP (
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCBJacobiGetSubKSP.html#PCBJacobiGetSubKSP
>>> ) requires to call KSPSetUp (or PCSetUp) first on its parent KSP, which I struggle in succeeding. I wonder which KSP (or if so PC) that is.
>>>
>>>
>>> This is how I attempt to do it (using PCKSP to provide a parent KSP for PCBJacobiGetSubKSP):
>>>
>>>
>>> call KSPGetPC(solver%ksp, solver%pc, ierr); CHKERRQ(ierr)
>>> call PCSetType(solver%pc, PCCOMPOSITE, ierr); CHKERRQ(ierr)
>>> call PCCompositeSetType(solver%pc, PC_COMPOSITE_MULTIPLICATIVE, ierr); CHKERRQ(ierr)
>>>
>>>
>>> ! 1st Stage
>>> call PCCompositeAddPC(solver%pc, PCGALERKIN, ierr); CHKERRQ(ierr)
>>> call PCCompositeGetPC(solver%pc, 0, T1, ierr); CHKERRQ(ierr)
>>>
>>>
>>> ! KSPPREONLY-PCNONE for testing
>>> call PCGalerkinGetKSP(T1, Ap_ksp, ierr); CHKERRQ(ierr)
>>> call KSPSetType(Ap_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)
>>> call KSPGetPC(Ap_ksp, Ap_pc, ierr); CHKERRQ(ierr)
>>> call PCSetType(Ap_pc, PCNONE, ierr); CHKERRQ(ierr)
>>>
>>>
>>> ! 2nd Stage
>>> call PCCompositeAddPC(solver%pc, PCKSP, ierr); CHKERRQ(ierr)
>>> call PCCompositeGetPC(solver%pc, 1, T2, ierr); CHKERRQ(ierr)
>>> call PCKSPGetKSP(T2, BJac_ksp, ierr); CHKERRQ(ierr)
>>> call KSPSetType(BJac_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)
>>> call KSPGetPC(BJac_ksp, BJac_pc, ierr); CHKERRQ(ierr)
>>> call PCSetType(BJac_pc, PCBJACOBI, ierr); CHKERRQ(ierr)
>>>
>>>
>>> call KSPSetUp(solver%ksp, ierr); CHKERRQ(ierr)
>>> ! call KSPSetUp(BJac_ksp, ierr); CHKERRQ(ierr)
>>> ! call PCSetUp(T2, ierr); CHKERRQ(ierr)
>>> ! call PCSetUp(BJac_pc, ierr); CHKERRQ(ierr)
>>>
>>>
>>> call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp, PETSC_NULL_KSP, ierr); CHKERRQ(ierr)
>>> allocate(sub_ksps(nsub_ksp))
>>> call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp, sub_ksps,ierr); CHKERRQ(ierr)
>>> do i = 1, nsub_ksp
>>> call KSPGetPC(sub_ksps(i), BJac_pc_sub, ierr); CHKERRQ(ierr)
>>> call PCFactorSetShiftType(BJac_pc_sub, MAT_SHIFT_INBLOCKS, ierr); CHKERRQ(ierr)
>>> call PCFactorSetZeroPivot(BJac_pc_sub, solver%linear_zero_pivot_tol, ierr); CHKERRQ(ierr)
>>> end do
>>> deallocate(sub_ksps)
>>> nullify(sub_ksps)
>>>
>>>
>>> Is using PCKSP a good idea at all?
>>>
>>>
>>> With KSPSetUp(solver%ksp) -> FGMRES
>>>
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: You requested a vector from a KSP that cannot provide one
>>> [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
>>> [0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:55:14 2017
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
>>> [0]PETSC ERROR: #1 KSPCreateVecs() line 939 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/iterativ.c
>>> [0]PETSC ERROR: #2 KSPSetUp_GMRES() line 85 in /home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: #3 KSPSetUp_FGMRES() line 41 in /home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
>>> [0]PETSC ERROR: #4 KSPSetUp() line 338 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c
>>> application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
>>> [mpiexec at mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
>>> [mpiexec at mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
>>> [mpiexec at mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>>> [mpiexec at mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>>> [mpiexec at mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>>>
>>>
>>>
>>> With KSPSetUp(BJac_ksp) -> KSPPREONLY
>>>
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Arguments are incompatible
>>> [0]PETSC ERROR: Both n and N cannot be PETSC_DECIDE
>>> likely a call to VecSetSizes() or MatSetSizes() is wrong.
>>> See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#split
>>>
>>> [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
>>> [0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:52:57 2017
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
>>> [0]PETSC ERROR: #1 PetscSplitOwnership() line 77 in /home/pujjad/Repositories/petsc/src/sys/utils/psplit.c
>>> [0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in /home/pujjad/Repositories/petsc/src/vec/is/utils/pmap.c
>>> [0]PETSC ERROR: #3 VecCreate_Seq_Private() line 847 in /home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c
>>> [0]PETSC ERROR: #4 VecCreateSeqWithArray() line 899 in /home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c
>>> [0]PETSC ERROR: #5 PCSetUp_BJacobi_Singleblock() line 786 in /home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
>>> [0]PETSC ERROR: #6 PCSetUp_BJacobi() line 136 in /home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
>>> [0]PETSC ERROR: #7 PCSetUp() line 924 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #8 KSPSetUp() line 379 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c
>>> [mpiexec at mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
>>> [mpiexec at mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
>>> [mpiexec at mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>>> [mpiexec at mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>>> [mpiexec at mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>>> [mpiexec at mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
>>> [mpiexec at mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
>>> [mpiexec at mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>>> [mpiexec at mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>>> [mpiexec at mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>>>
>>>
>>>
>>> With PCSetUp(T2) -> PCKSP
>>>
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: Matrix must be set first
>>> [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
>>> [0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:51:23 2017
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
>>> [0]PETSC ERROR: #1 PCSetUp() line 888 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
>>> application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
>>> [mpiexec at mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
>>> [mpiexec at mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
>>> [mpiexec at mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>>> [mpiexec at mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>>> [mpiexec at mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>>>
>>>
>>>
>>> With PCSetUp(BJac_pc) -> PCBJACOBI
>>>
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: Matrix must be set first
>>> [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
>>> [0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:42:10 2017
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
>>> [0]PETSC ERROR: #1 PCSetUp() line 888 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
>>> [mpiexec at mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
>>> [mpiexec at mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
>>> [mpiexec at mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
>>> [mpiexec at mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
>>> [mpiexec at mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
>>>
>>>
>>>
>>> Grateful for any help!
>>> Robert
>>>
>>>
>>>
>
More information about the petsc-users
mailing list