<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#330033" bgcolor="#FFFFFF">
Dear PETSc folks,<br>
<br>
<br>
I want a Block Jacobi PC to be the second PC in a two-stage
preconditioning scheme implemented via multiplicative PCCOMPOSITE,
with the outermost KSP an FGMRES.<br>
<br>
<br>
However, PCBJacobiGetSubKSP
(<a class="moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCBJacobiGetSubKSP.html#PCBJacobiGetSubKSP">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCBJacobiGetSubKSP.html#PCBJacobiGetSubKSP</a>)
requires to call KSPSetUp (or PCSetUp) first on its parent KSP,
which I struggle in succeeding. I wonder which KSP (or if so PC)
that is. <br>
<br>
<br>
This is how I attempt to do it (using PCKSP to provide a parent KSP
for PCBJacobiGetSubKSP):<br>
<br>
<br>
call KSPGetPC(solver%ksp, solver%pc, ierr); CHKERRQ(ierr)<br>
call PCSetType(solver%pc, PCCOMPOSITE, ierr); CHKERRQ(ierr)<br>
call PCCompositeSetType(solver%pc, PC_COMPOSITE_MULTIPLICATIVE,
ierr); CHKERRQ(ierr)<br>
<br>
<br>
! 1st Stage <br>
call PCCompositeAddPC(solver%pc, PCGALERKIN, ierr); CHKERRQ(ierr)<br>
call PCCompositeGetPC(solver%pc, 0, T1, ierr); CHKERRQ(ierr)<br>
<br>
<br>
! KSPPREONLY-PCNONE for testing<br>
call PCGalerkinGetKSP(T1, Ap_ksp, ierr); CHKERRQ(ierr)<br>
call KSPSetType(Ap_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)<br>
call KSPGetPC(Ap_ksp, Ap_pc, ierr); CHKERRQ(ierr)<br>
call PCSetType(Ap_pc, PCNONE, ierr); CHKERRQ(ierr)<br>
<br>
<br>
! 2nd Stage<br>
call PCCompositeAddPC(solver%pc, PCKSP, ierr); CHKERRQ(ierr)<br>
call PCCompositeGetPC(solver%pc, 1, T2, ierr); CHKERRQ(ierr)<br>
call PCKSPGetKSP(T2, BJac_ksp, ierr); CHKERRQ(ierr)<br>
call KSPSetType(BJac_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)<br>
call KSPGetPC(BJac_ksp, BJac_pc, ierr); CHKERRQ(ierr)<br>
call PCSetType(BJac_pc, PCBJACOBI, ierr); CHKERRQ(ierr)<br>
<br>
<br>
call KSPSetUp(solver%ksp, ierr); CHKERRQ(ierr)<br>
! call KSPSetUp(BJac_ksp, ierr); CHKERRQ(ierr)<br>
! call PCSetUp(T2, ierr); CHKERRQ(ierr)<br>
! call PCSetUp(BJac_pc, ierr); CHKERRQ(ierr)<br>
<br>
<br>
call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp,
PETSC_NULL_KSP, ierr); CHKERRQ(ierr)<br>
allocate(sub_ksps(nsub_ksp))<br>
call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp,
sub_ksps,ierr); CHKERRQ(ierr)<br>
do i = 1, nsub_ksp<br>
call KSPGetPC(sub_ksps(i), BJac_pc_sub, ierr); CHKERRQ(ierr)<br>
call PCFactorSetShiftType(BJac_pc_sub, MAT_SHIFT_INBLOCKS, ierr);
CHKERRQ(ierr)<br>
call PCFactorSetZeroPivot(BJac_pc_sub,
solver%linear_zero_pivot_tol, ierr); CHKERRQ(ierr)<br>
end do<br>
deallocate(sub_ksps)<br>
nullify(sub_ksps)<br>
<br>
<br>
Is using PCKSP a good idea at all? <br>
<br>
<br>
With KSPSetUp(solver%ksp) -> FGMRES<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Object is in wrong state<br>
[0]PETSC ERROR: You requested a vector from a KSP that cannot
provide one<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision:
v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500<br>
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue
Jun 27 16:55:14 2017<br>
[0]PETSC ERROR: Configure options --download-mpich=yes
--download-hdf5=yes --download-fblaslapack=yes --download-metis=yes
--download-parmetis=yes --download-eigen=yes --download-hypre=yes
--download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6
--with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2
PETSC_DIR=/home/pujjad/Repositories/petsc<br>
[0]PETSC ERROR: #1 KSPCreateVecs() line 939 in
/home/pujjad/Repositories/petsc/src/ksp/ksp/interface/iterativ.c<br>
[0]PETSC ERROR: #2 KSPSetUp_GMRES() line 85 in
/home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/gmres.c<br>
[0]PETSC ERROR: #3 KSPSetUp_FGMRES() line 41 in
/home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c<br>
[0]PETSC ERROR: #4 KSPSetUp() line 338 in
/home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c<br>
application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0<br>
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52):
Unrecognized PMI command: abort | cleaning up processes<br>
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable
to process PMI command<br>
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status<br>
[mpiexec@mother] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<br>
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager
error waiting for completion<br>
<br>
<br>
<br>
With KSPSetUp(BJac_ksp) -> KSPPREONLY<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Arguments are incompatible<br>
[0]PETSC ERROR: Both n and N cannot be PETSC_DECIDE<br>
likely a call to VecSetSizes() or MatSetSizes() is wrong.<br>
See <a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html#split">http://www.mcs.anl.gov/petsc/documentation/faq.html#split</a><br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision:
v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500<br>
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue
Jun 27 16:52:57 2017<br>
[0]PETSC ERROR: Configure options --download-mpich=yes
--download-hdf5=yes --download-fblaslapack=yes --download-metis=yes
--download-parmetis=yes --download-eigen=yes --download-hypre=yes
--download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6
--with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2
PETSC_DIR=/home/pujjad/Repositories/petsc<br>
[0]PETSC ERROR: #1 PetscSplitOwnership() line 77 in
/home/pujjad/Repositories/petsc/src/sys/utils/psplit.c<br>
[0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in
/home/pujjad/Repositories/petsc/src/vec/is/utils/pmap.c<br>
[0]PETSC ERROR: #3 VecCreate_Seq_Private() line 847 in
/home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c<br>
[0]PETSC ERROR: #4 VecCreateSeqWithArray() line 899 in
/home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c<br>
[0]PETSC ERROR: #5 PCSetUp_BJacobi_Singleblock() line 786 in
/home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c<br>
[0]PETSC ERROR: #6 PCSetUp_BJacobi() line 136 in
/home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c<br>
[0]PETSC ERROR: #7 PCSetUp() line 924 in
/home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: #8 KSPSetUp() line 379 in
/home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c<br>
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52):
Unrecognized PMI command: abort | cleaning up processes<br>
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable
to process PMI command<br>
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status<br>
[mpiexec@mother] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<br>
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager
error waiting for completion<br>
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52):
Unrecognized PMI command: abort | cleaning up processes<br>
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable
to process PMI command<br>
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status<br>
[mpiexec@mother] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<br>
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager
error waiting for completion<br>
<br>
<br>
<br>
With PCSetUp(T2) -> PCKSP<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Object is in wrong state<br>
[0]PETSC ERROR: Matrix must be set first<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision:
v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500<br>
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue
Jun 27 16:51:23 2017<br>
[0]PETSC ERROR: Configure options --download-mpich=yes
--download-hdf5=yes --download-fblaslapack=yes --download-metis=yes
--download-parmetis=yes --download-eigen=yes --download-hypre=yes
--download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6
--with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2
PETSC_DIR=/home/pujjad/Repositories/petsc<br>
[0]PETSC ERROR: #1 PCSetUp() line 888 in
/home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c<br>
application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0<br>
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52):
Unrecognized PMI command: abort | cleaning up processes<br>
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable
to process PMI command<br>
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status<br>
[mpiexec@mother] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<br>
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager
error waiting for completion<br>
<br>
<br>
<br>
With PCSetUp(BJac_pc) -> PCBJACOBI<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Object is in wrong state<br>
[0]PETSC ERROR: Matrix must be set first<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision:
v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500<br>
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue
Jun 27 16:42:10 2017<br>
[0]PETSC ERROR: Configure options --download-mpich=yes
--download-hdf5=yes --download-fblaslapack=yes --download-metis=yes
--download-parmetis=yes --download-eigen=yes --download-hypre=yes
--download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6
--with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2
PETSC_DIR=/home/pujjad/Repositories/petsc<br>
[0]PETSC ERROR: #1 PCSetUp() line 888 in
/home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c<br>
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52):
Unrecognized PMI command: abort | cleaning up processes<br>
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable
to process PMI command<br>
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status<br>
[mpiexec@mother] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<br>
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager
error waiting for completion<br>
<br>
<br>
<br>
Grateful for any help!<br>
Robert<br>
<br>
<br>
</body>
</html>