<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#330033" bgcolor="#FFFFFF">
<font face="Trebuchet MS">Interesting! And would fit into
configuring PFLOTRAN via its input decks (ie we could also provide
ASM instead of Block Jacobi)<br>
<br>
Thanks a lot!<br>
<br>
</font><br>
<div class="moz-cite-prefix">On 28/06/17 17:31, Barry Smith wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3669D977-1FA4-46AC-9DFD-C69E3C7320EF@mcs.anl.gov">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">On Jun 28, 2017, at 2:07 AM, Robert Annewandter <a class="moz-txt-link-rfc2396E" href="mailto:robert.annewandter@opengosim.com"><robert.annewandter@opengosim.com></a> wrote:
Thank you Barry!
We like to hard wire it into PFLOTRAN with CPR-AMG Block Jacobi Two-Stage Preconditioning potentially becoming the standard solver strategy.
</pre>
</blockquote>
<pre wrap="">
Understood. Note that you can embed the options into the program with PetscOptionsSetValue() so they don't need to be on the command line.
Barry
</pre>
<blockquote type="cite">
<pre wrap="">Using the options database is a great start to reverse engineer the issue!
Thanks!
Robert
On 27/06/17 23:45, Barry Smith wrote:
</pre>
<blockquote type="cite">
<pre wrap=""> It is difficult, if not impossible at times to get all the options where you want them to be using the function call interface. On the other hand it is generally easy (if there are no inner PCSHELLS) to do this via the options database
-pc_type composite
-pc_composite_type multiplicative
-pc_composite_pcs galerkin,bjacobi
-sub_0_galerkin_ksp_type preonly
-sub_0_galerkin_pc_type none
-sub_1_sub_pc_factor_shift_type inblocks
-sub_1_sub_pc_factor_zero_pivot zpiv
</pre>
<blockquote type="cite">
<pre wrap="">On Jun 27, 2017, at 11:24 AM, Robert Annewandter <a class="moz-txt-link-rfc2396E" href="mailto:robert.annewandter@opengosim.com"><robert.annewandter@opengosim.com></a>
wrote:
Dear PETSc folks,
I want a Block Jacobi PC to be the second PC in a two-stage preconditioning scheme implemented via multiplicative PCCOMPOSITE, with the outermost KSP an FGMRES.
However, PCBJacobiGetSubKSP (
<a class="moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCBJacobiGetSubKSP.html#PCBJacobiGetSubKSP">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCBJacobiGetSubKSP.html#PCBJacobiGetSubKSP</a>
) requires to call KSPSetUp (or PCSetUp) first on its parent KSP, which I struggle in succeeding. I wonder which KSP (or if so PC) that is.
This is how I attempt to do it (using PCKSP to provide a parent KSP for PCBJacobiGetSubKSP):
call KSPGetPC(solver%ksp, solver%pc, ierr); CHKERRQ(ierr)
call PCSetType(solver%pc, PCCOMPOSITE, ierr); CHKERRQ(ierr)
call PCCompositeSetType(solver%pc, PC_COMPOSITE_MULTIPLICATIVE, ierr); CHKERRQ(ierr)
! 1st Stage
call PCCompositeAddPC(solver%pc, PCGALERKIN, ierr); CHKERRQ(ierr)
call PCCompositeGetPC(solver%pc, 0, T1, ierr); CHKERRQ(ierr)
! KSPPREONLY-PCNONE for testing
call PCGalerkinGetKSP(T1, Ap_ksp, ierr); CHKERRQ(ierr)
call KSPSetType(Ap_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)
call KSPGetPC(Ap_ksp, Ap_pc, ierr); CHKERRQ(ierr)
call PCSetType(Ap_pc, PCNONE, ierr); CHKERRQ(ierr)
! 2nd Stage
call PCCompositeAddPC(solver%pc, PCKSP, ierr); CHKERRQ(ierr)
call PCCompositeGetPC(solver%pc, 1, T2, ierr); CHKERRQ(ierr)
call PCKSPGetKSP(T2, BJac_ksp, ierr); CHKERRQ(ierr)
call KSPSetType(BJac_ksp, KSPPREONLY, ierr); CHKERRQ(ierr)
call KSPGetPC(BJac_ksp, BJac_pc, ierr); CHKERRQ(ierr)
call PCSetType(BJac_pc, PCBJACOBI, ierr); CHKERRQ(ierr)
call KSPSetUp(solver%ksp, ierr); CHKERRQ(ierr)
! call KSPSetUp(BJac_ksp, ierr); CHKERRQ(ierr)
! call PCSetUp(T2, ierr); CHKERRQ(ierr)
! call PCSetUp(BJac_pc, ierr); CHKERRQ(ierr)
call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp, PETSC_NULL_KSP, ierr); CHKERRQ(ierr)
allocate(sub_ksps(nsub_ksp))
call PCBJacobiGetSubKSP(BJac_pc, nsub_ksp, first_sub_ksp, sub_ksps,ierr); CHKERRQ(ierr)
do i = 1, nsub_ksp
call KSPGetPC(sub_ksps(i), BJac_pc_sub, ierr); CHKERRQ(ierr)
call PCFactorSetShiftType(BJac_pc_sub, MAT_SHIFT_INBLOCKS, ierr); CHKERRQ(ierr)
call PCFactorSetZeroPivot(BJac_pc_sub, solver%linear_zero_pivot_tol, ierr); CHKERRQ(ierr)
end do
deallocate(sub_ksps)
nullify(sub_ksps)
Is using PCKSP a good idea at all?
With KSPSetUp(solver%ksp) -> FGMRES
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: You requested a vector from a KSP that cannot provide one
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:55:14 2017
[0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
[0]PETSC ERROR: #1 KSPCreateVecs() line 939 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: #2 KSPSetUp_GMRES() line 85 in /home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #3 KSPSetUp_FGMRES() line 41 in /home/pujjad/Repositories/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: #4 KSPSetUp() line 338 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c
application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
With KSPSetUp(BJac_ksp) -> KSPPREONLY
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Arguments are incompatible
[0]PETSC ERROR: Both n and N cannot be PETSC_DECIDE
likely a call to VecSetSizes() or MatSetSizes() is wrong.
See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html#split">http://www.mcs.anl.gov/petsc/documentation/faq.html#split</a>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:52:57 2017
[0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
[0]PETSC ERROR: #1 PetscSplitOwnership() line 77 in /home/pujjad/Repositories/petsc/src/sys/utils/psplit.c
[0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in /home/pujjad/Repositories/petsc/src/vec/is/utils/pmap.c
[0]PETSC ERROR: #3 VecCreate_Seq_Private() line 847 in /home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c
[0]PETSC ERROR: #4 VecCreateSeqWithArray() line 899 in /home/pujjad/Repositories/petsc/src/vec/vec/impls/seq/bvec2.c
[0]PETSC ERROR: #5 PCSetUp_BJacobi_Singleblock() line 786 in /home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: #6 PCSetUp_BJacobi() line 136 in /home/pujjad/Repositories/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: #7 PCSetUp() line 924 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #8 KSPSetUp() line 379 in /home/pujjad/Repositories/petsc/src/ksp/ksp/interface/itfunc.c
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
With PCSetUp(T2) -> PCKSP
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Matrix must be set first
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:51:23 2017
[0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
[0]PETSC ERROR: #1 PCSetUp() line 888 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
With PCSetUp(BJac_pc) -> PCBJACOBI
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Matrix must be set first
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3167-g03c0fad GIT Date: 2017-03-30 14:27:53 -0500
[0]PETSC ERROR: pflotran on a debug_g-6.2 named mother by pujjad Tue Jun 27 16:42:10 2017
[0]PETSC ERROR: Configure options --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-parmetis=yes --download-eigen=yes --download-hypre=yes --download-superlu_dist=yes --download-superlu=yes --with-cc=gcc-6 --with-cxx=g++-6 --with-fc=gfortran-6 PETSC_ARCH=debug_g-6.2 PETSC_DIR=/home/pujjad/Repositories/petsc
[0]PETSC ERROR: #1 PCSetUp() line 888 in /home/pujjad/Repositories/petsc/src/ksp/pc/interface/precon.c
[mpiexec@mother] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: abort | cleaning up processes
[mpiexec@mother] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command
[mpiexec@mother] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mother] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event
[mpiexec@mother] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion
Grateful for any help!
Robert
</pre>
</blockquote>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
</body>
</html>