[petsc-dev] telescope does not work with TS

Mon Feb 13 14:11:18 CST 2017

Telescope isn't my thing, but you're correct that the data (SNES or
otherwise) required by KSPComputeOperators isn't available on the
subcommunicator.  If you want this feature, I don't know any way to go
about it other than to use the hooks so the user can redistribute their
problem data to new communicators.  I agree it is difficult to implement
in general.

Barry Smith <bsmith at mcs.anl.gov> writes:

>    From the manual page: KSPSetComputeOperators() is not propagated to the sub KSP.
>
>    Yet there is some code 
> sred->ignore_kspcomputeoperators = PETSC_FALSE;
> if (sred->ignore_kspcomputeoperators) {
>           ierr = PetscViewerASCIIPrintf(subviewer,"  Telescope: ignoring KSPComputeOperators\n");CHKERRQ(ierr);
>         }
>   ierr = PetscOptionsBool("-pc_telescope_ignore_kspcomputeoperators","Ignore method used to compute A","PCTelescopeSetIgnoreKSPComputeOperators",sred->ignore_kspcomputeoperators,&sred->ignore_kspcomputeoperators,0);CHKERRQ(ierr);
>   etc
>
>    that seems to indicate that Dave May intended to have support for KSPComputeOperator ... but didn't complete it.
>
>    In this case it is using KSPComputeOperators_SNES(KSP ksp,Mat A,Mat B,void *ctx); which needs a couple of things changed to get it to work:
>
> 1) KSPComputeOperators_SNES() needs to work when the subdm is on a smaller set of processes so the call to SNESComputeJacobian() has to handle running on a subset of process (which of course it cannot do normally because the SNES has its full communicator while the DM only lives on a subset of processes).
>
> 2) The DMRestrictHook_SNESVecSol() has to handle switching to a DM on a smaller set of processes and remapping the "solution" vector to that smaller set.
>
>   Dave and Jed,
>
>     All and all looks awfully complex and fragile to get this working. What do you think?
>
>    Barry
>
>   Curse whoever designed PETSc originally and didn't take into account in its basic design the need to subcom PETSc objects to smaller communicators :-( 
>
>
>
>> On Feb 13, 2017, at 10:01 AM, Zhang, Hong <hongzhang at anl.gov> wrote:
>> 
>> Hi Barry,
>> 
>> When using telescope, I get an error complaining about missing Jacobian. But apparently I have a Jacobian set by TSSetIJacobian().
>> 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR:
>> [0]PETSC ERROR: Must call SNESSetJacobian(), DMSNESSetJacobian(), DMDASNESSetJacobianLocal(), etc
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.4-2764-g0ff3c95  GIT Date: 2017-01-10 15:40:21 -0600
>> [0]PETSC ERROR: ./ex5adj_ell on a arch-memalign-mkl-avx512-opt named isdp001.cels.anl.gov by hongzh Mon Feb 13 09:56:48 2017
>> [0]PETSC ERROR: Configure options --COPTFLAGS="-g -O3 -xMIC-AVX512" --CXXOPTFLAGS="-g -O3 -xMIC-AVX512" --FOPTFLAGS="-g -O3 -xMIC-AVX512" --PETSC_ARCH=arch-memalign-mkl-avx512-opt --PETSC_DIR=/homes/hongzh/Projects/petsc --download-cmake --download-revolve=1 --with-blaslapack-dir=/opt/intel/2017-initial/compilers_and_libraries_2017/linux/mkl/lib/intel64 --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=no --with-fc=mpiifort --with-memalign=64 --with-mpiexec=mpirun
>> [0]PETSC ERROR: #1 SNESComputeJacobian() line 2242 in /home/hongzh/Projects/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #2 KSPComputeOperators_SNES() line 569 in /home/hongzh/Projects/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #3 KSPSetUp() line 327 in /home/hongzh/Projects/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #4 KSPSolve() line 599 in /home/hongzh/Projects/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #5 PCApply_Telescope_dmda() line 952 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/telescope/telescope_dmda.c
>> [0]PETSC ERROR: #6 PCApply() line 458 in /home/hongzh/Projects/petsc/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #7 KSP_PCApply() line 251 in /homes/hongzh/Projects/petsc/include/petsc/private/kspimpl.h
>> [0]PETSC ERROR: #8 KSPSolve_PREONLY() line 22 in /home/hongzh/Projects/petsc/src/ksp/ksp/impls/preonly/preonly.c
>> [0]PETSC ERROR: #9 KSPSolve() line 656 in /home/hongzh/Projects/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #10 PCMGMCycle_Private() line 19 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #11 PCMGMCycle_Private() line 53 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 53 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #13 PCMGMCycle_Private() line 53 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #14 PCMGMCycle_Private() line 53 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #15 PCApply_MG() line 331 in /home/hongzh/Projects/petsc/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: #16 PCApply() line 458 in /home/hongzh/Projects/petsc/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #17 KSP_PCApply() line 251 in /homes/hongzh/Projects/petsc/include/petsc/private/kspimpl.h
>> [0]PETSC ERROR: #18 KSPInitialResidual() line 67 in /home/hongzh/Projects/petsc/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #19 KSPSolve_GMRES() line 233 in /home/hongzh/Projects/petsc/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #20 KSPSolve() line 656 in /home/hongzh/Projects/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #21 SNESSolve_NEWTONLS() line 224 in /home/hongzh/Projects/petsc/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #22 SNESSolve() line 3967 in /home/hongzh/Projects/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #23 TS_SNESSolve() line 171 in /home/hongzh/Projects/petsc/src/ts/impls/implicit/theta/theta.c
>> [0]PETSC ERROR: #24 TSStep_Theta() line 211 in /home/hongzh/Projects/petsc/src/ts/impls/implicit/theta/theta.c
>> [0]PETSC ERROR: #25 TSStep() line 3808 in /home/hongzh/Projects/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: #26 TSSolve() line 4053 in /home/hongzh/Projects/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: #27 main() line 171 in /homes/hongzh/Projects/petsc/src/ts/examples/tutorials/advection-diffusion-reaction/ex5adj_ell.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -da_refine 4
>> [0]PETSC ERROR: -dm_mat_type aij
>> [0]PETSC ERROR: -forwardonly
>> [0]PETSC ERROR: -ksp_converged_reason
>> [0]PETSC ERROR: -ksp_view
>> [0]PETSC ERROR: -mg_coarse_pc_telescope_reduction_factor 2
>> [0]PETSC ERROR: -mg_coarse_pc_type telescope
>> [0]PETSC ERROR: -mg_coarse_telescope_pc_type lu
>> [0]PETSC ERROR: -mg_levels_pc_type jacobi
>> [0]PETSC ERROR: -pc_type mg
>> [0]PETSC ERROR: -ts_dt 1
>> [0]PETSC ERROR: -ts_max_steps 1
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> 
>> Thanks,
>> Hong
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170213/4723bb44/attachment.sig>