<div dir="ltr"><div><div>Hi Barry,<br><br></div>I've encountered an issue with PCTelescope which I think is related to your changesets b2566f2, 8f5db7e which<br></div>introduce <span style="font-family:arial,helvetica,sans-serif"><span class="">PetscAllreduceBarrierCheck() and MPIU_Allreduce().<br></span></span><div><div><br></div><div>The error message is reported below.<br></div><div>This message<br></div><div><br>[0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors<br></div><div><br></div><div>appears to be produced by <span style="font-family:arial,helvetica,sans-serif"><span class="">PetscAllreduceBarrierCheck(), but I'm not sure why it would get thrown in my use case.<br></span></span></div><div><br></div><div>The same telescope job run here didn't produce an error with the original pull request.<br></div><div>It seems the error does not occur when the reduction factor equals the original number of MPI ranks in comm world.<br></div><div>Stupidly I didn't add a test for that particular case, otherwise this issue would have been caught earlier.<br><br></div><div>Do you have ideas what the cause/fix might be?<br><br></div><div><br></div><div>Thanks,<br></div><div>  Dave<br></div><div><br><br>/Users/dmay/software/petsc-developments/petscfork/arch-gpu-debug-single/bin/mpiexec -n 4 ./ex29 -ksp_type fgmres -ksp_monitor -da_grid_x 65 -da_grid_y 65 -pc_type telescope  -ksp_view -pc_telescope_reduction_factor 2 -telescope_pc_type mg -telescope_pc_mg_levels 2 -telescope_mg_levels_pc_type jacobi -telescope_pc_mg_galerkin -telescope_mg_levels_pc_type asm<br>[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[2]PETSC ERROR: Petsc has generated inconsistent data<br>[2]PETSC ERROR: c 4160 should equal 2 * Ml 65 * Nl -1<br>[2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[2]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03  GIT Date: 2015-12-04 18:13:24 -0600<br>[2]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named <a href="http://geop-337.ethz.ch">geop-337.ethz.ch</a> by dmay Tue Dec  8 14:18:58 2015<br>[2]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes --with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib= --with-debugging=yes --with-precision=single<br>[2]PETSC ERROR: #1 PCTelescopeSetUp_dmda_repart_coors2d() line 159 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Petsc has generated inconsistent data<br>[0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors<br>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03  GIT Date: 2015-12-04 18:13:24 -0600<br>[0]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named <a href="http://geop-337.ethz.ch">geop-337.ethz.ch</a> by dmay Tue Dec  8 14:18:58 2015<br>[0]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes --with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib= --with-debugging=yes --with-precision=single<br>[0]PETSC ERROR: #1 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[0]PETSC ERROR: #2 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[0]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c<br>[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[1]PETSC ERROR: Petsc has generated inconsistent data<br>[1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors<br>[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[1]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03  GIT Date: 2015-12-04 18:13:24 -0600<br>[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[2]PETSC ERROR: Petsc has generated inconsistent data<br>[2]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors<br>[2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[2]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03  GIT Date: 2015-12-04 18:13:24 -0600<br>[2]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named <a href="http://geop-337.ethz.ch">geop-337.ethz.ch</a> by dmay Tue Dec  8 14:18:58 2015<br>[2]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes --with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib= --with-debugging=yes --with-precision=single<br>[2]PETSC ERROR: #2 VecSetBlockSize() line 1421 in /Users/dmay/software/petsc-developments/petscfork/src/vec/vec/interface/vector.c<br>[2]PETSC ERROR: #3 VecSetBlockSize() line 1421 in /Users/dmay/software/petsc-developments/petscfork/src/vec/vec/interface/vector.c<br>[2]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 42 in /Users/dmay/software/petsc-developments/petscfork/src/dm/impls/da/dadist.c<br>[2]PETSC ERROR: #5 DMCreateGlobalVector() line 764 in /Users/dmay/software/petsc-developments/petscfork/src/dm/interface/dm.c<br>[2]PETSC ERROR: #6 DMGetGlobalVector() line 163 in /Users/dmay/software/petsc-developments/petscfork/src/dm/interface/dmget.c<br>[2]PETSC ERROR: #7 PCTelescopeSetUp_dmda_permutation_2d() line 568 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br>#4 ISCreateGeneral_Private() line 568 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[0]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[0]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[0]PETSC ERROR: #7 ISCreateGeneral() line 625 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[0]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br>./ex29 on a arch-gpu-debug-single named <a href="http://geop-337.ethz.ch">geop-337.ethz.ch</a> by dmay Tue Dec  8 14:18:58 2015<br>[1]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes --with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib= --with-debugging=yes --with-precision=single<br>[1]PETSC ERROR: #1 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[1]PETSC ERROR: #2 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[2]PETSC ERROR: #8 PCTelescopeSetUp_dmda() line 695 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br>[2]PETSC ERROR: #9 PCSetUp_Telescope() line 369 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope.c<br>[2]PETSC ERROR: #10 PCSetUp() line 984 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/interface/precon.c<br>[1]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c<br>[1]PETSC ERROR: #4 ISCreateGeneral_Private() line 568 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[1]PETSC ERROR: [2]PETSC ERROR: #11 KSPSetUp() line 384 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/ksp/interface/itfunc.c<br>[2]PETSC ERROR: #12 main() line 77 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/ksp/examples/tutorials/ex29.c<br>[2]PETSC ERROR: PETSc Option Table entries:<br>[2]PETSC ERROR: -da_grid_x 65<br>[2]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[1]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[1]PETSC ERROR: #7 ISCreateGeneral() line 625 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[1]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br>-da_grid_y 65<br>[2]PETSC ERROR: -ksp_monitor<br>[2]PETSC ERROR: -ksp_type fgmres<br>[2]PETSC ERROR: -ksp_view<br>[2]PETSC ERROR: -pc_telescope_reduction_factor 2<br>[2]PETSC ERROR: -pc_type telescope<br>[2]PETSC ERROR: -telescope_mg_levels_pc_type asm<br>[2]PETSC ERROR: -telescope_pc_mg_galerkin<br>[2]PETSC ERROR: -telescope_pc_mg_levels 2<br>[2]PETSC ERROR: -telescope_pc_type mg<br>[2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------<br>application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2<br>[cli_2]: aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2<br>[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[3]PETSC ERROR: Petsc has generated inconsistent data<br>[3]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors<br>[3]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[3]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03  GIT Date: 2015-12-04 18:13:24 -0600<br>[3]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named <a href="http://geop-337.ethz.ch">geop-337.ethz.ch</a> by dmay Tue Dec  8 14:18:58 2015<br>[3]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes --with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib= --with-debugging=yes --with-precision=single<br>[3]PETSC ERROR: #1 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[3]PETSC ERROR: #2 PetscSplitOwnership() line 84 in /Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c<br>[3]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c<br>[3]PETSC ERROR: #4 ISCreateGeneral_Private() line 568 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[3]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[3]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[3]PETSC ERROR: #7 ISCreateGeneral() line 625 in /Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c<br>[3]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in /Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c<br><br><br></div></div></div>