[petsc-dev] PCTelescope error
Dave May
dave.mayhem23 at gmail.com
Tue Dec 8 07:52:38 CST 2015
Hi Barry,
I've encountered an issue with PCTelescope which I think is related to your
changesets b2566f2, 8f5db7e which
introduce PetscAllreduceBarrierCheck() and MPIU_Allreduce().
The error message is reported below.
This message
[0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
appears to be produced by PetscAllreduceBarrierCheck(), but I'm not sure
why it would get thrown in my use case.
The same telescope job run here didn't produce an error with the original
pull request.
It seems the error does not occur when the reduction factor equals the
original number of MPI ranks in comm world.
Stupidly I didn't add a test for that particular case, otherwise this issue
would have been caught earlier.
Do you have ideas what the cause/fix might be?
Thanks,
Dave
/Users/dmay/software/petsc-developments/petscfork/arch-gpu-debug-single/bin/mpiexec
-n 4 ./ex29 -ksp_type fgmres -ksp_monitor -da_grid_x 65 -da_grid_y 65
-pc_type telescope -ksp_view -pc_telescope_reduction_factor 2
-telescope_pc_type mg -telescope_pc_mg_levels 2
-telescope_mg_levels_pc_type jacobi -telescope_pc_mg_galerkin
-telescope_mg_levels_pc_type asm
[2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: Petsc has generated inconsistent data
[2]PETSC ERROR: c 4160 should equal 2 * Ml 65 * Nl -1
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03 GIT
Date: 2015-12-04 18:13:24 -0600
[2]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named geop-337.ethz.ch by
dmay Tue Dec 8 14:18:58 2015
[2]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes
--with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib=
--with-debugging=yes --with-precision=single
[2]PETSC ERROR: #1 PCTelescopeSetUp_dmda_repart_coors2d() line 159 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03 GIT
Date: 2015-12-04 18:13:24 -0600
[0]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named geop-337.ethz.ch by
dmay Tue Dec 8 14:18:58 2015
[0]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes
--with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib=
--with-debugging=yes --with-precision=single
[0]PETSC ERROR: #1 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[0]PETSC ERROR: #2 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[0]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Petsc has generated inconsistent data
[1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03 GIT
Date: 2015-12-04 18:13:24 -0600
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: Petsc has generated inconsistent data
[2]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03 GIT
Date: 2015-12-04 18:13:24 -0600
[2]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named geop-337.ethz.ch by
dmay Tue Dec 8 14:18:58 2015
[2]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes
--with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib=
--with-debugging=yes --with-precision=single
[2]PETSC ERROR: #2 VecSetBlockSize() line 1421 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/vec/interface/vector.c
[2]PETSC ERROR: #3 VecSetBlockSize() line 1421 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/vec/interface/vector.c
[2]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 42 in
/Users/dmay/software/petsc-developments/petscfork/src/dm/impls/da/dadist.c
[2]PETSC ERROR: #5 DMCreateGlobalVector() line 764 in
/Users/dmay/software/petsc-developments/petscfork/src/dm/interface/dm.c
[2]PETSC ERROR: #6 DMGetGlobalVector() line 163 in
/Users/dmay/software/petsc-developments/petscfork/src/dm/interface/dmget.c
[2]PETSC ERROR: #7 PCTelescopeSetUp_dmda_permutation_2d() line 568 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
#4 ISCreateGeneral_Private() line 568 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #7 ISCreateGeneral() line 625 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
./ex29 on a arch-gpu-debug-single named geop-337.ethz.ch by dmay Tue Dec 8
14:18:58 2015
[1]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes
--with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib=
--with-debugging=yes --with-precision=single
[1]PETSC ERROR: #1 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[1]PETSC ERROR: #2 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[2]PETSC ERROR: #8 PCTelescopeSetUp_dmda() line 695 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
[2]PETSC ERROR: #9 PCSetUp_Telescope() line 369 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope.c
[2]PETSC ERROR: #10 PCSetUp() line 984 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c
[1]PETSC ERROR: #4 ISCreateGeneral_Private() line 568 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: [2]PETSC ERROR: #11 KSPSetUp() line 384 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #12 main() line 77 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/ksp/examples/tutorials/ex29.c
[2]PETSC ERROR: PETSc Option Table entries:
[2]PETSC ERROR: -da_grid_x 65
[2]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: #7 ISCreateGeneral() line 625 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
-da_grid_y 65
[2]PETSC ERROR: -ksp_monitor
[2]PETSC ERROR: -ksp_type fgmres
[2]PETSC ERROR: -ksp_view
[2]PETSC ERROR: -pc_telescope_reduction_factor 2
[2]PETSC ERROR: -pc_type telescope
[2]PETSC ERROR: -telescope_mg_levels_pc_type asm
[2]PETSC ERROR: -telescope_pc_mg_galerkin
[2]PETSC ERROR: -telescope_pc_mg_levels 2
[2]PETSC ERROR: -telescope_pc_type mg
[2]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2
[cli_2]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2
[3]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[3]PETSC ERROR: Petsc has generated inconsistent data
[3]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: v3.4-10790-g4c50f03 GIT
Date: 2015-12-04 18:13:24 -0600
[3]PETSC ERROR: ./ex29 on a arch-gpu-debug-single named geop-337.ethz.ch by
dmay Tue Dec 8 14:18:58 2015
[3]PETSC ERROR: Configure options --with-fc=0 --download-mpich=yes
--with-opencl --with-viennacl-include=../viennacl-dev --with-viennacl-lib=
--with-debugging=yes --with-precision=single
[3]PETSC ERROR: #1 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[3]PETSC ERROR: #2 PetscSplitOwnership() line 84 in
/Users/dmay/software/petsc-developments/petscfork/src/sys/utils/psplit.c
[3]PETSC ERROR: #3 PetscLayoutSetUp() line 143 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/utils/pmap.c
[3]PETSC ERROR: #4 ISCreateGeneral_Private() line 568 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[3]PETSC ERROR: #5 ISGeneralSetIndices_General() line 683 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[3]PETSC ERROR: #6 ISGeneralSetIndices() line 654 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[3]PETSC ERROR: #7 ISCreateGeneral() line 625 in
/Users/dmay/software/petsc-developments/petscfork/src/vec/is/is/impls/general/general.c
[3]PETSC ERROR: #8 PCTelescopeSetUp_dmda_repart_coors2d() line 163 in
/Users/dmay/software/petsc-developments/petscfork/src/ksp/pc/impls/telescope/telescope_dmda.c
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20151208/82a90cc9/attachment.html>
More information about the petsc-dev
mailing list