[petsc-users] Scatter error

Tue Oct 1 17:55:38 CDT 2013

Hi guys,

I have two ghosted vectors fxx, fyy which should store derivatives of
a quantity f. They are both obtained via calling VecDuplicate on f. To
minimize communications when computing both fxx and fyy I do the
following:

// loop over boundary points and compute fxx and fyy

VecGhostUpdateBegin(fxx, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateBegin(fyy, INSERT_VALUES, SCATTER_FORWARD);

// compute fxx and fyy for internal points

VecGhostUpdateEnd(fxx, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateEnd(fyy, INSERT_VALUES, SCATTER_FORWARD);

However this does not work and I get the following error:
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Object is in wrong state!
[1]PETSC ERROR:  Scatter ctx already in use!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Development HG revision:
d0a314355399bb0c38e384fde4b1feb01b721896  HG Date: Sat Nov 03 17:18:52
2012 -0500
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by
mohammad Tue Oct  1 15:47:38 2013
[1]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[1]PETSC ERROR: Configure run at Sat Nov  3 15:26:57 2012
[1]PETSC ERROR: Configure options --download-blacs=1
--download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1
--download-metis=1 --download-mumps=1 --download-parmetis=1
--download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx
--with-mpi-dir=/usr/bin --with-pthreadclasses=1
PETSC_ARCH=arch-linux2-cxx-debug
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: VecScatterBegin() line 1546 in
/home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c
[1]PETSC ERROR: VecGhostUpdateBegin() line 235 in
/home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c
[1]PETSC ERROR: constructing node neighboring information ... done in
0.0314 secs. on process 0 [Note: only showing root's timings]
Computing derivatives the new way ...
dxx_and_dyy_central() line 659 in
"unknowndirectory/"../../src/my_p4est_node_neighbors.cpp
terminate called after throwing an instance of 'PETSc::Exception'
  what():  std::exception
[mohammad-Desktop:19589] *** Process received signal ***
[mohammad-Desktop:19589] Signal: Aborted (6)
[mohammad-Desktop:19589] Signal code:  (-6)
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR:  Scatter ctx already in use!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
d0a314355399bb0c38e384fde4b1feb01b721896  HG Date: Sat Nov 03 17:18:52
2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by
mohammad Tue Oct  1 15:47:38 2013
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Sat Nov  3 15:26:57 2012
[0]PETSC ERROR: Configure options --download-blacs=1
--download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1
--download-metis=1 --download-mumps=1 --download-parmetis=1
--download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx
--with-mpi-dir=/usr/bin --with-pthreadclasses=1
PETSC_ARCH=arch-linux2-cxx-debug
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecScatterBegin() line 1546 in
/home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c
[0]PETSC ERROR: VecGhostUpdateBegin() line 235 in
/home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c
[0]PETSC ERROR: dxx_and_dyy_central() line 659 in
"unknowndirectory/"../../src/my_p4est_node_neighbors.cpp
terminate called after throwing an instance of 'PETSc::Exception'
  what():  std::exception
[mohammad-Desktop:19588] *** Process received signal ***
[mohammad-Desktop:19588] Signal: Aborted (6)
[mohammad-Desktop:19588] Signal code:  (-6)
[mohammad-Desktop:19589] [ 0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f12290f4cb0]
[mohammad-Desktop:19589] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1228a60425]
[mohammad-Desktop:19589] [ 2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1228a63b8b]
[mohammad-Desktop:19589] [ 3]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)
[0x7f122988069d]
[mohammad-Desktop:19589] [ 4]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f122987e846]
[mohammad-Desktop:19589] [ 5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f122987e873]
[mohammad-Desktop:19589] [ 6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f122987e96e]
[mohammad-Desktop:19589] [ 7]
./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c)
[0x5bf83c]
[mohammad-Desktop:19589] [ 8]
./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950)
[0x505c30]
[mohammad-Desktop:19589] [ 9] ./derivatives(main+0xe89) [0x4e4678]
[mohammad-Desktop:19589] [10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
[0x7f1228a4b76d]
[mohammad-Desktop:19589] [11] ./derivatives() [0x4e2ab9]
[mohammad-Desktop:19589] *** End of error message ***
[mohammad-Desktop:19588] [ 0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f48f260bcb0]
[mohammad-Desktop:19588] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f48f1f77425]
[mohammad-Desktop:19588] [ 2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f48f1f7ab8b]
[mohammad-Desktop:19588] [ 3]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)
[0x7f48f2d9769d]
[mohammad-Desktop:19588] [ 4]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f48f2d95846]
[mohammad-Desktop:19588] [ 5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f48f2d95873]
[mohammad-Desktop:19588] [ 6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f48f2d9596e]
[mohammad-Desktop:19588] [ 7]
./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c)
[0x5bf83c]
[mohammad-Desktop:19588] [ 8]
./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950)
[0x505c30]
[mohammad-Desktop:19588] [ 9] ./derivatives(main+0xe89) [0x4e4678]
[mohammad-Desktop:19588] [10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
[0x7f48f1f6276d]
[mohammad-Desktop:19588] [11] ./derivatives() [0x4e2ab9]
[mohammad-Desktop:19588] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 19589 on node
mohammad-Desktop exited on signal 6 (Aborted).
--------------------------------------------------------------------------

I assume I'm getting this since they somehow share a VecScatter
internally because of VecDuplicate? If so how can I fix this?

Also as a side question, if I stored fxx and fyy inside a block
vector, would I gain any performance in terms of MPI traffic? What
about in 3D?

Thanks