[petsc-dev] VecScatter scaling problem on KNL

[Mark Adams]

>
>     Is the scatter created with VecScatterCreateToAll()? If so, internally the VecScatterBegin/End will use VecScatterBegin_MPI_ToAll() which then uses a MPI_Allgatherv() to do the communication.  You can check  in the debugger for this (on 2 processes) by just putting a break point in VecScatterBegin_MPI_ToAll() to confirm if it is called.

Alas, not I did not use VecScatterCreateToAll and
VecScatterCreateToAll will take some code changes.

There are boundary conditions in the destination vector, and so we
scatter into a larger vector the the global size of the PETSc vector,
using a general IS. With code that looks like this:

  call ISCreateGeneral(PETSC_COMM_SELF,nreal,petsc_xgc,PETSC_COPY_VALUES,is,ierr)
  call VecScatterCreate(this%xVec,PETSC_NULL_OBJECT,vec,is,this%from_petsc,ierr)
! reverse scatter object

If we want to make this change then I could help a developer or you
can get me set up with a (small) test problem and a branch and I can
do it at NERSC.

Thanks,