[petsc-dev] VecScatter scaling problem on KNL

[Barry Smith]

  Mark,

   Ok, in this situation VecScatter cannot detect that it is an all to all so will generate a message from each process to each other process. Given my past experience with Cray MPI (why do they even have their own MPI when Intel provides one; in fact why does Cray even exist when they just take other people's products and put their name on them) I am not totally surprised if the Cray MPI chocks on this flood of messages.

   1) Test with Intel MPI, perhaps they handle this case in a scalable way

    2) If Intel MPI also produces poor performance then (interesting, how come on other systems in the past this wasn't a bottleneck for the code?) the easiest solution is to separate the operation into two parts. Use a VecScatterCreateToAll() to get all the data to all the processes and then use another (purely sequential) VecScatter to get the data from this intermediate buffer into the final vector that has the "extra" locations for the boundary conditions in the final destination vector.

  BTW: You know this already, but any implementation that requires storing the "entire" vector on each process is, by definition, not scalable and hence should not even be considered for funding by ECP or SciDAC.


  Barry


> On Mar 8, 2017, at 8:43 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
>> 
>>    Is the scatter created with VecScatterCreateToAll()? If so, internally the VecScatterBegin/End will use VecScatterBegin_MPI_ToAll() which then uses a MPI_Allgatherv() to do the communication.  You can check  in the debugger for this (on 2 processes) by just putting a break point in VecScatterBegin_MPI_ToAll() to confirm if it is called.
> 
> Alas, not I did not use VecScatterCreateToAll and
> VecScatterCreateToAll will take some code changes.
> 
> There are boundary conditions in the destination vector, and so we
> scatter into a larger vector the the global size of the PETSc vector,
> using a general IS. With code that looks like this:
> 
>  call ISCreateGeneral(PETSC_COMM_SELF,nreal,petsc_xgc,PETSC_COPY_VALUES,is,ierr)
>  call VecScatterCreate(this%xVec,PETSC_NULL_OBJECT,vec,is,this%from_petsc,ierr)
> ! reverse scatter object
> 
> If we want to make this change then I could help a developer or you
> can get me set up with a (small) test problem and a branch and I can
> do it at NERSC.
> 
> Thanks,