[petsc-dev] VecScatter scaling problem on KNL

[Mark Adams]

>    Ok, in this situation VecScatter cannot detect that it is an all to all so will generate a message from each process to each other process. Given my past experience with Cray MPI (why do they even have their own MPI when Intel provides one; in fact why does Cray even exist when they just take other people's products and put their name on them) I am not totally surprised if the Cray MPI chocks on this flood of messages.
>
>    1) Test with Intel MPI, perhaps they handle this case in a scalable way
>
>     2) If Intel MPI also produces poor performance then (interesting, how come on other systems in the past this wasn't a bottleneck for the code?) the easiest solution is to separate the operation into two parts. Use a VecScatterCreateToAll() to get all the data to all the processes and then use another (purely sequential) VecScatter to get the data from this intermediate buffer into the final vector that has the "extra" locations for the boundary conditions in the final destination vector.

Yes, this is what I am thinking I will do. This sort of problem will
only get worse so we might as well do it at some point and I would bet
that we could just use Intel MPI now to get this project moving now.