[petsc-users] Time cost by Vec Assembly

[Barry Smith]

> On Oct 7, 2016, at 10:44 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Barry Smith <bsmith at mcs.anl.gov> writes:
>>    VecAssemblyBegin/End() does a couple of all reduces and then message passing (if values need to be moved) to get the values onto the correct processes. So these calls should take very little time. Something is wonky on your system with that many MPI processes, with these calls. I don't know why, if you look at the code you'll see it is pretty straightforward.
> 
> Those MPI calls can be pretty sucky on some networks.  Dave encountered
> this years ago when they were using VecSetValues/VecAssembly rather
> heavily.  I think that most performance-aware PETSc applications
> typically never tried to use VecSetValues/VecAssembly or they did not
> need to do it very often (e.g., as part of a matrix-free solver).  The
> BTS implementation fixes the performance issue, but I'm still working on
> solving the corner case that has been reported.  Fortunately, the
> VecAssembly is totally superfluous to this user.

   Jed,

    There is still something wonky here, whether it is the MPI implementation or how PETSc handles the assembly. Without any values that need to be communicated it is unacceptably that these calls take so long. If we understood __exactly__ why the performance suddenly drops so dramatically we could perhaps fix it. I do not understand why.


  Barry