[petsc-dev] Implementing longer pipelines with VecDotBegin and VecDotEnd

[Wim Vanroose]

Dear  Petsc-Dev,

Over the last few year we have contributed several pipelined Krylov
solvers.  Such as KSPPIPECG and  most recently pipelined bicgstab (
pipebcgs).
These make use of asynchronous global reductions using VecDotBegin en
VecDotEnd to overlap the calculation of a dot product with the matrix
vector product.
Experiments by various authors show that these methods can offere better
scaling in the extremely large system limit.

We are now trying to introduce Krylov methods with longer  pipelines.  Such
that the dot-product can take multiple matrix vector products to complete.

Below is a scetch.  After the first SpMV we would like to start a
VecDotBegin,  That would only complete 2 Spmv's, or more, later.
After each SpMV we would start such global reduction.
While trying to implement a length-l version of pipelined CG in PETSc, we
ran across some trouble with the following type of construction
that are representative for the problem abouve.  Let R,U,V,W,X and Y  be
KSP work vectors, and gamma, delta and psi are PetscScalar:

 ierr = VecDotBegin(R,U,&gamma); CHKERRQ(ierr);

 ierr = KSP_MatMult(ksp,Amat, ..., ... ); CHKERRQ(ierr);

 ierr = VecDotBegin(W,V,&delta); CHKERRQ(ierr);

 ierr = KSP_MatMult(ksp,Amat,M,N); CHKERRQ(ierr);

 ierr = VecDotEnd(R,U,&gamma); CHKERRQ(ierr);
 ierr = VecDotBegin(X,Y,&psi); CHKERRQ(ierr);
....

 ierr = VecDotEnd(W,V,&delta); CHKERRQ(ierr);
 ierr = VecDotEnd(X,Y,&psi); CHKERRQ(ierr);

Maybe this is a trivial remark, but it appears that it is not possible to
put a new VecDotBegin (line 7) in between two VecDotEnd's (lines 6 and 8).
Do you have any ideas on why this can't be done (is it intrinsic to
VecDotBegin?), and whether a work-around for this issue is available?

Are there other methods in Petsc  that we should use?   Or are the
VecDotBegin and VecDotEnd not designed to be used in this way?

Thanks a lot for the input,
​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170323/82fdce34/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out (1).png
Type: image/png
Size: 44047 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170323/82fdce34/attachment.png>