[petsc-dev] Implementing longer pipelines with VecDotBegin and VecDotEnd

[Barry Smith]

   Jed,

      Ok I buy what you are saying as a possible solution but now I am wondering why you need the request argument. Instead, as soon as you hit the first End or PetscCommSplitReductionBegin why don't you start a new batch (internally) that all new Begin() calls feed into? Then each End call just checks through the previous (still incompleted) batches to find its begin partner (or we can require always first in, first out). Thus your first VecNormEnd(Y, would match the first begin with Y posted and your second would match the second. 

   No API changes.

   Barry



> On Mar 23, 2017, at 11:07 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> 
> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>> On Mar 23, 2017, at 10:54 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>> Meh,
>>>>> 
>>>>> VecNormBegin(X,&request1x);
>>>>> VecNormBegin(Y,&request1y);
>>>>> VecNormEnd(X,request1x,&norm);
>>>>> VecAXPY(Y,-1,X);
>>>>> VecNormBegin(Y,&request2y);
>>>>> VecNormEnd(Y,request2y,&norm2y);
>>>>> VecNormEnd(Y,request1y,&norm1y);
>>>> 
>>>>  I don't understand what you are getting at here. You don't seem to be understanding my use case where multiple inner products/norms share the same MPI communication (which was the original reason for VecNormBegin/End) see for example KSPSolve_CR
>>>> 
>>>>   Are you somehow (incompetently) saying that the first two VecNorms
>>>>   somehow share the same parallel communication (even though they
>>>>   have different request values) while the third Norm has its own
>>>>   MPI communication. 
>>> 
>>> Yeah, same as now.  Every time you call *Begin() using a communicator,
>>> you get a new request for something in that "batch".  When the batch is
>>> closed, either by a *End() or PetscCommSplitReductionBegin(), any future
>>> *Begin() calls will go into a new batch.
>> 
>>   Ok, so to support their use case they would need to explicitly call PetscCommSplitReductionBegin?
> 
> Yes, just like all the other pipelined methods do now.
> 
>>   Start first inner product
>>   Do mat mult
>>    do PetscCommSplitReductionBegin() to indicate the first batch is done?
>>    start second inner product (will be handled in a second batch)
>>   do stuff
>>   get result of first inner product
>>   ....
>> 
>>   Ok I can live with this model. But my "hoist" model would have, in some cases fewer objects managed by the user but it would require an explicit 
>> PetscCommSplitReductionCreate().
> 
> Yeah, but that sucks if an outer scope wants to start a reduction
> (perhaps a time integrator's CFL check) without changing the API of its
> solver to plumb the reduction in.