[petsc-dev] Expensive VecDuplicate_MPI_DA

[Smith, Barry F.]

  Ahh yes. Since the layout can get large we wanted to avoid each vector having its own copy. 

   But it seems sloppy since DMCreateGlobalVector_DA() does keep a new copy for each vector.

   We could revisit the entire VecDuplicate/DMCreateGlobalVector() infrastructure for all the vector types including the DM ones to 

1) prevent collective calls 
2) reference the layout instead of making new ones. 


  Barry

It looks like VecDuplicate_MPI() does the right thing.


> On Jun 19, 2018, at 10:35 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> "Smith, Barry F." <bsmith at mcs.anl.gov> writes:
> 
>>   Jed,
>> 
>>     You added these two lines. Any idea why they are needed?
> 
> That was just replacing the old interface
> 
> https://bitbucket.org/petsc/petsc/commits/077aedaf8cb5b060ad5dab7a12035daf9d64dd75#Lsrc/dm/impls/da/dadist.cF19
> 
> that you added here
> 
> https://bitbucket.org/petsc/petsc/commits/5ead1ec26b293130cbc6f8fa84f0a9ffce38a0bf
> 
>>    Thanks
>> 
>>    Barry
>> 
>> 3e08d2bebf99 src/dm/impls/da/dadist.c (Barry Smith          2012-09-18  14)   PetscFunctionBegin;
>> c688c0463f6c src/dm/impls/da/dadist.c (Matthew Knepley      2012-11-11  15)   ierr = VecGetDM(g, &da);CHKERRQ(ierr);
>> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith          2010-10-31  16)   ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr);
>> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown            2013-02-11  17)   ierr = VecGetLayout(g,&map);CHKERRQ(ierr);
>> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown            2013-02-11  18)   ierr = VecSetLayout(*gg,map);CHKERRQ(ierr);
>> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith          2010-10-31  19)   PetscFunctionReturn(0)
>> 
>> 
>>   If we were a bit more clever we could probably avoid all communication in VecDuplicate() but is it worth bothering unless we know it is problematic
>> 
>>> On Jun 19, 2018, at 4:30 PM, Junchao Zhang <jczhang at mcs.anl.gov> wrote:
>>> 
>>> I met an expensive VecDuplicate. See the attached call stack. VecDuplicate_MPI_DA indirectly calls MPI_Allreduce/Allgather to build the vector's layout, but it then immediately destroys it with VecGetLayout & VecSetLayout. Is it wrong?
>>> src/dm/impls/da/dadist.c
>>> PetscErrorCode  VecDuplicate_MPI_DA(Vec g,Vec *gg)
>>> {
>>>  PetscErrorCode ierr;
>>>  DM             da;
>>>  PetscLayout    map;
>>> 
>>>  PetscFunctionBegin;
>>>  ierr = VecGetDM(g, &da);CHKERRQ(ierr);
>>>  ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr);
>>>  ierr = VecGetLayout(g,&map);CHKERRQ(ierr);
>>>  ierr = VecSetLayout(*gg,map);CHKERRQ(ierr);
>>>  PetscFunctionReturn(0);
>>> }
>>> 
>>> --Junchao Zhang
>>> <VecDup.png>