[petsc-dev] Expensive VecDuplicate_MPI_DA

[Junchao Zhang]

I am not sure if VecDuplicate is problematic. But we should avoid
collective calls in general. They do more harm than VecScatters, which are
often neighborhood and do not sync all processors.

--Junchao Zhang

On Tue, Jun 19, 2018 at 9:16 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    Jed,
>
>      You added these two lines. Any idea why they are needed?
>
>     Thanks
>
>     Barry
>
> 3e08d2bebf99 src/dm/impls/da/dadist.c (Barry Smith          2012-09-18
> 14)   PetscFunctionBegin;
> c688c0463f6c src/dm/impls/da/dadist.c (Matthew Knepley      2012-11-11
> 15)   ierr = VecGetDM(g, &da);CHKERRQ(ierr);
> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith          2010-10-31
> 16)   ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr);
> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown            2013-02-11
> 17)   ierr = VecGetLayout(g,&map);CHKERRQ(ierr);
> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown            2013-02-11
> 18)   ierr = VecSetLayout(*gg,map);CHKERRQ(ierr);
> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith          2010-10-31
> 19)   PetscFunctionReturn(0)
>
>
>    If we were a bit more clever we could probably avoid all communication
> in VecDuplicate() but is it worth bothering unless we know it is problematic
>
> > On Jun 19, 2018, at 4:30 PM, Junchao Zhang <jczhang at mcs.anl.gov> wrote:
> >
> > I met an expensive VecDuplicate. See the attached call stack.
> VecDuplicate_MPI_DA indirectly calls MPI_Allreduce/Allgather to build the
> vector's layout, but it then immediately destroys it with VecGetLayout &
> VecSetLayout. Is it wrong?
> > src/dm/impls/da/dadist.c
> > PetscErrorCode  VecDuplicate_MPI_DA(Vec g,Vec *gg)
> > {
> >   PetscErrorCode ierr;
> >   DM             da;
> >   PetscLayout    map;
> >
> >   PetscFunctionBegin;
> >   ierr = VecGetDM(g, &da);CHKERRQ(ierr);
> >   ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr);
> >   ierr = VecGetLayout(g,&map);CHKERRQ(ierr);
> >   ierr = VecSetLayout(*gg,map);CHKERRQ(ierr);
> >   PetscFunctionReturn(0);
> > }
> >
> > --Junchao Zhang
> > <VecDup.png>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180619/b9bfd396/attachment.html>