[petsc-users] Does petsc duplicate the users communicator?

[Junchao Zhang]

On Thu, Jul 8, 2021 at 11:21 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>    Whenever PETSc is handed a communicator it looks for an attribute
> inside of the communicator that contains the "PETSc" version of that
> communicator. If it does not find the attribute it adds an attribute with a
> new communicator in, if it does find one it increases its reference count
> by one. The routine it uses to perform this is PetscCommDuplicate(). We do
> it this was so that PETSc communication will never potentially interfere
> with the users use of their communicators. PetscCommDestroy() decreases the
> reference count of the inner communicator by one. So, for example, if you
> use "comm" to create two PETSc objects, PETSc will create an attribute on
> "comm" with a new communicator, when both objects are destroy then
> PetscCommDestroy() will have been called twice and the inner (PETSc)
> communicator will be destroyed.
>
>   If someone did
>
>       Use MPI to create a new communicator
>       VecCreate(comm,...)
>       Use MPI to destroy the new communicator
>       ....
>       VecDestroy()
>
The code above will work correctly.  In 'Use MPI to destroy the new
communicator',  MPI finds out *comm* has an attribute
Petsc_InnerComm_keyval, so it invokes a petsc function
Petsc_InnerComm_Attr_Delete_Fn (which was given to MPI at PetscInitialize).
In Petsc_InnerComm_Attr_Delete_Fn, it cuts the link between *comm* and its
inner petsc comm (which is still used by vec in this example). The inner
petsc comm is still valid and accessible via PetscObjectComm(). It will be
destroyed when its reference count (managed by petsc) reaches zero
(probably in VecDestroy).



> I am not sure what will happen since PETSc keeps a reference to the outer
> communicator from its own inner communicator. And destroying the user
> communicator will cause an attempt to destroy the attribute containing the
> inner PETSc communicator.  I had always just assumed the user would not be
> deleting any MPI communicators they made and pass to PETSc until they were
> done with PETSc. It may work correctly but may not.
>
> The reality is very few MPI codes have complicated life cycles for MPI
> communicators.
>
> Barry
>
>
> > On Jul 8, 2021, at 10:17 PM, Kozdon, Jeremy (CIV) <jekozdon at nps.edu>
> wrote:
> >
> > Sorry if this is clearly stated somewhere in the docs, I'm still getting
> familiar with the petsc codebase and was also unable to find the answer
> searching (nor could I determine where this would be done in the source).
> >
> > Does petsc duplicate MPI communicators? Or does the users program need
> to make sure that the communicator remains valid for the life of a petsc
> object?
> >
> > The attached little test code seems to suggest that there is some
> duplication of MPI communicators behind the scenes.
> >
> > This came up when working on Julia wrappers for petsc. (Julia has a
> garbage collector so we need to make sure that references are properly kept
> if needed.)
> >
> > <try.c>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210709/2f879757/attachment.html>