[petsc-users] Does petsc duplicate the users communicator?

[Kozdon, Jeremy (CIV)]

This is all super helpful! Thanks.

It seems to me that we do not need to carry around a reference to the communicator in Julia then.

Mainly I wanted to use `PetscObjectGetComm` everywhere once `PetscObjects` were created, but someone pointed out this might run in to problems with the garbage collector.

In my mind it makes sense to rely on `PetscObjectGetComm` since you don’t know if this is some object that matches the object you originally created or some derived object with a different processor distribution (such as what I believe happens with multigrid).

> On Jul 9, 2021, at 8:52 AM, Junchao Zhang <junchao.zhang at gmail.com> wrote:
> 
> 
> NPS WARNING: *external sender* verify before acting.
> 
> 
> 
> 
> 
> On Thu, Jul 8, 2021 at 11:21 PM Barry Smith <bsmith at petsc.dev> wrote:
> 
>    Whenever PETSc is handed a communicator it looks for an attribute inside of the communicator that contains the "PETSc" version of that communicator. If it does not find the attribute it adds an attribute with a new communicator in, if it does find one it increases its reference count by one. The routine it uses to perform this is PetscCommDuplicate(). We do it this was so that PETSc communication will never potentially interfere with the users use of their communicators. PetscCommDestroy() decreases the reference count of the inner communicator by one. So, for example, if you use "comm" to create two PETSc objects, PETSc will create an attribute on "comm" with a new communicator, when both objects are destroy then PetscCommDestroy() will have been called twice and the inner (PETSc) communicator will be destroyed. 
> 
>   If someone did 
> 
>       Use MPI to create a new communicator
>       VecCreate(comm,...)
>       Use MPI to destroy the new communicator
>       ....
>       VecDestroy()
> The code above will work correctly.  In 'Use MPI to destroy the new communicator',  MPI finds out comm has an attribute Petsc_InnerComm_keyval, so it invokes a petsc function Petsc_InnerComm_Attr_Delete_Fn (which was given to MPI at PetscInitialize).
> In Petsc_InnerComm_Attr_Delete_Fn, it cuts the link between comm and its inner petsc comm (which is still used by vec in this example). The inner petsc comm is still valid and accessible via PetscObjectComm(). It will be destroyed when its reference count (managed by petsc) reaches zero (probably in VecDestroy).
> 
>  
> I am not sure what will happen since PETSc keeps a reference to the outer communicator from its own inner communicator. And destroying the user communicator will cause an attempt to destroy the attribute containing the inner PETSc communicator.  I had always just assumed the user would not be deleting any MPI communicators they made and pass to PETSc until they were done with PETSc. It may work correctly but may not.
> 
> The reality is very few MPI codes have complicated life cycles for MPI communicators. 
> 
> Barry
> 
> 
> > On Jul 8, 2021, at 10:17 PM, Kozdon, Jeremy (CIV) <jekozdon at nps.edu> wrote:
> > 
> > Sorry if this is clearly stated somewhere in the docs, I'm still getting familiar with the petsc codebase and was also unable to find the answer searching (nor could I determine where this would be done in the source).
> > 
> > Does petsc duplicate MPI communicators? Or does the users program need to make sure that the communicator remains valid for the life of a petsc object?
> > 
> > The attached little test code seems to suggest that there is some duplication of MPI communicators behind the scenes.
> > 
> > This came up when working on Julia wrappers for petsc. (Julia has a garbage collector so we need to make sure that references are properly kept if needed.)
> > 
> > <try.c>
>