[petsc-dev] PETSc object management of MPI communicators

[Barry Smith]

On Jan 9, 2014, at 3:00 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> "Garth N. Wells" <gnw20 at cam.ac.uk> writes:
> 
>> Hi Jed,
>> 
>> I've been poking around the PETSc code to see how objects manage the MPI 
>> communicator that is passed in, and running some simple tests using 
>> MPI_Comm_compare. Does a PETSc object (say a Vec) always duplicate the 
>> communicator? From tests with MPI_Comm_compare this looks to be the 
>> case.
> 
> The first time a communicator is encountered by a PETSc object,
> PetscCommDuplicate dups it, yielding an "inner" communicator.  It
> attaches the inner communicator, a reference count, and a tag dispenser
> to the outer communicator that you just passed in.  We always use the
> tag dispenser to get new tags, always perform operations on the inner
> communicator.  The user never handles the inner communicator.
> Subsequent calls to PetscCommDuplicate will find the attribute and just
> pull out the inner communicator.
> 
> The outer communicator can be passed around, including through other
> libraries, and back into PETSc, yet there will still only ever be one
> inner comm per outer comm.  When an object is destroyed, the reference
> count decreases and the inner communicator is destroyed when the
> reference count drops to zero.
> 
> Barry, this code has been around for ages, but we should write a short
> report explaining what we do and why.

   We do this because it is necessary to do this (or something similar), for any MPI based library. I am hoping that MPI will be replaced by another programming model before it is necessary to get the wider community to understand why it is necessary to do this and how we do it :-(

   Perhaps we should pull out the code into its own little library that can be used by the various library writing groups? 

   There are many things we could/should do, but how much time to we want to spend on dying legacies :-)


  Barry