[petsc-dev] MPI Keyvalue usage in PETSc

[Jed Brown]

"Teranishi, Keita" <knteran at sandia.gov> writes:

> Jed,
>
> I am curious how you use MPI Keyvalue.  Do you have any document or
> source files related to this?

It is used to manage inner/outer communicators.  The first time a PETSc
object is created on a communicator, we dup it, creating a
reference-counted inner communicator that we attach via keyval to the
outer and vice-versa.  So any time we need to create an object, we
check:

* it is an inner comm -- use it
* it is an outer comm that has an attached inner -- get out the inner and use it
* it has not been seen before -- create the inner and attach

The inner comm has a tag dispenser and a reference count.  When the
reference count goes to zero, it is detached from the outer and cleaned
up.

This is way better for library interfaces than having a wrapper class
that goes around the outside of an MPI_Comm, because with our system,
communicators can be passed back and forth between libraries easily,
without needing to re-dup or find the appropriate container.

A story: BG/Q V1R2M0 made MPI_Comm_dup use O(P) memory per rank each
time it is called.  This was a catastrophe for applications that used
MPI_Comm_dup for each object that used a communicator.  Nek had this
problem, and V1R2M0 was pushed onto Mira shortly before SIAM CSE, in
order to hit some milestone that IBM needed for business reasons.

A later efix resolved the problem, but it was still disruptive for Paul
who had a plenary to give.

You'll find the bulk of the implementation it tagm.c:

https://bitbucket.org/petsc/petsc/src/a0a914e661bf6402b8edabe0f5a2dad46323f69f/src/sys/objects/tagm.c?at=master