[petsc-dev] MPI Keyvalue usage in PETSc
jedbrown at mcs.anl.gov
Thu Jun 6 15:47:33 CDT 2013
"Teranishi, Keita" <knteran at sandia.gov> writes:
> I am curious how you use MPI Keyvalue. Do you have any document or
> source files related to this?
It is used to manage inner/outer communicators. The first time a PETSc
object is created on a communicator, we dup it, creating a
reference-counted inner communicator that we attach via keyval to the
outer and vice-versa. So any time we need to create an object, we
* it is an inner comm -- use it
* it is an outer comm that has an attached inner -- get out the inner and use it
* it has not been seen before -- create the inner and attach
The inner comm has a tag dispenser and a reference count. When the
reference count goes to zero, it is detached from the outer and cleaned
This is way better for library interfaces than having a wrapper class
that goes around the outside of an MPI_Comm, because with our system,
communicators can be passed back and forth between libraries easily,
without needing to re-dup or find the appropriate container.
A story: BG/Q V1R2M0 made MPI_Comm_dup use O(P) memory per rank each
time it is called. This was a catastrophe for applications that used
MPI_Comm_dup for each object that used a communicator. Nek had this
problem, and V1R2M0 was pushed onto Mira shortly before SIAM CSE, in
order to hit some milestone that IBM needed for business reasons.
A later efix resolved the problem, but it was still disruptive for Paul
who had a plenary to give.
You'll find the bulk of the implementation it tagm.c:
More information about the petsc-dev