<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Everyone,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I cannot fault Lawrence's explanation, that is precisely what I'm implementing. The only difference is I was adding most of the logic for the "resurrected objects map" to petsc4py rather than PETSc. Given that this solution is truly Python agnostic, I will
move what I have written to C and merely add the interface to the functionality to petsc4py.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Indeed, this works out better for me as I was not enjoying writing all the code in Cython! I'll post an update once there is a working prototype in my PETSc fork, and the code is ready for testing.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Jack<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Lawrence Mitchell <wence@gmx.li><br>
<b>Sent:</b> 25 October 2021 12:34<br>
<b>To:</b> Stefano Zampini <stefano.zampini@gmail.com><br>
<b>Cc:</b> Barry Smith <bsmith@petsc.dev>; "Alberto F. Martín" <amartin@cimne.upc.edu>; PETSc users list <petsc-users@mcs.anl.gov>; Francesc Verdugo <fverdugo@cimne.upc.edu>; Betteridge, Jack D <j.betteridge@imperial.ac.uk><br>
<b>Subject:</b> Re: [petsc-users] Why PetscDestroy global collective semantics?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText"><br>
*******************<br>
This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender.
<br>
If you trust the sender, add them to your safe senders list <a href="https://spam.ic.ac.uk/SpamConsole/Senders.aspx">
https://spam.ic.ac.uk/SpamConsole/Senders.aspx</a> to disable email stamping for this address.<br>
*******************<br>
Hi all,<br>
<br>
(I cc Jack who is doing the implementation in the petsc4py setting)<br>
<br>
> On 24 Oct 2021, at 06:51, Stefano Zampini <stefano.zampini@gmail.com> wrote:<br>
> <br>
> Non-deterministic garbage collection is an issue from Python too, and firedrake folks are also working on that.<br>
> <br>
> We may consider deferring all calls to MPI_Comm_free done on communicators with 1 as ref count (i.e., the call will actually wipe out some internal MPI data) in a collective call that can be either run by the user (on PETSC_COMM_WORLD), or at PetscFinalize()
stage.<br>
> I.e., something like that<br>
> <br>
> #define MPI_Comm_free(comm) PutCommInAList(comm)<br>
> <br>
> Comm creation is collective by definition, and thus collectiveness of the order of the destruction can be easily enforced.<br>
> I don't see problems with 3rd party libraries using comms, since we always duplicate the comm we passed them<br>
<br>
> Lawrence, do you think this may help you?<br>
<br>
I think that it is not just MPI_Comm_free that is potentially problematic.<br>
<br>
Here are some additional areas off the top of my head:<br>
<br>
1. PetscSF with -sf_type window. Destroy (when the refcount drops to zero) calls MPI_Win_free (which is collective over comm)<br>
2. Deallocation of MUMPS objects is tremendously collective. <br>
<br>
In general the solution of just punting MPI_Comm_free to PetscFinalize (or some user-defined time) is, I think, insufficient since it requires us to audit the collectiveness of all `XXX_Destroy` functions (including in third-party packages).<br>
<br>
Barry's suggestion of resurrecting objects in finalisation using PetscObjectRegisterDestroy and then collectively clearing that array periodically is pretty close to the proposal that we cooked up I think.<br>
<br>
Jack can correct any missteps I make in explanation, but perhaps this is helpful for Alberto:<br>
<br>
1. Each PETSc communicator gets two new attributes "creation_index" [an int64], "resurrected_objects" [a set-like thing]<br>
2. PetscHeaderCreate grabs the next creation_index out of the input communicator and stashes it on the object. Since object creation is collective this is guaranteed to agree on any given communicator across processes.<br>
3. When the Python garbage collector tries to destroy PETSc objects we resurrect the _C_ object in finalisation and stash it in "resurrected_objects" on the communicator.<br>
4. Periodically (as a result of user intervention in the first instance), we do garbage collection collectively on these resurrected objects by performing a set intersection of the creation_indices across the communicator's processes, and then calling XXXDestroy
in order on the sorted_by_creation_index set intersection.<br>
<br>
<br>
I think that most of this infrastructure is agnostic of the managed language, so Jack was doing implementation in PETSc (rather than petsc4py).<br>
<br>
This wasn't a perfect solution (I recall that we could still cook up situations in which objects would not be collected), but it did seem to (in theory) solve any potential deadlock issues.<br>
<br>
Lawrence<br>
</div>
</span></font></div>
</div>
</body>
</html>