<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Thanks all for this second round of detailed responses. Highly
appreciated! <br>
</p>
<p><br>
</p>
<p>I think that I have enough material to continue exploring a
solution in our particular context.</p>
<p><br>
</p>
<p>Best regards,</p>
<p> Alberto.<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 25/10/21 11:12 pm, Betteridge, Jack
D wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB9PR06MB7387F85CD980AC87A25535E9BA839@DB9PR06MB7387.eurprd06.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hi Everyone,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
I cannot fault Lawrence's explanation, that is precisely what
I'm implementing. The only difference is I was adding most of
the logic for the "resurrected objects map" to petsc4py rather
than PETSc. Given that this solution is truly Python agnostic, I
will move what I have written to C and merely add the interface
to the functionality to petsc4py.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Indeed, this works out better for me as I was not enjoying
writing all the code in Cython! I'll post an update once there
is a working prototype in my PETSc fork, and the code is ready
for testing.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Jack<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
Lawrence Mitchell <a class="moz-txt-link-rfc2396E" href="mailto:wence@gmx.li"><wence@gmx.li></a><br>
<b>Sent:</b> 25 October 2021 12:34<br>
<b>To:</b> Stefano Zampini <a class="moz-txt-link-rfc2396E" href="mailto:stefano.zampini@gmail.com"><stefano.zampini@gmail.com></a><br>
<b>Cc:</b> Barry Smith <a class="moz-txt-link-rfc2396E" href="mailto:bsmith@petsc.dev"><bsmith@petsc.dev></a>; "Alberto F.
Martín" <a class="moz-txt-link-rfc2396E" href="mailto:amartin@cimne.upc.edu"><amartin@cimne.upc.edu></a>; PETSc users list
<a class="moz-txt-link-rfc2396E" href="mailto:petsc-users@mcs.anl.gov"><petsc-users@mcs.anl.gov></a>; Francesc Verdugo
<a class="moz-txt-link-rfc2396E" href="mailto:fverdugo@cimne.upc.edu"><fverdugo@cimne.upc.edu></a>; Betteridge, Jack D
<a class="moz-txt-link-rfc2396E" href="mailto:j.betteridge@imperial.ac.uk"><j.betteridge@imperial.ac.uk></a><br>
<b>Subject:</b> Re: [petsc-users] Why PetscDestroy global
collective semantics?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span
style="font-size:11pt">
<div class="PlainText"><br>
*******************<br>
This email originates from outside Imperial. Do not
click on links and attachments unless you recognise the
sender.
<br>
If you trust the sender, add them to your safe senders
list <a
href="https://spam.ic.ac.uk/SpamConsole/Senders.aspx"
moz-do-not-send="true">
https://spam.ic.ac.uk/SpamConsole/Senders.aspx</a> to
disable email stamping for this address.<br>
*******************<br>
Hi all,<br>
<br>
(I cc Jack who is doing the implementation in the
petsc4py setting)<br>
<br>
> On 24 Oct 2021, at 06:51, Stefano Zampini
<a class="moz-txt-link-rfc2396E" href="mailto:stefano.zampini@gmail.com"><stefano.zampini@gmail.com></a> wrote:<br>
> <br>
> Non-deterministic garbage collection is an issue
from Python too, and firedrake folks are also working on
that.<br>
> <br>
> We may consider deferring all calls to
MPI_Comm_free done on communicators with 1 as ref count
(i.e., the call will actually wipe out some internal MPI
data) in a collective call that can be either run by the
user (on PETSC_COMM_WORLD), or at PetscFinalize() stage.<br>
> I.e., something like that<br>
> <br>
> #define MPI_Comm_free(comm) PutCommInAList(comm)<br>
> <br>
> Comm creation is collective by definition, and thus
collectiveness of the order of the destruction can be
easily enforced.<br>
> I don't see problems with 3rd party libraries using
comms, since we always duplicate the comm we passed them<br>
<br>
> Lawrence, do you think this may help you?<br>
<br>
I think that it is not just MPI_Comm_free that is
potentially problematic.<br>
<br>
Here are some additional areas off the top of my head:<br>
<br>
1. PetscSF with -sf_type window. Destroy (when the
refcount drops to zero) calls MPI_Win_free (which is
collective over comm)<br>
2. Deallocation of MUMPS objects is tremendously
collective. <br>
<br>
In general the solution of just punting MPI_Comm_free to
PetscFinalize (or some user-defined time) is, I think,
insufficient since it requires us to audit the
collectiveness of all `XXX_Destroy` functions (including
in third-party packages).<br>
<br>
Barry's suggestion of resurrecting objects in
finalisation using PetscObjectRegisterDestroy and then
collectively clearing that array periodically is pretty
close to the proposal that we cooked up I think.<br>
<br>
Jack can correct any missteps I make in explanation, but
perhaps this is helpful for Alberto:<br>
<br>
1. Each PETSc communicator gets two new attributes
"creation_index" [an int64], "resurrected_objects" [a
set-like thing]<br>
2. PetscHeaderCreate grabs the next creation_index out
of the input communicator and stashes it on the object.
Since object creation is collective this is guaranteed
to agree on any given communicator across processes.<br>
3. When the Python garbage collector tries to destroy
PETSc objects we resurrect the _C_ object in
finalisation and stash it in "resurrected_objects" on
the communicator.<br>
4. Periodically (as a result of user intervention in the
first instance), we do garbage collection collectively
on these resurrected objects by performing a set
intersection of the creation_indices across the
communicator's processes, and then calling XXXDestroy in
order on the sorted_by_creation_index set intersection.<br>
<br>
<br>
I think that most of this infrastructure is agnostic of
the managed language, so Jack was doing implementation
in PETSc (rather than petsc4py).<br>
<br>
This wasn't a perfect solution (I recall that we could
still cook up situations in which objects would not be
collected), but it did seem to (in theory) solve any
potential deadlock issues.<br>
<br>
Lawrence<br>
</div>
</span></font></div>
</div>
</blockquote>
</body>
</html>