<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 5, 2020 at 3:39 PM Hudson, Stephen Tobias P via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It seems I do have to bypass Python's multiprocessing somewhat limited interface. E.g.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
self.process._popen._send_signal(signal.SIGINT)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
which works, but I am by-passing the API.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I would support allowing the user to configure at run-time the signal handling for SIGTERM to exit without MPI_ABORT. I think I understand MPI_ABORT being the default, I've experienced hangs due to errors on single processes.<br></div></div></blockquote>“hangs due to errors on single processes". If the single processes call exit(), then there will be no hang. <div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div id="gmail-m_8190126136938760286appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8190126136938760286divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Sent:</b> Friday, June 5, 2020 2:41 PM<br>
<b>To:</b> Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Cc:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks, I will experiment with this. <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am working through the multiprocessing interface, but I can see that the routines provided there are pretty much wrappers to the process signal functions.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I guess the alternative is SIGKILL.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Steve<br>
</div>
<div id="gmail-m_8190126136938760286x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8190126136938760286x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 4:54 PM<br>
<b>To:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Cc:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>(1) You can use <font face="monospace">PETSc.Sys.pushErrorHandler("abort")</font>, but it will not help you. What you really need is to override PETSc's default signal handling </div>
<div></div>
</div>
<div><br>
</div>
<div>(2) While it is true that PETSc overrides the signal handler, you can override it again from python after
<font face="monospace">from petsc4py import PETSc</font>.<br>
</div>
<div><br>
</div>
<div>For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following.<br>
</div>
<div><br>
</div>
<div><font face="monospace">from petsc4py import PETSc</font></div>
<div>
<div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">import signal</font></div>
<div></div>
</div>
<div></div>
</div>
<div><font face="monospace">signal.signal(signal.SIGINT, signal.default_int_handler)<br>
</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">...</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">if __name__ == "__main__":</font></div>
<div><font face="monospace"> try:</font></div>
<div><font face="monospace"> main()</font></div>
<div><font face="monospace"> except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT </font></div>
<div><font face="monospace"> ... # do cleanup if needed</font></div>
<div><br>
</div>
<div>Otherwise, you just need <font face="monospace">signal.signal(signal.SIGINT, signal.SIG_DFL)</font></div>
<div><br>
</div>
<div><br>
</div>
<div>PS: I'm not in favor of changing current PETSc's signal handling behavior.</div>
<div>This particular issue is fixable with two lines of Python code:</div>
<div><br>
</div>
<div><font face="monospace">from signal import signal, SIGINT, SIG_DFL</font></div>
<div><font face="monospace">signal(SIGINT, SIG_DFL)<br>
</font></div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Lisandro,<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I don't see an interface to set this through petsc4py. Is it possible?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Steve<br>
</div>
<div id="gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 2:47 PM<br>
<b>To:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sounds good. I will have a look at how to set this through petsc4py.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Steve<br>
</div>
<div id="gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 2:32 PM<br>
<b>To:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>I don't completely understand the issue here. How is sequential run different than parallel run?<br>
<br>
In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with:<br>
<br>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html</a><br>
<br>
And there are a few default error handlers to choose<br>
<br>
<br>
PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
<br>
Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort<br>
<br>
Or perhaps you want to completely disable error handler with: -no_signal_handler<br>
<br>
cc: petsc-users<br>
<br>
Satish<br>
<br>
On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote:<br>
<br>
> Satish,<br>
> <br>
> We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send
a terminate signal that just kills the process.<br>
> <br>
> Is there a way to turn off the mpi_abort?<br>
> <br>
> Thanks,<br>
> <br>
> Steve<br>
> <br>
> <br>
<br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>Lisandro Dalcin<br>
============<br>
Research Scientist<br>
Extreme Computing Research Center (ECRC)<br>
King Abdullah University of Science and Technology (KAUST)<br>
<a href="http://ecrc.kaust.edu.sa/" target="_blank">http://ecrc.kaust.edu.sa/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>