<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt">Ok, having looked at this a bit more, I'm inclined to support Junchao's approach, but there seems to be concern that, even if the standards support it, there could be issues<br>
in some scenarios.</span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt">I don't have enough information to dispute this. But if this was put in many years ago, I'm interested in what the other MPI libraries do now - eg. does trilinos etc use MPI_ABORT in the signal handler. If not, do
users report issues hanging on terminate?</span></font><br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Friday, June 5, 2020 7:26 PM<br>
<b>To:</b> Hudson, Stephen Tobias P <shudson@anl.gov><br>
<b>Cc:</b> Lisandro Dalcin <dalcinl@gmail.com>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Jun 5, 2020 at 3:39 PM Hudson, Stephen Tobias P via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
It seems I do have to bypass Python's multiprocessing somewhat limited interface. E.g.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
self.process._popen._send_signal(signal.SIGINT)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
which works, but I am by-passing the API.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I would support allowing the user to configure at run-time the signal handling for SIGTERM to exit without MPI_ABORT. I think I understand MPI_ABORT being the default, I've experienced hangs due to errors on single processes.<br>
</div>
</div>
</blockquote>
“hangs due to errors on single processes". If the single processes call exit(), then there will be no hang.
<div><br>
</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div id="x_gmail-m_8190126136938760286appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_8190126136938760286divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Sent:</b> Friday, June 5, 2020 2:41 PM<br>
<b>To:</b> Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Cc:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks, I will experiment with this. <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am working through the multiprocessing interface, but I can see that the routines provided there are pretty much wrappers to the process signal functions.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I guess the alternative is SIGKILL.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Steve<br>
</div>
<div id="x_gmail-m_8190126136938760286x_appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_8190126136938760286x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 4:54 PM<br>
<b>To:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Cc:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>(1) You can use <font face="monospace">PETSc.Sys.pushErrorHandler("abort")</font>, but it will not help you. What you really need is to override PETSc's default signal handling </div>
<div></div>
</div>
<div><br>
</div>
<div>(2) While it is true that PETSc overrides the signal handler, you can override it again from python after
<font face="monospace">from petsc4py import PETSc</font>.<br>
</div>
<div><br>
</div>
<div>For implementing (2), maybe you should try sending SIGINT and not SIGTERM, such that you can do the following.<br>
</div>
<div><br>
</div>
<div><font face="monospace">from petsc4py import PETSc</font></div>
<div>
<div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">import signal</font></div>
<div></div>
</div>
<div></div>
</div>
<div><font face="monospace">signal.signal(signal.SIGINT, signal.default_int_handler)<br>
</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">...</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">if __name__ == "__main__":</font></div>
<div><font face="monospace"> try:</font></div>
<div><font face="monospace"> main()</font></div>
<div><font face="monospace"> except KeyboardInterrupt: # Triggered if Ctrl+C or signaled with SIGINT </font></div>
<div><font face="monospace"> ... # do cleanup if needed</font></div>
<div><br>
</div>
<div>Otherwise, you just need <font face="monospace">signal.signal(signal.SIGINT, signal.SIG_DFL)</font></div>
<div><br>
</div>
<div><br>
</div>
<div>PS: I'm not in favor of changing current PETSc's signal handling behavior.</div>
<div>This particular issue is fixable with two lines of Python code:</div>
<div><br>
</div>
<div><font face="monospace">from signal import signal, SIGINT, SIG_DFL</font></div>
<div><font face="monospace">signal(SIGINT, SIG_DFL)<br>
</font></div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, 4 Jun 2020 at 23:39, Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Lisandro,<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I don't see an interface to set this through petsc4py. Is it possible?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Steve<br>
</div>
<div id="x_gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694appendonsend">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 2:47 PM<br>
<b>To:</b> Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Sounds good. I will have a look at how to set this through petsc4py.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Steve<br>
</div>
<div id="x_gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694x_appendonsend">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_8190126136938760286x_x_gmail-m_7254306572247330694x_divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
<b>Sent:</b> Thursday, June 4, 2020 2:32 PM<br>
<b>To:</b> Hudson, Stephen Tobias P <<a href="mailto:shudson@anl.gov" target="_blank">shudson@anl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>><br>
<b>Subject:</b> Re: Terminating a process running petsc via petsc4py without mpi_abort</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>I don't completely understand the issue here. How is sequential run different than parallel run?<br>
<br>
In both cases - a PetscErrorHandler is likely getting invoked. One can change this behavior with:<br>
<br>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPushErrorHandler.html</a><br>
<br>
And there are a few default error handlers to choose<br>
<br>
<br>
PETSC_EXTERN PetscErrorCode PetscTraceBackErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscIgnoreErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscEmacsClientErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscMPIAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscAbortErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscAttachDebuggerErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
PETSC_EXTERN PetscErrorCode PetscReturnErrorHandler(MPI_Comm,int,const char*,const char*,PetscErrorCode,PetscErrorType,const char*,void*);<br>
<br>
Some of the are accessible via command line option. for ex: -on_error_abort or -on_error_mpiabort<br>
<br>
Or perhaps you want to completely disable error handler with: -no_signal_handler<br>
<br>
cc: petsc-users<br>
<br>
Satish<br>
<br>
On Thu, 4 Jun 2020, Hudson, Stephen Tobias P wrote:<br>
<br>
> Satish,<br>
> <br>
> We are having issues caused by MPI_abort getting called when we try to terminate a sub-process running petsc4py. Ideally we would always use a serial build of petsc/petsc4py in this mode, but many users will have a parallel build. We need to be able to send
a terminate signal that just kills the process.<br>
> <br>
> Is there a way to turn off the mpi_abort?<br>
> <br>
> Thanks,<br>
> <br>
> Steve<br>
> <br>
> <br>
<br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>Lisandro Dalcin<br>
============<br>
Research Scientist<br>
Extreme Computing Research Center (ECRC)<br>
King Abdullah University of Science and Technology (KAUST)<br>
<a href="http://ecrc.kaust.edu.sa/" target="_blank">http://ecrc.kaust.edu.sa/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>