<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for the details.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
That sequence of calls seems like it ought to work with Darshan, so would be good to see if we can get it right. I'll see if I can test something out locally and report back.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Adrian Jackson <a.jackson@epcc.ed.ac.uk><br>
<b>Sent:</b> Tuesday, July 12, 2022 1:08 PM<br>
<b>To:</b> Snyder, Shane <ssnyder@mcs.anl.gov>; darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Darshan-users] darshan execl issue</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Ah, good question. No, I don't think I am, I'd not picked up that was
<br>
what execl did.<br>
<br>
For full disclosure, I'm not actually calling execl, this is a Fortran <br>
program that is calling system(), which according to the man pages on <br>
our Cray does "The system() library function uses fork(2) to create a <br>
child process that executes the shell command specified in command using <br>
execl(3) as follows:<br>
<br>
execl("/bin/sh", "sh", "-c", command, (char *) 0);<br>
"<br>
<br>
Hence the execl call.<br>
<br>
I think I'll just tell the code owners not to use system() for now.<br>
<br>
cheers<br>
<br>
adrianj<br>
<br>
<br>
<br>
On 12/07/2022 18:57, Snyder, Shane wrote:<br>
> This email was sent to you by someone outside the University.<br>
> You should only click on links or attachments if you are certain that <br>
> the email is genuine and the content is safe.<br>
> Just to clarify, are you doing anything explicit to spawn a new process <br>
> (i.e., fork) ahead of the call to execl? My understanding is that execl <br>
> replaces the calling process, so generally speaking it shouldn't result <br>
> in 2 processes (MPI one + one for system tasks)?<br>
> <br>
> --Shane<br>
> ------------------------------------------------------------------------<br>
> *From:* Adrian Jackson <a.jackson@epcc.ed.ac.uk><br>
> *Sent:* Tuesday, July 12, 2022 10:58 AM<br>
> *To:* Snyder, Shane <ssnyder@mcs.anl.gov>; <br>
> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
> *Subject:* Re: [Darshan-users] darshan execl issue<br>
> Hi Shane,<br>
> <br>
> Thanks for the reply. This is spawning an additional process that runs<br>
> for a bit then ends, the original MPI processes are still all there,<br>
> it's just there is an additional one for a bit. The spawned process<br>
> doesn't do any MPI, it's just doing some system interaction stuff. If<br>
> Darshan intercepts or is triggered by a process ending I think that<br>
> would explain it.<br>
> <br>
> It's something we can work around anyway, we can just not use Darshan<br>
> for this exectuable, I was just checking if it's expected behaviour with<br>
> Darshan or whether we'd stumbled across "an accidental feature" :)<br>
> <br>
> cheers<br>
> <br>
> adrianj<br>
> <br>
> On 12/07/2022 16:53, Snyder, Shane wrote:<br>
>> This email was sent to you by someone outside the University.<br>
>> You should only click on links or attachments if you are certain that <br>
>> the email is genuine and the content is safe.<br>
>> Hi Adrian,<br>
>> <br>
>> So you are replacing one of the MPI processes in MPI_COMM_WOLRD with a <br>
>> new process? In that case, it is probably that this new replacing <br>
>> process is not calling MPI_Finalize which ultimately causes Darshan to <br>
>> hang -- Darshan is intercepting the shutdown call and performing some <br>
>> collective operations for MPI applications, and if one of the ranks <br>
>> disappears these calls will likely just hang. If that's the issue, you <br>
>> could probably reproduce without using Darshan by having your MPI <br>
>> processes run a collective on MPI_COMM_WORLD (like a barrier) _after_ <br>
>> the execl call.<br>
>> <br>
>> A couple of different ideas:<br>
>> <br>
>> * If possible, it might be worth trying to fork ahead of the execl<br>
>> call so that you still have all MPI processes hanging around at<br>
>> shutdown time?<br>
>> * You may be able to run Darshan in non-MPI mode at runtime (using<br>
>> 'export DARSHAN_ENABLE_NONMPI=1') to workaround this problem. This<br>
>> would prevent Darshan from running collectives at shutdown time, but<br>
>> will result in a different log file for each process in your<br>
>> application.<br>
>> <br>
>> Thanks,<br>
>> --Shane<br>
>> ------------------------------------------------------------------------<br>
>> *From:* Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on <br>
>> behalf of Adrian Jackson <a.jackson@epcc.ed.ac.uk><br>
>> *Sent:* Tuesday, July 12, 2022 8:13 AM<br>
>> *To:* darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
>> *Subject:* [Darshan-users] darshan execl issue<br>
>> Hi,<br>
>> <br>
>> I've encountered an issue using Darshan (3.3.1) with a code which calls<br>
>> execl from one MPI process. Using with Darshan the MPI run just hangs.<br>
>> Is spawning processes from a subset of MPI processes an issue for<br>
>> Darshan? I would say that I can still spawn processes (i.e. using fork)<br>
>> and it seems to work, but using execl doesn't.<br>
>> <br>
>> cheers<br>
>> <br>
>> adrianj<br>
>> --<br>
>> Tel: +44 131 6506470 skype: remoteadrianj<br>
>> The University of Edinburgh is a charitable body, registered in <br>
>> Scotland, with registration number SC005336. Is e buidheann carthannais <br>
>> a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh <br>
>> SC005336.<br>
>> _______________________________________________<br>
>> Darshan-users mailing list<br>
>> Darshan-users@lists.mcs.anl.gov<br>
>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a>
<br>
> <<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a>><br>
>> <<a href=""></a>https://lists.mcs.anl.gov/mailman/listinfo/darshan-users <br>
> <<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a>>><br>
> <br>
> -- <br>
> Tel: +44 131 6506470 skype: remoteadrianj<br>
<br>
-- <br>
Tel: +44 131 6506470 skype: remoteadrianj<br>
</div>
</span></font></div>
</body>
</html>