<html>
<head></head>
<body> <span style="font-family:sans-serif">Hello Shane<br><br></span> <span style="font-family:sans-serif">Thank you for investigating this and providing work arounds. I really appreciate it.<br><br></span> <span style="font-family:sans-serif">Thank you for your time, stay safe<br> Yours sincerely<br> Neeraj Rajesh <br><br></span>
<div>
<p>Aug 7, 2020 07:21:30 Snyder, Shane <ssnyder@mcs.anl.gov>:</p>
<blockquote dir="ltr" style="">
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
Hi Neeraj,
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
<br style="">
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
Apologies for the delay, but I finally had a chance to dig into this further.
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
<br style="">
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
It turns out the issue is that fio is using the fork() call to spawn another child process that does the actual I/O to the file, with the parent process just using stat a few times on that file (which is why we only see the stat calls of the parent in the log files).
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
<br style="">
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
I recently had another Darshan use case that was using fork(), and I re-worked Darshan a bit so that it would actually generate logs for both the parent and the child processes, which I thought would be all that we would need to get fio working...but it didn't actually work for me when I tried it. Digging further, the child process that does the I/O calls _exit() (rather than exit(), as processes generally do when they terminate), which short-circuits a lot of the shutdown procedures processes will go through, including Darshan's shutdown procedures that generate the log files.
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
<br style="">
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
There's not really a way to work around this, unfortunately. As a matter of best practice, child processes are supposed to call _exit(), so fio is not really doing anything wrong, we just lack the hooks to get instrumentation out of the child. You could manually modify the code so that the child calls exit() instead if you really want instrumentation, it shouldn't cause anything to explode. You could also consider running fio with the '--thread' option which uses pthreads to do the I/O to the file rather than forking another process -- Darhan has no problems instrumenting different threads like that, and I confirmed the logs look normal in that case.
<br style="">
</div>
<br style="">
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
Thanks,
</div>
<div style=" color: rgb(0, 0, 0); font-size: 12pt;font-family: Calibri, Arial, Helvetica, sans-serif">
--Shane
<br style="">
</div>
<div id="appendonsend" style=""></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr" style=""> <font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b style="">From:</b> Neeraj Rajesh <nrajesh@hawk.iit.edu><br style=""> <b style="">Sent:</b> Wednesday, July 22, 2020 5:33 PM<br style=""> <b style="">To:</b> Snyder, Shane <ssnyder@mcs.anl.gov><br style=""> <b style="">Cc:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br style=""> <b style="">Subject:</b> Re: [Ext] Re: [Darshan-users] Issues with Fio benchmark tool being profiled with Darshan</font>
<div style="">
</div>
</div>
<div style=""> <span style="">Thank you Shane. I really appreciate that.<br style=""> <br style=""> </span><span style="">Thank you, stay safe<br style=""> Yours sincerely<br style=""> Neeraj Rajesh<br style=""> <br style=""> </span>
<div style="">
<p style="">Jul 22, 2020 08:35:15 Snyder, Shane <ssnyder@mcs.anl.gov>:</p>
<blockquote dir="ltr" style="">
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Thanks for the update, Neeraj, and thanks for taking the time to test that out for us. That's unfortunate that the problem wasn't as simple as we hoped, but let me take an action item to do some testing with FIO to see if I can reproduce and find a workaround for the problem.
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br style="">
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
I'll let you know what I find.
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br style="">
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
--Shane
<br style="">
</div>
<div id="x_appendonsend" style=""></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr" style=""> <font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b style="">From:</b> Neeraj Rajesh <nrajesh@hawk.iit.edu><br style=""> <b style="">Sent:</b> Tuesday, July 21, 2020 12:47 PM<br style=""> <b style="">To:</b> Snyder, Shane <ssnyder@mcs.anl.gov><br style=""> <b style="">Cc:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br style=""> <b style="">Subject:</b> Re: [Ext] Re: [Darshan-users] Issues with Fio benchmark tool being profiled with Darshan</font>
<div style="">
</div>
</div>
<div style="">
<div dir="ltr" style="">
<div style="">
Hello Shane
</div>
<div style="">
<br style="">
</div>
<div style="">
Sorry for the delay, it looks like there is still an issue with profiling Fio with darshan-master and the counters are still 0.
</div>
<div style="">
<br style="">
</div>
<div style="">
Thank you, stay safe
</div>
<div style="">
Yours sincerely
<br style="">
</div>
<div style="">
Neeraj Rajesh
<br style="">
</div>
</div>
<br style="">
<div class="x_x_gmail_quote" style="">
<div dir="ltr" class="x_x_gmail_attr" style="">
On Wed, Jul 15, 2020 at 1:43 PM Neeraj Rajesh <<a href="mailto:nrajesh@hawk.iit.edu" style="">nrajesh@hawk.iit.edu</a>> wrote:
<br style="">
</div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" style="">
<div style="">
Hello Shane
</div>
<div style="">
<br style="">
</div>
<div style="">
I am using the release version on the Darshan website, (3.2.1). I shall check out the master branch to see if I can generate accurate profiles and can report back on it.
</div>
<div style="">
<br style="">
</div>
<div style="">
Thank you, Stay safe
</div>
<div style="">
Yours Sincerely
<br style="">
</div>
<div style="">
Neeraj Rajesh
<br style="">
</div>
</div>
<br style="">
<div class="x_x_gmail_quote" style="">
<div dir="ltr" class="x_x_gmail_attr" style="">
On Wed, Jul 15, 2020 at 1:35 PM Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank" style="">ssnyder@mcs.anl.gov</a>> wrote:
<br style="">
</div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" style="">
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Hi Neeraj,
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br style="">
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
It looks like FIO might be using the 'openat()' call to open the files it is doing I/O to, which we only recently added support for. Assuming you aren't already, would it be possible for you to try the master branch of Darshan to see whether you can generate accurate profiles using that?
<br style="">
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br style="">
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Thanks,
</div>
<div style="color:rgb(0,0,0); font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
--Shane
<br style="">
</div>
<div id="x_x_gmail-m_6853491819472064376gmail-m_-3199653000104152923appendonsend" style="">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_x_gmail-m_6853491819472064376gmail-m_-3199653000104152923divRplyFwdMsg" dir="ltr" style=""> <font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b style="">From:</b> Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank" style="">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of Neeraj Rajesh <<a href="mailto:nrajesh@hawk.iit.edu" target="_blank" style="">nrajesh@hawk.iit.edu</a>><br style=""> <b style="">Sent:</b> Tuesday, July 14, 2020 3:15 PM<br style=""> <b style="">To:</b> <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank" style=""> darshan-users@lists.mcs.anl.gov</a> <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank" style="">darshan-users@lists.mcs.anl.gov</a>><br style=""> <b style="">Subject:</b> [Darshan-users] Issues with Fio benchmark tool being profiled with Darshan</font>
<div style="">
</div>
</div>
<div style="">
<div dir="ltr" style="">
<div dir="ltr" style="">
<div style="">
Hello
<br style="">
</div>
<div style="">
<br style="">
</div>
<div style="">
I am trying to profile an Fio command, where I do roughly a gigabyte of reads or writes or mixed at a fixed block size.
<br style="">
</div>
<div style="">
eg.
<br style="">
</div>
<div style="">
```
</div>
<div style="">
fio --filename=./test --size=1024MB --direct=1 --rw=randrw --bs=4k --ioengine=sync --iodepth=256 --numjobs=1 --name=test-job --minimal --stats=0
</div>
<div style="">
```
<br style="">
</div>
<div style="">
I have tested it with the following 'ioengines':
</div>
<div style="">
- sync
</div>
<div style="">
- psync
</div>
<div style="">
- posixaio
</div>
<div style="">
- pvsync
</div>
<div style=""></div>
<div style="">
<br style="">
</div>
<div style="">
I do get a darshan file, but all of the POSIX counters are 0, except for POSIX_F_META_TIME, POSIX_STATS, POSIX_FILE_ALIGNMENT and POSIX_MMAPS. I have ensured the operation is not simultaneously doing both POSIX and STDIO.
</div>
<div style="">
<br style="">
</div>
<div style="">
As per my configuration it should not be getting 0s especially for the counters POSIX_SIZE_WRITE_1K_10K, POSIX_WRITES, POSIX_READS, POSIX_BYTES_WRITTEN, POSIX_BYTES_READ etc.
</div>
<div style="">
<br style="">
</div>
<div style="">
Kindly advise what to do next.
</div>
<div style="">
<br style="">
</div>
<div style="">
Thank you for your time, Stay safe
<br style="">
</div>
<div style="">
Yours Sincerely
<br style="">
</div>
<div style="">
Neeraj Rajesh
<br style="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>