<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi Shane,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I tried the WIP branch. I set these environment variables:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US">export DARSHAN_LOGHINTS=<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US">export DARSHAN_MODMEM=32<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US">export DARSHAN_CONF_PATH=<absolute-path>/darshan.config<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">And my darshan.config has:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US"># comments are allowed using '#'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US"># KEY VALUE MODULES<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US">MOD_ENABLE POSIX,STDIO<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New";mso-fareast-language:EN-US">MAX_RECORDS 16384 *<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">However, no matter the value I set for the MAX_RECORDS, the darshan log does not change. Am I using the config file correctly? I am sure that this code does not read/write 16k files.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Also, as I have not enabled the H5D module in the config file, should it complain about the incomplete data when using darshan-parser? For a test, I reduced the MAX_RECORDS for another
application that uses NetCDF-4 files; and I got an error about incomplete data for the H5D module.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Vineet<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Snyder, Shane <ssnyder@mcs.anl.gov>
<br>
<b>Sent:</b> Tuesday, November 9, 2021 6:36 PM<br>
<b>To:</b> Vineet Soni <vsoni@mercator-ocean.fr>; Harms, Kevin <harms@alcf.anl.gov>; darshan-users@lists.mcs.anl.gov<br>
<b>Subject:</b> Re: Darshan & IPM results not the same<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">Hi Vineet,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">We've been getting reports from users more and more about this problem recently, but as Kevin mentions we hope to have something more configurable for our next release.
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black">Copying a response I had recently for another user:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #C8C8C8 2.25pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">We do have a work-in-progress PR on our repo that is supposed to help workaround this limit:
</span><a href="https://github.com/darshan-hpc/darshan/pull/405"><span lang="EN-US" style="font-size:12.0pt">https://github.com/darshan-hpc/darshan/pull/405</span></a><span style="font-size:12.0pt;color:black">
<span lang="EN-US"><o:p></o:p></span></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">That PR is generally about modifying Darshan to accept different configuration options from users at runtime (currently only using a config file, but ultimately there will be corresponding
env variables to control things), including an ability to control how many records are pre-allocated for each module (via MAX_RECORDS configuration setting). I think things work well enough on that branch that you could try requesting more records for the
POSIX module, or whatever modules you're interested in, and see how things work. The PR has some details on how to do this.<o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">So there is a WIP branch you can use, but it still needs to be cleaned up. I think you should have better luck if you use the config file to set a more appropriate MAX_RECORDS value
for POSIX module (and whatever other modules you have that run out of memory). You should probably set DARSHAN_MODMEM to a higher value than the default, too, but it seems really unlikely to me that you'd need GiBs of memory as you mention in your last response
(and this in fact could cause Darshan to crash) -- a reasonable way to estimate a value is to assume Darshan needs 1 or 2 KiB of memory for each record it tracks (i.e. MAX_RECORDS value from above).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black">Hopefully that's enough to get you started, but please let us know if you have more problems, questions, or feedback.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Thanks!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">--Shane<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span lang="EN-US" style="color:black">From:</span></b><span lang="EN-US" style="color:black"> Darshan-users <</span><a href="mailto:darshan-users-bounces@lists.mcs.anl.gov"><span lang="EN-US">darshan-users-bounces@lists.mcs.anl.gov</span></a><span lang="EN-US" style="color:black">>
on behalf of Vineet Soni <</span><a href="mailto:vsoni@mercator-ocean.fr"><span lang="EN-US">vsoni@mercator-ocean.fr</span></a><span lang="EN-US" style="color:black">><br>
<b>Sent:</b> Tuesday, November 9, 2021 9:48 AM<br>
<b>To:</b> Harms, Kevin <</span><a href="mailto:harms@alcf.anl.gov"><span lang="EN-US">harms@alcf.anl.gov</span></a><span lang="EN-US" style="color:black">>;
</span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US" style="color:black"> <</span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US" style="color:black">><br>
<b>Subject:</b> Re: [Darshan-users] Darshan & IPM results not the same</span><span lang="EN-US">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks Kevin for the explanation.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Unfortunately, all of the files are read from a single directory. So, it's not possible to exclude any.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Do you have any rough estimation of when this new version will be available? Or, is there a gitlab-branch that I can test?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Vineet<o:p></o:p></p>
</div>
</div>
<div id="x_id-e8520250-72ab-47c6-a607-cb030be34553">
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" noshade="" style="color:black" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Harms, Kevin <</span><a href="mailto:harms@alcf.anl.gov"><span lang="EN-US">harms@alcf.anl.gov</span></a><span lang="EN-US">><br>
<b>Sent:</b> Tuesday, November 9, 2021 3:58 PM<br>
<b>To:</b> Vineet Soni; </span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US"><br>
<b>Subject:</b> Re: Darshan & IPM results not the same <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">Vineet,<br>
<br>
ok, so the problem seems to be you are exceeding the maximum limit of files per process (1024). After darshan hits this limit, it will not record any other files. Raising the memory limit will not change the file limit. If the files you don't care about are
in a different directory than the files you do care about, you can use this variable:<br>
<br>
DARSHAN_EXCLUDE_DIRS<br>
<br>
A list of comma-separated paths that Darshan will not instrument at runtime (in addition to Darshan’s default blacklist)<br>
<br>
We are working on an updated version that will allow users to specify a higher file limit as well as more complex patterns for excluding files.<br>
<br>
kevin<br>
<br>
<br>
________________________________________<br>
From: Vineet Soni <</span><a href="mailto:vsoni@mercator-ocean.fr"><span lang="EN-US">vsoni@mercator-ocean.fr</span></a><span lang="EN-US">><br>
Sent: Tuesday, November 9, 2021 3:47 AM<br>
To: Harms, Kevin; </span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US"><br>
Subject: RE: Darshan & IPM results not the same<br>
<br>
Hi Kevin,<br>
<br>
The code does not use threading. And yes, there are many files I don't see in the darshan log, and they are relatively large compared to the ones intercepted.<br>
And, the application does have fread() calls. But, the STDIO module does not have a significant value in total_STDIO_F_READ_TIME.<br>
<br>
I realized that there are warnings in POSIX and STDIO modules about the incomplete data in the log. However, I see no change in the log even after setting DARSHAN_MODMEM to 1024 MiB.<br>
Also, even though the application occupies only 110 GB memory out of 256 GB per node, setting DARSHAN_MODMEM to higher values such as 4096 MiB crashes the job (which makes me think that this value is per process - 128 per node?).<br>
<br>
Is there any runtime environment variable to set for excluding a group of files instead of directories?<br>
<br>
Thanks,<br>
Vineet<br>
<br>
<br>
-----Original Message-----<br>
From: Harms, Kevin <</span><a href="mailto:harms@alcf.anl.gov"><span lang="EN-US">harms@alcf.anl.gov</span></a><span lang="EN-US">><br>
Sent: Monday, November 8, 2021 8:36 PM<br>
To: Vineet Soni <</span><a href="mailto:vsoni@mercator-ocean.fr"><span lang="EN-US">vsoni@mercator-ocean.fr</span></a><span lang="EN-US">>;
</span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US"><br>
Subject: Re: Darshan & IPM results not the same<br>
<br>
Vineet,<br>
<br>
a few ideas:<br>
- is the I/O done using fread() or similar? These are accounted under the STDIO module rather than the POSIX module. Can you check to see what STDIO module shows?<br>
- is the application threaded? It's possible an issue with threading, but given the disparity that seems less likely.<br>
- Perhaps an issue with darshan not intercepting a subset of the calls your application is making. If you look at the file name list, does it seem obvious that darshan is missing I/O from some set of files? (This could also be due to files being caught under
the exclude list)<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <</span><a href="mailto:darshan-users-bounces@lists.mcs.anl.gov"><span lang="EN-US">darshan-users-bounces@lists.mcs.anl.gov</span></a><span lang="EN-US">> on behalf of Vineet Soni <</span><a href="mailto:vsoni@mercator-ocean.fr"><span lang="EN-US">vsoni@mercator-ocean.fr</span></a><span lang="EN-US">><br>
Sent: Monday, November 8, 2021 4:05 AM<br>
To: </span><a href="mailto:darshan-users@lists.mcs.anl.gov"><span lang="EN-US">darshan-users@lists.mcs.anl.gov</span></a><span lang="EN-US"><br>
Subject: [Darshan-users] Darshan & IPM results not the same<br>
<br>
Hello,<br>
<br>
I am trying to analyze the IO behavior of our codes with Darshan.<br>
Darshan: 3.3.0<br>
Compilers: Intel 2018<br>
MPI: Intel MPI 2018<br>
FS: Lustre (lustre-module disabled in Darshan configuration)<br>
Darshan profiling: LD_PRELOAD<br>
<br>
I observe a big difference in IO results from Darshan and IPM (v2.0.5) for one of our codes. I guess that both profilers are not profiling the same POSIX calls?<br>
<br>
The POSIXIO calls profiled in IPM are:<br>
<br>
fopen, fdopen, freopen, open, open64<br>
fclose, close<br>
fflush<br>
fread, read<br>
fwrite, write<br>
fseek, lseek, lseek64<br>
ftell<br>
rewind<br>
fgetpos, fsetpos, fgetc, getc, ungetc<br>
creat<br>
truncate, ftruncate, truncate64, ftruncate64<br>
<br>
While the ones profiled by Darshan are: </span><a href="https://github.com/darshan-hpc/darshan/blob/main/darshan-runtime/lib/darshan-posix.c"><span lang="EN-US">https://github.com/darshan-hpc/darshan/blob/main/darshan-runtime/lib/darshan-posix.c</span></a><span lang="EN-US">
?<br>
<br>
However, the huge difference is observed in the “read” call, which exists in both the profilers.<br>
<br>
+-------------------+------------+-----------+<br>
| | IPM | Darshan |<br>
+-------------------+------------+-----------+<br>
| Read (s) | 324.57 | 6.02 |<br>
+-------------------+------------+-----------+<br>
| Agg. Read (count) | 34 766 456 | 2 946 271 |<br>
+-------------------+------------+-----------+<br>
<br>
I tested Darshan and IPM with other codes (reading NOT the same files) to check if this issue is faced in them as well. But, I got the same results.<br>
So, I don't understand what could be the reason that this application is not giving the same results.<br>
<br>
Do you have any idea of why this could happen?<br>
<br>
Thanks in advance.<br>
<br>
PS: The application does a lot of IO, and is expected to spend a significant time in read operations.<br>
<br>
</span>Best regards,<br>
Vineet<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>