<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="ltr">
<div></div>
<div>
<div dir="ltr">Thanks Kevin for the explanation.</div>
<div dir="ltr">Unfortunately, all of the files are read from a single directory. So, it's not possible to exclude any.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Do you have any rough estimation of when this new version will be available? Or, is there a gitlab-branch that I can test?</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Thanks,</div>
<div dir="ltr">Vineet</div>
<div id="ms-outlook-mobile-signature" dir="ltr">
<div></div>
</div>
</div>
<div id="id-e8520250-72ab-47c6-a607-cb030be34553" class="ms-outlook-mobile-reference-message">
<hr style="display: inline-block; width: 98%; font-family: -webkit-standard; font-size: 12pt; color: rgb(0, 0, 0);" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif"><b>From:</b> Harms, Kevin <harms@alcf.anl.gov><br>
<b>Sent:</b> Tuesday, November 9, 2021 3:58 PM<br>
<b>To:</b> Vineet Soni; darshan-users@lists.mcs.anl.gov<br>
<b>Subject:</b> Re: Darshan & IPM results not the same
<div> </div>
</font></div>
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Vineet,<br>
<br>
ok, so the problem seems to be you are exceeding the maximum limit of files per process (1024). After darshan hits this limit, it will not record any other files. Raising the memory limit will not change the file limit. If the files you don't care about are
in a different directory than the files you do care about, you can use this variable:<br>
<br>
DARSHAN_EXCLUDE_DIRS<br>
<br>
A list of comma-separated paths that Darshan will not instrument at runtime (in addition to Darshan’s default blacklist)<br>
<br>
We are working on an updated version that will allow users to specify a higher file limit as well as more complex patterns for excluding files.<br>
<br>
kevin<br>
<br>
<br>
________________________________________<br>
From: Vineet Soni <vsoni@mercator-ocean.fr><br>
Sent: Tuesday, November 9, 2021 3:47 AM<br>
To: Harms, Kevin; darshan-users@lists.mcs.anl.gov<br>
Subject: RE: Darshan & IPM results not the same<br>
<br>
Hi Kevin,<br>
<br>
The code does not use threading. And yes, there are many files I don't see in the darshan log, and they are relatively large compared to the ones intercepted.<br>
And, the application does have fread() calls. But, the STDIO module does not have a significant value in total_STDIO_F_READ_TIME.<br>
<br>
I realized that there are warnings in POSIX and STDIO modules about the incomplete data in the log. However, I see no change in the log even after setting DARSHAN_MODMEM to 1024 MiB.<br>
Also, even though the application occupies only 110 GB memory out of 256 GB per node, setting DARSHAN_MODMEM to higher values such as 4096 MiB crashes the job (which makes me think that this value is per process - 128 per node?).<br>
<br>
Is there any runtime environment variable to set for excluding a group of files instead of directories?<br>
<br>
Thanks,<br>
Vineet<br>
<br>
<br>
-----Original Message-----<br>
From: Harms, Kevin <harms@alcf.anl.gov><br>
Sent: Monday, November 8, 2021 8:36 PM<br>
To: Vineet Soni <vsoni@mercator-ocean.fr>; darshan-users@lists.mcs.anl.gov<br>
Subject: Re: Darshan & IPM results not the same<br>
<br>
Vineet,<br>
<br>
a few ideas:<br>
- is the I/O done using fread() or similar? These are accounted under the STDIO module rather than the POSIX module. Can you check to see what STDIO module shows?<br>
- is the application threaded? It's possible an issue with threading, but given the disparity that seems less likely.<br>
- Perhaps an issue with darshan not intercepting a subset of the calls your application is making. If you look at the file name list, does it seem obvious that darshan is missing I/O from some set of files? (This could also be due to files being caught under
the exclude list)<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Vineet Soni <vsoni@mercator-ocean.fr><br>
Sent: Monday, November 8, 2021 4:05 AM<br>
To: darshan-users@lists.mcs.anl.gov<br>
Subject: [Darshan-users] Darshan & IPM results not the same<br>
<br>
Hello,<br>
<br>
I am trying to analyze the IO behavior of our codes with Darshan.<br>
Darshan: 3.3.0<br>
Compilers: Intel 2018<br>
MPI: Intel MPI 2018<br>
FS: Lustre (lustre-module disabled in Darshan configuration)<br>
Darshan profiling: LD_PRELOAD<br>
<br>
I observe a big difference in IO results from Darshan and IPM (v2.0.5) for one of our codes. I guess that both profilers are not profiling the same POSIX calls?<br>
<br>
The POSIXIO calls profiled in IPM are:<br>
<br>
fopen, fdopen, freopen, open, open64<br>
fclose, close<br>
fflush<br>
fread, read<br>
fwrite, write<br>
fseek, lseek, lseek64<br>
ftell<br>
rewind<br>
fgetpos, fsetpos, fgetc, getc, ungetc<br>
creat<br>
truncate, ftruncate, truncate64, ftruncate64<br>
<br>
While the ones profiled by Darshan are: <a href="https://github.com/darshan-hpc/darshan/blob/main/darshan-runtime/lib/darshan-posix.c">
https://github.com/darshan-hpc/darshan/blob/main/darshan-runtime/lib/darshan-posix.c</a> ?<br>
<br>
However, the huge difference is observed in the “read” call, which exists in both the profilers.<br>
<br>
+-------------------+------------+-----------+<br>
| | IPM | Darshan |<br>
+-------------------+------------+-----------+<br>
| Read (s) | 324.57 | 6.02 |<br>
+-------------------+------------+-----------+<br>
| Agg. Read (count) | 34 766 456 | 2 946 271 |<br>
+-------------------+------------+-----------+<br>
<br>
I tested Darshan and IPM with other codes (reading NOT the same files) to check if this issue is faced in them as well. But, I got the same results.<br>
So, I don't understand what could be the reason that this application is not giving the same results.<br>
<br>
Do you have any idea of why this could happen?<br>
<br>
Thanks in advance.<br>
<br>
PS: The application does a lot of IO, and is expected to spend a significant time in read operations.<br>
<br>
Best regards,<br>
Vineet<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>