<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<font size="3" class="">Hi Shane,</font>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><font size="3" class="">Thank you so much for your reply!</font></div>
<div class=""><font size="3" class="">I have tested your branch. Maybe there are still some problems. </font></div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><font size="3" class="">The file structure of my dataset is like the following, which is a typical ImageNet file structure. </font></div>
<div class=""><br class="">
</div>
<div class=""><font size="3" class="">train/<br class="">
|-- n01440764<br class="">
| |-- n01440764_10026.JPEG<br class="">
| |-- n01440764_10027.JPEG<br class="">
| |-- n01440764_10029.JPEG<br class="">
| |-- n01440764_10040.JPEG<br class="">
| |-- n01440764_10042.JPEG</font></div>
<div class=""><font size="3" class="">…</font></div>
<div class=""><font size="3" class="">val/</font></div>
<div class=""><font size="3" class="">…</font></div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><span style="font-size: medium;" class="">n01440764 means this is one of the 1000 classes of this dataset. The whole train folder has 1000 folders which means the dataset has 1000 classes of different classes representing different kinds of items
in images.</span><br class="">
</div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><span style="font-size: medium;" class="">I have two filesystems: one is local SSD with xfs on compute node, another is a Lustre filesystem. I do the tests on both of the two filesystems and the results show same results.</span></div>
<div class=""><font size="3" class="">Here are what I do to test Darshan and Python.</font></div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><font size="3" class="">I compile Darshan using 'snyder/dev-log-filters’ branch.</font></div>
<div class=""><font size="3" class="">The Test Case 1-4 are based on a simple image reading Python program. The program just walks through some folders and uses Python PIL library (which is most common used image reading library in PyTorch computer vision community)
to read the JPEG images into memory and converts JPEG images to RGB. </font></div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><font size="3" class="">Test Case 1:</font></div>
<div class=""><font size="3" class=""><b class="">WITHOUT</b> DARSHAN_CONF_PATH and DARSHAN_MODMEM=2048. </font></div>
<div class=""><font size="3" class="">I only read one folder like n01440764. There are 1300 JPEG images in this folder. The size of images ranges from 10K - 100K Bytes.</font></div>
<div class=""><font size="3" class="">The collected log shows that The POSIX module contains incomplete data.</font></div>
<div class=""><font size="3" class=""><br class="">
</font></div>
<div class=""><span style="font-size: medium;" class="">Test Case 2:</span></div>
<div class=""><span style="font-size: medium;" class=""><b class="">WITH</b> DARSHAN_CONF_PATH which set </span><font size="3" class="">MAX_RECORDS to a very big value like 1200000
</font><span style="font-size: medium;" class="">and DARSHAN_MODMEM=2048. </span></div>
<div class=""><span style="font-size: medium;" class="">The Python program and the folder I read are same with Test Case 1. No more </span><font size="3" class="">incomplete data. Use grep to check the log from darshan-parser shows the number of recorded files
are exactly what the Python program would read. Total Bytes Read is correct.</font></div>
<div class=""><font size="3" class="">I guess the </font><span style="font-size: medium;" class="">DARSHAN_CONF_PATH can take effect. Darshan log before parser is </span><font size="3" class="">141K. Darshan-parser generated log is 23M.</font></div>
<div class=""><span style="font-size: medium;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: medium;" class="">Test Case 3:</span></div>
<div class=""><span style="font-size: medium;" class="">WITH DARSHAN_CONF_PATH which set </span><font size="3" class="">MAX_RECORDS to a very big value like 1200000 </font><span style="font-size: medium;" class="">and DARSHAN_MODMEM=2048. </span></div>
<div class=""><span style="font-size: medium;" class="">I read more folders in Python program. In total the program would read
<b class="">2 folders and 2600 images</b>. 2 and more folders shows i</span><span style="font-size: medium;" class="">ncomplete data again. </span><span style="font-size: medium;" class="">Use grep to check the log from darshan-parser shows the number of recorded
files are 100 less than what the Python program would read.</span></div>
<div class=""><span style="font-size: medium;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: medium;" class="">Test Case 4:</span></div>
<div class=""><span style="font-size: medium;" class="">With DARSHAN_CONF_PATH which set </span><font size="3" class="">MAX_RECORDS to a very big value like 1200000 </font><span style="font-size: medium;" class="">and<b class=""> DARSHAN_MODMEM=4096</b>. </span></div>
<div class=""><font size="3" class="">Python program is same with Test Case 1. Now I get: darshan_library_warning: error compressing job record.</font></div>
<div class=""><font size="3" class="">darshan_library_warning: unable to write job record to file. The warning log probably relates with a previous problem I mention in this thread and maybe a zlib related problem(</font><a href="https://github.com/darshan-hpc/darshan/blob/e85b8bc929da91e54ff68fb1210dfe7bee3261a3/darshan-runtime/lib/darshan-core.c#L2039" class="">https://github.com/darshan-hpc/darshan/blob/e85b8bc929da91e54ff68fb1210dfe7bee3261a3/darshan-runtime/lib/darshan-core.c#L2039</a><span style="font-size: medium;" class="">).</span></div>
<div class=""><span style="font-size: medium;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: medium;" class="">Test Case 5:</span></div>
<div class=""><span style="font-size: medium;" class=""><b class="">WITH</b> DARSHAN_CONF_PATH which set </span><font size="3" class="">MAX_RECORDS to a very big value like
<b class="">3000000</b> </font><span style="font-size: medium;" class="">and DARSHAN_MODMEM=2048. </span></div>
<div class=""><span style="font-size: medium;" class="">I use a<b class=""> typical PyTorch ImageNet training program</b> which includes image reading, data processing and neural network training. The </span><span style="font-size: medium;" class="">darshan-parser
shows that Darshan could not get all the counters recorded correctly. Logs are not complete and total bytes read is 0.</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><font size="3" class="">So I guess the</font> <font size="3" class="">DARSHAN_CONF_PATH can take effect. But for larger number of files, Darshan’s POSIX module may encounter out of memory issue.</font></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">2021年7月3日 上午4:28，Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" class="">ssnyder@mcs.anl.gov</a>> 写道：</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Hi Lu,</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Sure, I can give you some details on how to use it. Most of the details are actually contained in this PR:<a href="https://github.com/darshan-hpc/darshan/pull/405" id="LPlnk470435" class="">https://github.com/darshan-hpc/darshan/pull/405</a><br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
So, to use, you would need to check out the branch that PR is based on (<span title="darshan-hpc/darshan:snyder/dev-log-filters" class="css-truncate user-select-contain expandable head-ref commit-ref"><a title="darshan-hpc/darshan:snyder/dev-log-filters" class=" no-underline" href="https://github.com/darshan-hpc/darshan/tree/snyder/dev-log-filters"><span class="css-truncate-target"></span></a></span>'snyder/dev-log-filters')
and build it just like you would normally build Darshan.<span class="Apple-converted-space"> </span><br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
The only additional trick is that you can specify a config file for Darshan to use at runtime via the DARSHAN_CONF_PATH environment variable (i.e., export DARSHAN_CONF_PATH=/path/to/my/darshan/config/file). You can add whatever lines you need to your config
file to control various aspects of Darshan's runtime behavior as outlined in the PR. Most relevant for you is probably just the ability to request that the POSIX module use more than the default 1,024 records, like this:</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
# request POSIX store 1.2 million file records rather than 1024 default<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
MAX_RECORDS 1200000 POSIX<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
You may also consider using NAME_EXCLUDE options to provide regular expressions of files to ignore that are not related to your ImageNet test case.</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
# e.g., ignore files in /home and files that end in .txt</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
NAME_EXCLUDE ^/home *</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
NAME_EXCLUDE .txt$ *<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
I have not tried to get Darshan to instrument such a massive single process workload, but will be interested to see if you have success. As I mentioned in my previous email, you'll probably want to bump DARSHAN_MODMEM up to around 2 GB to handle this, at the
very least.<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Another disclaimer I'll mention is that, since this stuff is experimental, some of these steps or naming conventions could change by the time we merge this into our main branch for eventual release. Not a big deal for now, but just a heads up.<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Thanks,</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
--Shane<br class="">
</div>
<div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
</div>
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 748.71875px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Lu Weizheng <<a href="mailto:luweizheng36@hotmail.com" class="">luweizheng36@hotmail.com</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, July 2, 2021 4:02 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" class="">ssnyder@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: Using darshan to instrument PyTorch</font>
<div class=""> </div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; line-break: after-white-space;">
Hi Shane,
<div class=""><br class="">
</div>
<div class="">Could you tell me more info about the experimental branch. Is it on github? I want to try it.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">2021年6月18日 下午11:11，Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" class="">ssnyder@mcs.anl.gov</a>> 写道：</div>
<br class="x_Apple-interchange-newline">
<div class=""><span class="" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline !important;">Those
changes are in an experimental branch right now while I fine tune the implementation, but if you're interested in trying it out I could give you some details.</span></div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>