<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span class="" style="font-size: 15px;">Hi Shane,</span>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;">Thank you so much for you reply about my problem.</span></div>
<div class=""><span class="" style="font-size: 15px;">After some experiments on darshan, I guess that the main reason for my incorrect darshan logs is probably about Python multiprocessing module. Here are what I tried:</span></div>
<div class=""><span class="" style="font-size: 15px;">I add some excluding dirs for darshan like this:</span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><font face="Menlo" class="" style="font-size: 15px;">export DARSHAN_EXCLUDE_DIRS=/proc,/etc,/dev,/sys,~/jupyterlab/dlprof/,~/.conda/envs/torch1.8/</font></div>
<div class=""><font face="Menlo" class="" style="font-size: 15px;"><br class="">
</font></div>
<div class=""><span class="" style="font-size: 15px;">After excluding the python dir, I do not get multiple logs anymore. There are only 3 darshan logs about python IO behavior, each about a python process. As my program’s main process start 2 sub-processes.
However the results in the logs are not correct. The total transferred file size is 0 as showed in the PDF genernated in darshan-job-summary.pl tool: <i class="">I/O performance estimate (at the POSIX layer): transferred 0.0 MiB at 10.50 MiB/s</i>. And using darshan-parser
to get the detailed info about the darshan log, there is no POSIX or STDIO counters on the files which my workload would read and write. So I guess darshan could not get the IO that my workload is doing.</span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;">I check the source code of PyTorch which I am instrumenting on. The data loading part relies on Python’s multiprocessing module. PyTorch use multiprocessing to start a pool and read data using multiple cores.
I also find a GitHub issue about darshan on Python multiprocessing (<a href="https://github.com/darshan-hpc/darshan/issues/293" class="">https://github.com/darshan-hpc/darshan/issues/293</a>). But in the issue there are no solution and I guess the problem
still exists. </span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;">So is there any suggestions about how to instrument PyTorch workloads? Or is there any workaround about multiprocessing?</span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;">Best Regards,</span></div>
<div class=""><span class="" style="font-size: 15px;"><br class="">
</span></div>
<div class=""><span class="" style="font-size: 15px;">Lu</span></div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">2021年6月18日 上午5:26，Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" class="">ssnyder@mcs.anl.gov</a>> 写道：</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">Hi Lu,</span>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
(sending to the entire mailing list now)<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
Unfortunately, we don't currently have a tool for either combining multiple logs from a workflow into a single log file or analysis tools that work on sets of logs.<span class="Apple-converted-space"> </span><br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
We do have a utility called 'darshan-merge' that was written to help merge together Darshan logs for another use case, but I don't think it will work right for this case from some quick testing. I've opened an issue on our GitHub page (<a href="https://github.com/darshan-hpc/darshan/issues/401" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-linkindex="0" class="">https://github.com/darshan-hpc/darshan/issues/401</a>)
to remind myself to see if I can rework this tool to be more helpful in cases like yours.</div>
<div class=""></div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
At some point, we'd like to offer some of our own analysis tools that are workflow aware and can summarize data from multiple Darshan logs. That's something that's going to take some time though, as we are just now starting to look at revamping some of our
analysis tools using the new PyDarshan interface to Darshan logs. BTW, PyDarshan might be something you could consider using if you wanted to come up with your own analysis tools for Darshan data, but that might be more work than you're looking for. In case
it's helpful, here's some documentation on PyDarshan:<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/research/projects/darshan/docs/pydarshan/index.html" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-linkindex="1" class="">https://www.mcs.anl.gov/research/projects/darshan/docs/pydarshan/index.html</a><br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
<br class="">
</div>
<div style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">
Thanks,</div>
<span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class="">--Shane</span><br class="">
</div>
<div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 748.71875px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" class="">darshan-users-bounces@lists.mcs.anl.gov</a>>
on behalf of Lu Weizheng <<a href="mailto:luweizheng36@hotmail.com" class="">luweizheng36@hotmail.com</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, June 15, 2021 3:43 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:darshan-users@lists.mcs.anl.gov" class="">darshan-users@lists.mcs.anl.gov</a> <<a href="mailto:darshan-users@lists.mcs.anl.gov" class="">darshan-users@lists.mcs.anl.gov</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[Darshan-users] Using darshan to instrument PyTorch</font>
<div class=""> </div>
</div>
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
I am using darshan to instrument PyTorch on a local machine. My workload is an image classification problem on ImageNet dataset. When the training process ended, there are a lot of logs generated. Like:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
u2020000_python_id4719_6-15-41351-17690910011763757569_1.darshan </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
u2020000_python_id5012_6-15-42860-17690910011763757569_1.darshan
<div class="">u2020000_python_id4721_6-15-41352-17690910011763757569_1.darshan </div>
<div class="">u2020000_uname_id4720_6-15-41351-17690910011763757569_1.darshan</div>
<div class="">u2020000_python_id4722_6-15-41352-17690910011763757569_1.darshan </div>
<div class="">u2020000_uname_id4723_6-15-41354-17690910011763757569_1.darshan</div>
u2020000_python_id4758_6-15-41830-17690910011763757569_1.darshan </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
u2020000_uname_id4724_6-15-41354-17690910011763757569_1.darshan</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
...</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
After using the darshan-util analysis tool for one of the above log file, it shows: I/O performance estimate (at the POSIX layer): transferred 7.5 MiB at 36.02 MiB/s</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
The transferred data showed in the PDF report is far less than the whole dataset size.<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">As PyTorch DataLoader is a multi-process program, I guess darshan generate every
log for every process. </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
My question is: how can I get the IO analysis for the whole PyTorch workload task instead of these process logs?</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>