<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Jeff,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Existing Darshan releases do have some hard coded limits that have been increasingly problematic for our users, it seems. The limit you are likely hitting is just that Darshan instrumentation modules do not track more than 1,024 file records currently. This
isn't really tunable in any way, unfortunately.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
You can get a list of files that Darshan did instrument by running darshan-parser with the '--file-list' option. That might give you some more ideas on directories you could potentially exclude to force Darshan to reserve instrumentation resources for other
files, but that may not even be sufficient depending on your workload.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We do have some functionality we are hoping to have merged in for our next release to help address this issue. In fact, it's available to try out in a branch in our repo if you're really motivated to get this working soon. There are more details here in a PR
on our GitHub: <a href="https://github.com/darshan-hpc/darshan/pull/405" id="LPlnk476838">
https://github.com/darshan-hpc/darshan/pull/405</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Essentially, you can use a config file to control a number of different Darshan settings, including the ability to change the hard coded file maximum from above and to provide regular expressions (rather than just directory names) for files Darshan should
exclude from instrumentation. If you have more specific questions or feedback about this functionality, please let us know.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Jeffrey Layton <laytonjb@gmail.com><br>
<b>Sent:</b> Monday, July 26, 2021 9:15 AM<br>
<b>To:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> [Darshan-users] Error in job_summary</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Good morning,</div>
<div><br>
</div>
<div>I'm post-processing a darshan file for a Tensorflow training of a simple model (CIFAR-10). The post-processing completes just fine, but I see an error on the first page:</div>
<div><br>
</div>
<div><br>
</div>
<div><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">WARNING</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">:</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">This</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">Darshan</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">log</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">contains</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">incomplete</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">data.</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">This</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">happens</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">when</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">a</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">module</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">runs</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">out</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">of</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">memory</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">to</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">store</span><br role="presentation">
<span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">new</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">record</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">data.</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">Please</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">run</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">darshan-parser</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">on</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">the</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">log</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">file</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">for</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">more</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">
</span><span role="presentation" dir="ltr" style="font-size:18.1818px; font-family:sans-serif">information.</span></div>
<div><br>
</div>
<div>So I ran darshan-parser on the file and I see the following at the end.</div>
<div><br>
</div>
<div><br>
</div>
<div># *******************************************************<br>
# POSIX module data<br>
# *******************************************************<br>
<br>
# *ERROR*: The POSIX module contains incomplete data!<br>
# This happens when a module runs out of<br>
# memory to store new record data.<br>
<br>
# To avoid this error, consult the darshan-runtime<br>
# documentation and consider setting the<br>
# DARSHAN_EXCLUDE_DIRS environment variable to prevent<br>
# Darshan from instrumenting unecessary files.<br>
<br>
# You can display the (incomplete) data that is<br>
# present in this log using the --show-incomplete<br>
# option to darshan-parser.</div>
<div><br>
</div>
<div><br>
</div>
<div>I have a bunch of file systems excluded: /proc,/etc,/dev,/sys,/snap,/run . <br>
</div>
<div><br>
</div>
<div>How can I get a list of files that Darshan tracked? Is there a way to increase the amount of memory?</div>
<div><br>
</div>
<div>Thanks!</div>
<div><br>
</div>
<div>Jeff</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</body>
</html>