<div dir="ltr"><div>Good morning,</div><div><br></div><div>I'm post-processing a darshan file for a Tensorflow training of a simple model (CIFAR-10). The post-processing completes just fine, but I see an error on the first page:</div><div><br></div><div><br></div><div><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">WARNING</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">:</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">This</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">Darshan</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">log</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">contains</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">incomplete</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">data.</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">This</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">happens</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">when</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">a</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">module</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">runs</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">out</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">of</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">memory</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">to</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">store</span><br role="presentation"><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">new</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">record</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">data.</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">Please</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">run</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">darshan-parser</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">on</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">the</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">log</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">file</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">for</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">more</span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr"> </span><span style="font-size:18.1818px;font-family:sans-serif" role="presentation" dir="ltr">information.</span></div><div><br></div><div>So I ran darshan-parser on the file and I see the following at the end.</div><div><br></div><div><br></div><div># *******************************************************<br># POSIX module data<br># *******************************************************<br><br># *ERROR*: The POSIX module contains incomplete data!<br>#            This happens when a module runs out of<br>#            memory to store new record data.<br><br># To avoid this error, consult the darshan-runtime<br># documentation and consider setting the<br># DARSHAN_EXCLUDE_DIRS environment variable to prevent<br># Darshan from instrumenting unecessary files.<br><br># You can display the (incomplete) data that is<br># present in this log using the --show-incomplete<br># option to darshan-parser.</div><div><br></div><div><br></div><div>I have a bunch of file systems excluded: /proc,/etc,/dev,/sys,/snap,/run . <br></div><div><br></div><div>How can I get a list of files that Darshan tracked? Is there a way to increase the amount of memory?</div><div><br></div><div>Thanks!</div><div><br></div><div>Jeff</div><div><br></div><div><br></div><div><br></div></div>