<div dir="ltr">Aha! I just realized there is an obvious "prepare.sh" script that I didn't run, I found out by trial and error though, could be more documented :).<div><br></div><div>Now I'm further.. With a config file:</div><div>MAX_RECORDS 102400 POSIX,MPI-IO,STDIO<br></div><div>MODMEM 1024<br></div><div>APP_EXCLUDE git,ls<br></div><div><br></div><div>I'm getting for:</div><div>darshan-parser --show-incomplete caucau_python_id127447-127447_4-27-48556-1842455298968263838_1.darshan |grep incomplete<br></div><div><br></div><div>output:</div><div># *WARNING*: The POSIX module contains incomplete data!<br># *WARNING*: The STDIO module contains incomplete data!<br>Warning: no log utility handlers defined for module (null), SKIPPING.<br></div><div><br></div><div>I don't think I have more than 100000 files to be touched by my poor tiny python script, right?</div><div><br></div><div>By the way I've encountered another problem, not sure whether to put it to another thread. If I compile with HDF5 (the results above are without it):</div><div>./configure --with-log-path=/gpfs/raid/darshan-logs --with-jobid-env=PBS_JOBID CC=mpicc --enable-hdf5-mod --with-hdf5=/gpfs/raid/SDSSCube/ext_lib//hdf5-1.12.0/hdf5/<br></div><div><br></div><div>It messes up my runtime and causes python to crash:</div><div>mpirun -x DARSHAN_CONFIG_PATH=/gpfs/raid/SDSSCube/darshan.conf -x LD_PRELOAD=/gpfs/raid/shared_libs/darshan/darshan-runtime/lib/.libs/libdarshan.so:/gpfs/raid/SDSSCube/ext_lib/hdf5-1.12.0/hdf5/lib/libhdf5.so -np 65 --hostfile hosts --map-by node /gpfs/raid/SDSSCube/venv_par/bin/python hisscube.py --truncate ../sdss_data/ results/SDSS_cube_c_par.h5<br></div><div><br></div><div>Resulting in:</div><div>INFO:rank[0]:Rank 0 pid: 137058<br>Darshan HDF5 module error: runtime library version (1.12) incompatible with Darshan module (1.10-).<br>Traceback (most recent call last):<br> File "hisscube.py", line 74, in <module><br> writer.ingest(fits_image_path, fits_spectra_path, truncate_file=args.truncate)<br> File "/gpfs/raid/SDSSCube/hisscube/ParallelWriterMWMR.py", line 45, in ingest<br> self.process_metadata(image_path, image_pattern, spectra_path, spectra_pattern, truncate_file)<br> File "/gpfs/raid/SDSSCube/hisscube/CWriter.py", line 150, in process_metadata<br> h5_file = self.open_h5_file_serial(truncate_file)<br> File "/gpfs/raid/SDSSCube/hisscube/CWriter.py", line 170, in open_h5_file_serial<br> return h5py.File(self.h5_path, 'w', fs_strategy="page", fs_page_size=4096, libver="latest")<br> File "/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py", line 533, in __init__<br> fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)<br> File "/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py", line 232, in make_fid<br> fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)<br> File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper<br> File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper<br> File "h5py/h5f.pyx", line 126, in h5py.h5f.create<br> File "h5py/defs.pyx", line 693, in h5py.defs.H5Fcreate<br>RuntimeError: Unspecified error in H5Fcreate (return value <0)<br></div><div><br></div><div>You are saying that darshan should be compatible with HDF5 > 1.8, which 1.12 should be, right?</div><div><br></div><div>Thanks for help!</div><div><br></div><div>Cheers,</div><div><br></div><div>Jiri</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">st 27. 4. 2022 v 8:43 odesílatel Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com">nadvornik.ji@gmail.com</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I think I will chew through the documentation just fine but two things are not clear:</div><div><ol><li>Does the darshan library provide its own config file that I need to change or do I need to always create my own?</li><li>How can I build the git version? I didn't find any instructions and the usual autoconf just throws:</li><ol><li>root@kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime# autoconf<br><a href="http://configure.ac:19" target="_blank">configure.ac:19</a>: error: possibly undefined macro: AC_CONFIG_MACRO_DIRS<br> If this token and others are legitimate, please use m4_pattern_allow.<br> See the Autoconf documentation.<br><a href="http://configure.ac:21" target="_blank">configure.ac:21</a>: error: possibly undefined macro: AM_INIT_AUTOMAKE<br><a href="http://configure.ac:22" target="_blank">configure.ac:22</a>: error: possibly undefined macro: AM_SILENT_RULES<br><a href="http://configure.ac:23" target="_blank">configure.ac:23</a>: error: possibly undefined macro: AM_MAINTAINER_MODE<br><a href="http://configure.ac:713" target="_blank">configure.ac:713</a>: error: possibly undefined macro: AM_CONDITIONAL<br>root@kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime# ./configure<br>configure: error: cannot find install-sh, install.sh, or shtool in ../maint/scripts "."/../maint/scripts<br></li></ol></ol><div>Thanks for help.</div></div><div><br></div><div>Cheers,</div><div><br></div><div>Jiri</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">út 26. 4. 2022 v 17:43 odesílatel Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Jiri,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
For some background, Darshan enforces some internal memory limits to avoid ballooning memory usage at runtime. Specifically, all of our instrumentation modules should pre-allocate file records for up to 1,024 files opened by the app -- if your app opens more
than 1,024 files per-process, Darshan stops instrumenting and issues those warning messages when parsing the log file.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We have users hit this issue pretty frequently now, and we actually just wrapped up development of some new mechanisms to help out with this. They were just merged into our main branch, and we will be formally releasing a pre-release version of this code in
the next week or so. For the time being, you should be able to use the 'main' branch of our repo (<a href="https://github.com/darshan-hpc/darshan" id="gmail-m_8980332545816639580gmail-m_8205458234077072596LPNoLPOWALinkPreview" target="_blank">https://github.com/darshan-hpc/darshan</a>) to leverage this new functionality.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
There are 2 new mechanisms that can help out, both of which require you to provide a configuration file to Darshan at runtime:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<ul style="list-style-position:inside">
<li>MAX_RECORDS setting can be used to bump up the number of pre-allocated records for different modules. In your case, you might try to bump up the default number of records for the POSIX, MPI-IO, and STDIO modules by setting something like this in your config
file (this would allow you to instrument up to 4000 files per-process for each of these modules):</li><ul>
<li>MAX_RECORDS 4000 POSIX,MPI-IO,STDIO</li></ul>
<li>An alternative (or complementary) approach to bumping up the record limit is to limit instrumentation to particular files. You can use the NAME_EXCLUDE setting to avoid instrumenting specific directory paths, file extensions, etc by specifying regular expressions.
E.g, the following settings would avoid instrumenting files with .so prefixes or files located in a directory we don't care about for all modules (* denotes all modules):</li><ul>
<li>NAME_EXCLUDE .so$ *</li><li>NAME_EXCLUDE ^/path/to/avoid *</li></ul>
</ul>
<div>I'm attaching the updated runtime documentation for Darshan for your reference. Section 8 provides a ton of details on how to provide a config file to Darshan that should help clear up any missing gaps in my description above.</div>
<div><br>
</div>
<div>Please let us know if you have any further questions or issues, though!</div>
<div><br>
</div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
--Shane<br>
</div>
<div id="gmail-m_8980332545816639580gmail-m_8205458234077072596appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8980332545816639580gmail-m_8205458234077072596divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>><br>
<b>Sent:</b> Sunday, April 24, 2022 3:00 PM<br>
<b>To:</b> <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a> <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>><br>
<b>Subject:</b> [Darshan-users] Module contains incomplete data</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi All,
<div><br>
</div>
<div>I just tried out Darshan and the potential output seems perfect for my HDF5 MPI application! Although I'm not able to get there :(.</div>
<div><br>
</div>
<div>I have a log that has a big stamp "This darshan log contains incomplete data".</div>
<div><br>
</div>
<div>When I run:</div>
<div>darshan-parser --show-incomplete mylog.darshan |grep incomplete<br>
</div>
<div>Output is:</div>
<div># *WARNING*: The POSIX module contains incomplete data!<br>
# *WARNING*: The MPI-IO module contains incomplete data!<br>
# *WARNING*: The STDIO module contains incomplete data!<br>
</div>
<div><br>
</div>
<div>Would you be able to point me to some setting that would improve the measurements? Can I actually rely on the profiling results if it says the data is incomplete in some of the categories?</div>
<div><br>
</div>
<div>Thank you very much for your help!</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Jiri</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>