<div dir="ltr">Hi Phil and others,<div><br></div><div>in the end the problem was somewhere else and much more prosaic :/. The path where darshan was installed by default (lib folder specifically) was missing from the <span style="font-family:Calibri;font-size:11pt">LD_LIBRARY_PATH environment variable and therefore the library could not be found at runtime.</span></div><div><span style="font-family:Calibri;font-size:11pt"><br></span></div><div><span style="font-family:Calibri;font-size:11pt">BR,</span></div><div><span style="font-family:Calibri;font-size:11pt"><br></span></div><div><span style="font-family:Calibri;font-size:11pt">Jiri</span></div><div><span style="font-family:Calibri;font-size:11pt"></span><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">pá 6. 5. 2022 v 15:56 odesílatel Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com">nadvornik.ji@gmail.com</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Aha,<div><br></div><div>maybe I formulated it wrongly. The problem is that the version that is installed by make install is taken from the .lib folder and not from the darshan-util one. So when trying to run for example darshan-parser, it complains about ".libs/libdarshan-util.so.0" missing.</div><div><br></div><div>Cheers,</div><div><br></div><div>Jiri</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">st 4. 5. 2022 v 22:41 odesílatel Phil Carns <<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

  
  <div>
    <p>Hi Jiri,</p>
    <p>For #2, are you talking about .libs/darshan-parser in the build
      path?  If so, I wouldn't expect  to be able to run that directly
      (in general, not particular to Darshan's build system).</p>
    <p>Autotools and libtool create executables in .libs subdirs as part
      of the build process, but those are intermediate executables that
      don't have final library paths set.  The copy installed in
      <prefix>/bin should be fine (and should use corresponding
      installed libraries), and you should also be able to run
      darshan-parser one level up in the build tree (that's actually a
      shell script wrapper created by libtool that will run the
      .libs/darshan-parser with library paths set to the build tree).</p>
    <p>Sometimes .libs/ executables might work, but it's a little dicey
      what libraries they will pick up so it's usually not a good idea.<br>
    </p>
    <p>thanks,</p>
    <p>-Phil<br>
    </p>
    <div>On 4/27/22 1:34 PM, Jiří Nádvorník
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Hi,
        <div><br>
        </div>
        <div>to reproduce the installation issue:</div>
        <div>mkdir darshan_root</div>
        <div>cd darshan_root</div>
        <div>git clone <a href="https://github.com/darshan-hpc/darshan.git" target="_blank">https://github.com/darshan-hpc/darshan.git</a>
          .<br>
          <div><br>
          </div>
          <div>Then cd darshan-utils/ and then run:</div>
          <div>autoconf</div>
          <div>configure</div>
          <div>make install</div>
          <div><br>
          </div>
          <div>Then:</div>
          <div>
            <ol>
              <li>If running darshan-parser within the same folder it
                runs fine.</li>
              <li>If running .lib/darshan-parser (which is installed by
                make install) it crashes with the library not available,
                see the previous email.</li>
            </ol>
          </div>
          <div>Cheers,</div>
          <div><br>
            Jiri</div>
          <div><br>
          </div>
          <div><br>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">st 27. 4. 2022 v 18:23
          odesílatel Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>>
          napsal:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Great,
              I'm glad that you were able to get the instrumentation
              mostly working!</div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I
              think it's sensible to ignore Python source/compiled code
              for most cases -- I doubt there's any insight to gain and
              you'll just end up trying to filter them out when
              analyzing logs anyways.</div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I'm
              not sure what's going on with the installation issues you
              mention. If you think something might be wrong with
              Darshan's build, then would you mind sharing how you ran
              configure, etc.? I could see if I'm able to reproduce
              anything.
              <br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">If
              you wouldn't mind starting a new thread related to the
              HDF5 issue, I think that would be helpful -- it might help
              if other users ever want to search the list archive for
              h5py/HDF5 related issues if you include those in the
              title.<br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
            </div>
            <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">--Shane<br>
            </div>
            <hr style="display:inline-block;width:98%">
            <div id="gmail-m_-9014243290627863420gmail-m_6519552878307446660gmail-m_-2273691217974330162divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri,
                sans-serif" color="#000000"><b>From:</b> Jiří Nádvorník
                <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>><br>
                <b>Sent:</b> Wednesday, April 27, 2022 11:06 AM<br>
                <b>To:</b> Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>><br>
                <b>Cc:</b> <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>
                <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>><br>
                <b>Subject:</b> Re: [Darshan-users] Module contains
                incomplete data</font>
              <div> </div>
            </div>
            <div>
              <div dir="ltr">
                <div>Hi,</div>
                <div dir="ltr"><br>
                </div>
                <div dir="ltr">yes, that NAMEMEM got it done. I also
                  excluded .py and .pyc files - those reads are only
                  loading of them, right? No data access itself (and no,
                  I'm not reading and manually interpreting my own
                  python files :), so I'm not interested into that ).
                  Actually, I'm reading thousands of small files which
                  I'm ingesting into HDF5 and I'm interested into how
                  many reads, etc. are happening. </div>
                <div dir="ltr"><br>
                </div>
                <div dir="ltr">I'm trying to make some sense of what I
                  see but for now I'm just going to say it's very
                  valuable data for me. Pity I can't get the Hdf5
                  module, if it would give me more granularity it would
                  be very helpful.
                  <div><br>
                  </div>
                  <div>Regarding the darshan-utils you were right, I
                    didn't reinstall them. I actually ran into an
                    install problem - for some reason, the git
                    installation takes the .lib/darshan-parser when
                    installing it to /usr/local/... and that one throws:</div>
                  <div>darshan-parser: error while loading shared
                    libraries: libdarshan-util.so.0: cannot open shared
                    object file: No such file or directory</div>
                  <div><br>
                  </div>
                  <div>But if I run the darshan-parser within
                    darshan_root_folder/darshan-util/ then the error is
                    gone and --show-incomplete |grep incomplete prints
                    nothing.</div>
                  <div><br>
                  </div>
                  <div>Could we now focus on the HDF5 issue or should I
                    create a new thread for clarity?</div>
                  <div><br>
                  </div>
                  <div>Cheers,</div>
                  <div><br>
                  </div>
                  <div>Jiri</div>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                  </div>
                </div>
                <br>
                <div>
                  <div dir="ltr">st 27. 4. 2022 v 17:21 odesílatel
                    Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>>
                    napsal:<br>
                  </div>
                  <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div dir="ltr">
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Thanks
                        for working through the build issues and giving
                        this a shot.</div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
                      </div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">A
                        couple of things stand out to me (ignoring your
                        HDF5 issue for now):</div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                        <ul>
                          <li>It looks like at least the MPI-IO module
                            is no longer reporting partial data? Small
                            progress...</li>
                          <li>There is a new warning about there being
                            no log utility handlers for a "null" module.
                            Are you perhaps parsing a log generated by
                            your prior Darshan install? Maybe you have
                            not completely re-installed a new
                            darshan-util? We should figure out what's
                            going on there, too, to be safe.<br>
                          </li>
                        </ul>
                      </div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I'd
                        also suggest two things for your config file:</div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                        <ul>
                          <li>Dial back MODMEM and MAX_RECORDS values.
                            Your MODMEM value is asking Darshan to
                            allocate a GiB of memory (it is expressed in
                            MiB units and you set to 1024), which
                            Darshan will happily try to do, I'm not sure
                            it's a good idea though. I'd probably start
                            with a MODMEM value of 8 and MAX_RECORDS of
                            2000, and just double those again if needed
                            -- anything beyond that would be surprising
                            unless you know your workload is really
                            opening hundreds of thousands of files. You
                            might also have a look at the files Darshan
                            is currently instrumenting and see if you
                            really want it to -- I've noticed when
                            instrumenting Python frameworks that you can
                            get tons of records for things like shared
                            libraries, source files, etc. that can just
                            be ignored using NAME_EXCLUDE mechanisms.
                            <br>
                          </li>
                          <li>Add "NAMEMEM  2" to your config file to
                            force Darshan to allocate more memory (2
                            MiB) for storing the filenames associated
                            with each record. This might actually be the
                            main reason your log is reporting partial
                            data rather than actually running out of
                            module data, which is another reason not to
                            get too aggressive with the
                            MODMEM/MAX_RECORDS parameters. I should have
                            mentioned this setting originally as there
                            have been other users who have reported
                            exceeding it recently.<br>
                          </li>
                        </ul>
                        <div>Hopefully that gets you further along and
                          we can move onto the HDF5 issue you mention.<br>
                        </div>
                        <div><br>
                        </div>
                      </div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Thanks,<br>
                      </div>
                      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">--Shane<br>
                      </div>
                      <hr style="display:inline-block;width:98%">
                      <div id="gmail-m_-9014243290627863420gmail-m_6519552878307446660gmail-m_-2273691217974330162x_gmail-m_6650929377004058598divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
                          Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>><br>
                          <b>Sent:</b> Wednesday, April 27, 2022 6:37 AM<br>
                          <b>To:</b> Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>><br>
                          <b>Cc:</b> <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>
                          <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>><br>
                          <b>Subject:</b> Re: [Darshan-users] Module
                          contains incomplete data</font>
                        <div> </div>
                      </div>
                      <div>
                        <div dir="ltr">Aha! I just realized there is an
                          obvious "prepare.sh" script that I didn't run,
                          I found out by trial and error though, could
                          be more documented :).
                          <div><br>
                          </div>
                          <div>Now I'm further.. With a config file:</div>
                          <div>MAX_RECORDS     102400    
                            POSIX,MPI-IO,STDIO<br>
                          </div>
                          <div>MODMEM  1024<br>
                          </div>
                          <div>APP_EXCLUDE     git,ls<br>
                          </div>
                          <div><br>
                          </div>
                          <div>I'm getting for:</div>
                          <div>darshan-parser --show-incomplete
                             caucau_python_id127447-127447_4-27-48556-1842455298968263838_1.darshan
                            |grep incomplete<br>
                          </div>
                          <div><br>
                          </div>
                          <div>output:</div>
                          <div># *WARNING*: The POSIX module contains
                            incomplete data!<br>
                            # *WARNING*: The STDIO module contains
                            incomplete data!<br>
                            Warning: no log utility handlers defined for
                            module (null), SKIPPING.<br>
                          </div>
                          <div><br>
                          </div>
                          <div>I don't think I have more than 100000
                            files to be touched by my poor tiny python
                            script, right?</div>
                          <div><br>
                          </div>
                          <div>By the way I've encountered another
                            problem, not sure whether to put it to
                            another thread. If I compile with HDF5 (the
                            results above are without it):</div>
                          <div>./configure
                            --with-log-path=/gpfs/raid/darshan-logs
                            --with-jobid-env=PBS_JOBID CC=mpicc
                            --enable-hdf5-mod
                            --with-hdf5=/gpfs/raid/SDSSCube/ext_lib//hdf5-1.12.0/hdf5/<br>
                          </div>
                          <div><br>
                          </div>
                          <div>It messes up my runtime and causes python
                            to crash:</div>
                          <div>mpirun -x
                            DARSHAN_CONFIG_PATH=/gpfs/raid/SDSSCube/darshan.conf
                            -x
LD_PRELOAD=/gpfs/raid/shared_libs/darshan/darshan-runtime/lib/.libs/libdarshan.so:/gpfs/raid/SDSSCube/ext_lib/hdf5-1.12.0/hdf5/lib/libhdf5.so
                            -np 65 --hostfile hosts --map-by node
                            /gpfs/raid/SDSSCube/venv_par/bin/python
                            hisscube.py --truncate ../sdss_data/
                            results/SDSS_cube_c_par.h5<br>
                          </div>
                          <div><br>
                          </div>
                          <div>Resulting in:</div>
                          <div><a>INFO:rank</a>[0]:Rank 0 pid: 137058<br>
                            Darshan HDF5 module error: runtime library
                            version (1.12) incompatible with Darshan
                            module (1.10-).<br>
                            Traceback (most recent call last):<br>
                              File "hisscube.py", line 74, in
                            <module><br>
                                writer.ingest(fits_image_path,
                            fits_spectra_path,
                            truncate_file=args.truncate)<br>
                              File
                            "/gpfs/raid/SDSSCube/hisscube/ParallelWriterMWMR.py",
                            line 45, in ingest<br>
                                self.process_metadata(image_path,
                            image_pattern, spectra_path,
                            spectra_pattern, truncate_file)<br>
                              File
                            "/gpfs/raid/SDSSCube/hisscube/CWriter.py",
                            line 150, in process_metadata<br>
                                h5_file =
                            self.open_h5_file_serial(truncate_file)<br>
                              File
                            "/gpfs/raid/SDSSCube/hisscube/CWriter.py",
                            line 170, in open_h5_file_serial<br>
                                return h5py.File(self.h5_path, 'w',
                            fs_strategy="page", fs_page_size=4096,
                            libver="latest")<br>
                              File
"/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py",
                            line 533, in __init__<br>
                                fid = make_fid(name, mode,
                            userblock_size, fapl, fcpl, swmr=swmr)<br>
                              File
"/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py",
                            line 232, in make_fid<br>
                                fid = h5f.create(name, h5f.ACC_TRUNC,
                            fapl=fapl, fcpl=fcpl)<br>
                              File "h5py/_objects.pyx", line 54, in
                            h5py._objects.with_phil.wrapper<br>
                              File "h5py/_objects.pyx", line 55, in
                            h5py._objects.with_phil.wrapper<br>
                              File "h5py/h5f.pyx", line 126, in
                            h5py.h5f.create<br>
                              File "h5py/defs.pyx", line 693, in
                            h5py.defs.H5Fcreate<br>
                            RuntimeError: Unspecified error in H5Fcreate
                            (return value <0)<br>
                          </div>
                          <div><br>
                          </div>
                          <div>You are saying that darshan should be
                            compatible with HDF5 > 1.8, which 1.12
                            should be, right?</div>
                          <div><br>
                          </div>
                          <div>Thanks for help!</div>
                          <div><br>
                          </div>
                          <div>Cheers,</div>
                          <div><br>
                          </div>
                          <div>Jiri</div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                          <div><br>
                          </div>
                        </div>
                        <br>
                        <div>
                          <div dir="ltr">st 27. 4. 2022 v 8:43
                            odesílatel Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>>
                            napsal:<br>
                          </div>
                          <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                            <div dir="ltr">Hi,
                              <div><br>
                              </div>
                              <div>I think I will chew through the
                                documentation just fine but two things
                                are not clear:</div>
                              <div>
                                <ol>
                                  <li>Does the darshan library provide
                                    its own config file that I need to
                                    change or do I need to always create
                                    my own?</li>
                                  <li>How can I build the git version? I
                                    didn't find any instructions and the
                                    usual autoconf just throws:</li>
                                  <ol>
                                    <li>root@kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime#
                                      autoconf<br>
                                      <a href="http://configure.ac:19" target="_blank">configure.ac:19</a>:
                                      error: possibly undefined macro:
                                      AC_CONFIG_MACRO_DIRS<br>
                                            If this token and others are
                                      legitimate, please use
                                      m4_pattern_allow.<br>
                                            See the Autoconf
                                      documentation.<br>
                                      <a href="http://configure.ac:21" target="_blank">configure.ac:21</a>:
                                      error: possibly undefined macro:
                                      AM_INIT_AUTOMAKE<br>
                                      <a href="http://configure.ac:22" target="_blank">configure.ac:22</a>:
                                      error: possibly undefined macro:
                                      AM_SILENT_RULES<br>
                                      <a href="http://configure.ac:23" target="_blank">configure.ac:23</a>:
                                      error: possibly undefined macro:
                                      AM_MAINTAINER_MODE<br>
                                      <a href="http://configure.ac:713" target="_blank">configure.ac:713</a>:
                                      error: possibly undefined macro:
                                      AM_CONDITIONAL<br>
root@kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime# ./configure<br>
                                      configure: error: cannot find
                                      install-sh, install.sh, or shtool
                                      in ../maint/scripts
                                      "."/../maint/scripts<br>
                                    </li>
                                  </ol>
                                </ol>
                                <div>Thanks for help.</div>
                              </div>
                              <div><br>
                              </div>
                              <div>Cheers,</div>
                              <div><br>
                              </div>
                              <div>Jiri</div>
                            </div>
                            <br>
                            <div>
                              <div dir="ltr">út 26. 4. 2022 v 17:43
                                odesílatel Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>>
                                napsal:<br>
                              </div>
                              <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div dir="ltr">
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Hi
                                    Jiri,</div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
                                  </div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">For
                                    some background, Darshan enforces
                                    some internal memory limits to avoid
                                    ballooning memory usage at runtime.
                                    Specifically, all of our
                                    instrumentation modules should
                                    pre-allocate file records for up to
                                    1,024 files opened by the app -- if
                                    your app opens more than 1,024 files
                                    per-process, Darshan stops
                                    instrumenting and issues those
                                    warning messages when parsing the
                                    log file.</div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
                                  </div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">We
                                    have users hit this issue pretty
                                    frequently now, and we actually just
                                    wrapped up development of some new
                                    mechanisms to help out with this.
                                    They were just merged into our main
                                    branch, and we will be formally
                                    releasing a pre-release version of
                                    this code in the next week or so.
                                    For the time being, you should be
                                    able to use the 'main' branch of our
                                    repo (<a href="https://github.com/darshan-hpc/darshan" id="gmail-m_-9014243290627863420gmail-m_6519552878307446660gmail-m_-2273691217974330162x_gmail-m_6650929377004058598x_gmail-m_8980332545816639580gmail-m_8205458234077072596LPNoLPOWALinkPreview" target="_blank">https://github.com/darshan-hpc/darshan</a>)
                                    to leverage this new functionality.</div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
                                  </div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">There
                                    are 2 new mechanisms that can help
                                    out, both of which require you to
                                    provide a configuration file to
                                    Darshan at runtime:</div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                                    <ul style="list-style-position:inside">
                                      <li>MAX_RECORDS setting can be
                                        used to bump up the number of
                                        pre-allocated records for
                                        different modules. In your case,
                                        you might try to bump up the
                                        default number of records for
                                        the POSIX, MPI-IO, and STDIO
                                        modules  by setting something
                                        like this in your config file
                                        (this would allow you to
                                        instrument up to 4000 files
                                        per-process for each of these
                                        modules):</li>
                                      <ul>
                                        <li>MAX_RECORDS    4000   
                                          POSIX,MPI-IO,STDIO</li>
                                      </ul>
                                      <li>An alternative (or
                                        complementary) approach to
                                        bumping up the record limit is
                                        to limit instrumentation to
                                        particular files. You can use
                                        the NAME_EXCLUDE setting to
                                        avoid instrumenting specific
                                        directory paths, file
                                        extensions, etc by specifying
                                        regular expressions. E.g, the
                                        following settings would avoid
                                        instrumenting files with .so
                                        prefixes or files located in a
                                        directory we don't care about
                                        for all modules (* denotes all
                                        modules):</li>
                                      <ul>
                                        <li>NAME_EXCLUDE    .so$    *</li>
                                        <li>NAME_EXCLUDE   
                                          ^/path/to/avoid    *</li>
                                      </ul>
                                    </ul>
                                    <div>I'm attaching the updated
                                      runtime documentation for Darshan
                                      for your reference. Section 8
                                      provides a ton of details on how
                                      to provide a config file to
                                      Darshan that should help clear up
                                      any missing gaps in my description
                                      above.</div>
                                    <div><br>
                                    </div>
                                    <div>Please let us know if you have
                                      any further questions or issues,
                                      though!</div>
                                    <div><br>
                                    </div>
                                  </div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Thanks,</div>
                                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">--Shane<br>
                                  </div>
                                  <div id="gmail-m_-9014243290627863420gmail-m_6519552878307446660gmail-m_-2273691217974330162x_gmail-m_6650929377004058598x_gmail-m_8980332545816639580gmail-m_8205458234077072596appendonsend"></div>
                                  <hr style="display:inline-block;width:98%">
                                  <div id="gmail-m_-9014243290627863420gmail-m_6519552878307446660gmail-m_-2273691217974330162x_gmail-m_6650929377004058598x_gmail-m_8980332545816639580gmail-m_8205458234077072596divRplyFwdMsg" dir="ltr">
                                    <font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
                                      Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>>
                                      on behalf of Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>><br>
                                      <b>Sent:</b> Sunday, April 24,
                                      2022 3:00 PM<br>
                                      <b>To:</b> <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>
                                      <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>><br>
                                      <b>Subject:</b> [Darshan-users]
                                      Module contains incomplete data</font>
                                    <div> </div>
                                  </div>
                                  <div>
                                    <div dir="ltr">Hi All,
                                      <div><br>
                                      </div>
                                      <div>I just tried out Darshan and
                                        the potential output seems
                                        perfect for my HDF5 MPI
                                        application! Although I'm not
                                        able to get there :(.</div>
                                      <div><br>
                                      </div>
                                      <div>I have a log that has a big
                                        stamp "This darshan log contains
                                        incomplete data".</div>
                                      <div><br>
                                      </div>
                                      <div>When I run:</div>
                                      <div>darshan-parser
                                        --show-incomplete  mylog.darshan
                                        |grep incomplete<br>
                                      </div>
                                      <div>Output is:</div>
                                      <div># *WARNING*: The POSIX module
                                        contains incomplete data!<br>
                                        # *WARNING*: The MPI-IO module
                                        contains incomplete data!<br>
                                        # *WARNING*: The STDIO module
                                        contains incomplete data!<br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>Would you be able to point me
                                        to some setting that would
                                        improve the measurements? Can I
                                        actually rely on the profiling
                                        results if it says the data is
                                        incomplete in some of the
                                        categories?</div>
                                      <div><br>
                                      </div>
                                      <div>Thank you very much for your
                                        help!</div>
                                      <div><br>
                                      </div>
                                      <div>Cheers,</div>
                                      <div><br>
                                      </div>
                                      <div>Jiri</div>
                                      <div><br>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
Darshan-users mailing list
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.gov</a>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a>
</pre>
    </blockquote>
  </div>

_______________________________________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><br>
</blockquote></div>
</blockquote></div>