[Darshan-users] Module contains incomplete data

[Jiří Nádvorník]

Hi,

I think I will chew through the documentation just fine but two things are
not clear:

   1. Does the darshan library provide its own config file that I need to
   change or do I need to always create my own?
   2. How can I build the git version? I didn't find any instructions and
   the usual autoconf just throws:
      1. root at kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime#
      autoconf
      configure.ac:19: error: possibly undefined macro: AC_CONFIG_MACRO_DIRS
            If this token and others are legitimate, please use
      m4_pattern_allow.
            See the Autoconf documentation.
      configure.ac:21: error: possibly undefined macro: AM_INIT_AUTOMAKE
      configure.ac:22: error: possibly undefined macro: AM_SILENT_RULES
      configure.ac:23: error: possibly undefined macro: AM_MAINTAINER_MODE
      configure.ac:713: error: possibly undefined macro: AM_CONDITIONAL
      root at kub-b1:/gpfs/raid/shared_libs/darshan/darshan-runtime#
      ./configure
      configure: error: cannot find install-sh, install.sh, or shtool in
      ../maint/scripts "."/../maint/scripts

Thanks for help.

Cheers,

Jiri

út 26. 4. 2022 v 17:43 odesílatel Snyder, Shane <ssnyder at mcs.anl.gov>
napsal:

> Hi Jiri,
>
> For some background, Darshan enforces some internal memory limits to avoid
> ballooning memory usage at runtime. Specifically, all of our
> instrumentation modules should pre-allocate file records for up to 1,024
> files opened by the app -- if your app opens more than 1,024 files
> per-process, Darshan stops instrumenting and issues those warning messages
> when parsing the log file.
>
> We have users hit this issue pretty frequently now, and we actually just
> wrapped up development of some new mechanisms to help out with this. They
> were just merged into our main branch, and we will be formally releasing a
> pre-release version of this code in the next week or so. For the time
> being, you should be able to use the 'main' branch of our repo (
> https://github.com/darshan-hpc/darshan) to leverage this new
> functionality.
>
> There are 2 new mechanisms that can help out, both of which require you to
> provide a configuration file to Darshan at runtime:
>
>    - MAX_RECORDS setting can be used to bump up the number of
>    pre-allocated records for different modules. In your case, you might try to
>    bump up the default number of records for the POSIX, MPI-IO, and STDIO
>    modules  by setting something like this in your config file (this would
>    allow you to instrument up to 4000 files per-process for each of these
>    modules):
>       - MAX_RECORDS    4000    POSIX,MPI-IO,STDIO
>    - An alternative (or complementary) approach to bumping up the record
>    limit is to limit instrumentation to particular files. You can use the
>    NAME_EXCLUDE setting to avoid instrumenting specific directory paths, file
>    extensions, etc by specifying regular expressions. E.g, the following
>    settings would avoid instrumenting files with .so prefixes or files located
>    in a directory we don't care about for all modules (* denotes all modules):
>       - NAME_EXCLUDE    .so$    *
>       - NAME_EXCLUDE    ^/path/to/avoid    *
>
> I'm attaching the updated runtime documentation for Darshan for your
> reference. Section 8 provides a ton of details on how to provide a config
> file to Darshan that should help clear up any missing gaps in my
> description above.
>
> Please let us know if you have any further questions or issues, though!
>
> Thanks,
> --Shane
> ------------------------------
> *From:* Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf
> of Jiří Nádvorník <nadvornik.ji at gmail.com>
> *Sent:* Sunday, April 24, 2022 3:00 PM
> *To:* darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
> *Subject:* [Darshan-users] Module contains incomplete data
>
> Hi All,
>
> I just tried out Darshan and the potential output seems perfect for my
> HDF5 MPI application! Although I'm not able to get there :(.
>
> I have a log that has a big stamp "This darshan log contains incomplete
> data".
>
> When I run:
> darshan-parser --show-incomplete  mylog.darshan |grep incomplete
> Output is:
> # *WARNING*: The POSIX module contains incomplete data!
> # *WARNING*: The MPI-IO module contains incomplete data!
> # *WARNING*: The STDIO module contains incomplete data!
>
> Would you be able to point me to some setting that would improve the
> measurements? Can I actually rely on the profiling results if it says the
> data is incomplete in some of the categories?
>
> Thank you very much for your help!
>
> Cheers,
>
> Jiri
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20220427/5d742569/attachment.html>