<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Darshan-runtime installation and usage</title><link rel="stylesheet" type="text/css" href="docbook-xsl.css" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /></head><body><div xml:lang="en" class="article" lang="en"><div class="titlepage"><div><div><h2 class="title"><a id="idm1"></a>Darshan-runtime installation and usage</h2></div></div><hr /></div><div class="toc"><p><strong>Table of Contents</strong></p><dl class="toc"><dt><span class="section"><a href="#_introduction">1. Introduction</a></span></dt><dt><span class="section"><a href="#_requirements">2. Requirements</a></span></dt><dt><span class="section"><a href="#_conventional_installation">3. Conventional installation</a></span></dt><dd><dl><dt><span class="section"><a href="#_compilation">3.1. Compilation</a></span></dt><dt><span class="section"><a href="#_environment_preparation">3.2. Environment preparation</a></span></dt></dl></dd><dt><span class="section"><a href="#_spack_installation">4. Spack installation</a></span></dt><dt><span class="section"><a href="#_instrumenting_applications">5. Instrumenting applications</a></span></dt><dd><dl><dt><span class="section"><a href="#_option_1_instrumenting_mpi_applications_at_compile_time">5.1. Option 1: Instrumenting MPI applications at compile time</a></span></dt><dt><span class="section"><a href="#_option_2_instrumenting_mpi_applications_at_runtime">5.2. Option 2: Instrumenting MPI applications at runtime</a></span></dt><dt><span class="section"><a href="#_option_3_instrumenting_non_mpi_applications_at_runtime">5.3. Option 3: Instrumenting non-MPI applications at runtime</a></span></dt><dt><span class="section"><a href="#_using_other_profiling_tools_at_the_same_time_as_darshan">5.4. Using other profiling tools at the same time as Darshan</a></span></dt></dl></dd><dt><span class="section"><a href="#_using_the_darshan_extended_tracing_dxt_module">6. Using the Darshan eXtended Tracing (DXT) module</a></span></dt><dt><span class="section"><a href="#_using_autoperf_instrumentation_modules">7. Using AutoPerf instrumentation modules</a></span></dt><dt><span class="section"><a href="#_configuring_darshan_library_at_runtime">8. Configuring Darshan library at runtime</a></span></dt><dd><dl><dt><span class="section"><a href="#_darshan_library_config_settings">8.1. Darshan library config settings</a></span></dt><dt><span class="section"><a href="#_example_darshan_configuration">8.2. Example Darshan configuration</a></span></dt></dl></dd><dt><span class="section"><a href="#_darshan_installation_recipes">9. Darshan installation recipes</a></span></dt><dd><dl><dt><span class="section"><a href="#_cray_platforms_xe_xc_or_similar">9.1. Cray platforms (XE, XC, or similar)</a></span></dt><dt><span class="section"><a href="#_linux_clusters_using_mpich">9.2. Linux clusters using MPICH</a></span></dt><dt><span class="section"><a href="#_linux_clusters_using_intel_mpi">9.3. Linux clusters using Intel MPI</a></span></dt><dt><span class="section"><a href="#_linux_clusters_using_open_mpi">9.4. Linux clusters using Open MPI</a></span></dt></dl></dd><dt><span class="section"><a href="#_debugging">10. Debugging</a></span></dt><dd><dl><dt><span class="section"><a href="#_no_log_file">10.1. No log file</a></span></dt></dl></dd></dl></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_introduction"></a>1. Introduction</h2></div></div></div><p>This document describes darshan-runtime, which is the instrumentation
portion of the Darshan characterization tool. It should be installed on the
system where you intend to collect I/O characterization information.</p><p>Darshan instruments applications via either compile time wrappers or
dynamic library preloading. An application that has been instrumented
with Darshan will produce a single log file each time it is executed.
This log summarizes the I/O access patterns used by the application.</p><p>The darshan-runtime instrumentation has traditionally only supported MPI
applications (specifically, those that call <code class="literal">MPI_Init()</code> and <code class="literal">MPI_Finalize()</code>),
but, as of version 3.2.0, Darshan also supports instrumentation of non-MPI
applications. Regardless of whether MPI is used, Darshan provides detailed
statistics about POSIX level file accesses made by the application.
In the case of MPI applications, Darshan additionally captures details on MPI-IO
and HDF5 level access, as well as limited information about PnetCDF access.
Note that instrumentation of non-MPI applications is currently only supported
in Darshan’s shared library, which applications must <code class="literal">LD_PRELOAD</code>.</p><p>Starting in version 3.0.0, Darshan also exposes an API that can be used to develop
and add new instrumentation modules (for other I/O library interfaces or to gather
system-specific data, for instance), as detailed in
<a class="ulink" href="http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-modularization.html" target="_top">this document</a>.
Newly contributed modules include a module for gathering system-specific parameters
for jobs running on BG/Q systems, a module for gathering Lustre striping data for
files on Lustre file systems, and a module for instrumenting stdio (i.e., stream I/O
functions like <code class="literal">fopen()</code>, <code class="literal">fread()</code>, etc).</p><p>Starting in version 3.1.3, Darshan also allows for full tracing of application I/O
workloads using the newly developed Darshan eXtended Tracing (DxT) instrumentation
module. This module can be selectively enabled at runtime to provide high-fidelity
traces of an application’s I/O workload, as opposed to the coarse-grained I/O summary
data that Darshan has traditionally provided. Currently, DxT only traces at the POSIX
and MPI-IO layers. Initial <a class="ulink" href="DXT-overhead.pdf" target="_top">performance results</a> demonstrate the
low overhead of DxT tracing, offering comparable performance to Darshan’s traditional
coarse-grained instrumentation methods.</p><p>This document provides generic installation instructions, but "recipes" for
several common HPC systems are provided at the end of the document as well.</p><p>More information about Darshan can be found at the
<a class="ulink" href="http://www.mcs.anl.gov/darshan" target="_top">Darshan web site</a>.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_requirements"></a>2. Requirements</h2></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
C compiler (preferably GCC-compatible)
</li><li class="listitem">
zlib development headers and library
</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_conventional_installation"></a>3. Conventional installation</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_compilation"></a>3.1. Compilation</h3></div></div></div><p><strong>Configure and build example (with MPI support). </strong>
</p><pre class="screen">tar -xvzf darshan-<version-number>.tar.gz
cd darshan-<version-number>/darshan-runtime
./configure --with-log-path=/darshan-logs --with-jobid-env=PBS_JOBID CC=mpicc
make
make install</pre><p>
</p><p><strong>Configure and build example (without MPI support). </strong>
</p><pre class="screen">tar -xvzf darshan-<version-number>.tar.gz
cd darshan-<version-number>/darshan-runtime
./configure --with-log-path=/darshan-logs --with-jobid-env=PBS_JOBID --without-mpi CC=gcc
make
make install</pre><p>
</p><div class="itemizedlist"><p class="title"><strong>Explanation of configure arguments:</strong></p><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
<code class="literal">--with-mem-align=NUM</code>: This value is system-dependent and will be used by
Darshan to determine if the buffer for a read or write operation is aligned
in memory (default is 8).
</li><li class="listitem">
<code class="literal">--with-jobid-env=NAME</code> (mandatory): this specifies the environment variable
that Darshan should check to determine the jobid of a job. Common values are
<code class="literal">PBS_JOBID</code> or <code class="literal">COBALT_JOBID</code>. If you are not using a scheduler (or your
scheduler does not advertise the job ID) then you can specify <code class="literal">NONE</code> here.
Darshan will fall back to using the pid of the rank 0 process if the
specified environment variable is not set.
</li><li class="listitem"><p class="simpara">
<code class="literal">--with-log-path=DIR</code> (this, or <code class="literal">--with-log-path-by-env</code>, is mandatory): This
specifies the parent directory for the directory tree where Darshan logs will
be placed.
</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem">
NOTE: after installation, any user can display the configured path with the <code class="literal">darshan-config --log-path</code> command
</li></ul></div></li><li class="listitem">
<code class="literal">--with-log-path-by-env=NAME1,NAME2,...</code>: specifies a comma separated list of
environment variables to check at runtime for log path location before the
one set by <code class="literal">--with-log-path=DIR</code> at configure time.
</li><li class="listitem">
<code class="literal">--with-log-hints=hint1=x;hint2=y,...</code>: specifies hints to use when writing
the Darshan log file. See <code class="literal">./configure --help</code> for details.
</li><li class="listitem">
<code class="literal">--with-mod-mem=NUM</code>: specifies the maximum amount of memory (in MiB) that
active Darshan instrumentation modules can collectively consume.
</li><li class="listitem">
<code class="literal">--with-zlib=DIR</code>: specifies an alternate location for the zlib development
header and library.
</li><li class="listitem">
<code class="literal">--without-mpi</code>: disables MPI support when building Darshan - MPI support is
assumed if not specified.
</li><li class="listitem">
<code class="literal">--enable-mmap-logs</code>: enables the use of Darshan’s mmap log file mechanism.
</li><li class="listitem">
<code class="literal">--disable-cuserid</code>: disables use of cuserid() at runtime.
</li><li class="listitem">
<code class="literal">--disable-ld-preload</code>: disables building of the Darshan LD_PRELOAD library
</li><li class="listitem">
<code class="literal">--enable-group-readable-logs</code>: sets Darshan log file permissions to allow
group read access.
</li><li class="listitem">
<code class="literal">CC=</code>: specifies the C compiler to use for compilation.
</li></ul></div><div class="itemizedlist"><p class="title"><strong>Configure arguments for controlling which Darshan modules to use:</strong></p><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
<code class="literal">--disable-posix-mod</code>: disables compilation and use of Darshan’s POSIX module
(default=enabled)
</li><li class="listitem">
<code class="literal">--disable-mpiio-mod</code>: disables compilation and usee of Darshan’s MPI-IO
module (default=enabled)
</li><li class="listitem">
<code class="literal">--disable-stdio-mod</code>: disables compilation and use of Darshan’s STDIO module
(default=enabled)
</li><li class="listitem">
<code class="literal">--disable-dxt-mod</code>: disables compilation and use of Darshan’s DXT module
(default=enabled)
</li><li class="listitem">
<code class="literal">--enable-hdf5-mod</code>: enables compilation and use of Darshan’s HDF5 module
(default=disabled)
</li><li class="listitem"><p class="simpara">
<code class="literal">--with-hdf5=DIR</code>: installation directory for HDF5
</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem">
NOTE: Users must call <code class="literal">--enable-hdf5-mod</code> to enable HDF5 modules, <code class="literal">--with-hdf5</code> is only used to additionally provide an HDF5 install prefix.
</li><li class="listitem">
NOTE: HDF5 instrumentation only works on HDF5 library versions >=1.8, and further requires that the HDF5 library used to build Darshan and the HDF5 library being linked in either both be version >=1.10 or both be version <1.10
</li><li class="listitem">
NOTE: This option does not work with the profile configuration instrumentation method described in the "Instrumenting applications" section (<a class="ulink" href="darshan-runtime.html#_using_a_profile_configuration" target="_top">Using a profile configuration</a>)
</li></ul></div></li><li class="listitem">
<code class="literal">--disable-pnetcdf-mod</code>: disables compilation and use of Darshan’s PNetCDF
module (default=enabled)
</li><li class="listitem">
<code class="literal">--disable-lustre-mod</code>: disables compilation and use of Darshan’s Lustre
module (default=enabled)
</li><li class="listitem">
<code class="literal">--enable-mdhim-mod</code>: enables compilation and use of Darshan’s MDHIM module
(default=disabled)
</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_environment_preparation"></a>3.2. Environment preparation</h3></div></div></div><p>Once darshan-runtime has been installed, you must prepare a location
in which to store the Darshan log files and configure an instrumentation method.</p><p>This step can be safely skipped if you configured darshan-runtime using the
<code class="literal">--with-log-path-by-env</code> option. A more typical configuration uses a static
directory hierarchy for Darshan log
files.</p><p>The <code class="literal">darshan-mk-log-dirs.pl</code> utility will configure the path specified at
configure time to include
subdirectories organized by year, month, and day in which log files will be
placed. The deepest subdirectories will have sticky permissions to enable
multiple users to write to the same directory. If the log directory is
shared system-wide across many users then the following script should be run
as root.</p><pre class="screen">darshan-mk-log-dirs.pl</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">A note about finding log paths after installation</h3><p>Regardless of whether a Darshan installation is using the --with-log-path or
--with-log-path-by-env option, end users can display the path (and/or
environment variables) at any time by running <code class="literal">darshan-config --log-path</code>
on the command line.</p></div><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">A note about log directory permissions</h3><p>All log files written by Darshan have permissions set to only allow
read access by the owner of the file. You can modify this behavior,
however, by specifying the --enable-group-readable-logs option at
configure time. One notable deployment scenario would be to configure
Darshan and the log directories to allow all logs to be readable by both the
end user and a Darshan administrators group. This can be done with the
following steps:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
set the --enable-group-readable-logs option at configure time
</li><li class="listitem">
create the log directories with darshan-mk-log-dirs.pl
</li><li class="listitem">
recursively set the group ownership of the log directories to the Darshan
administrators group
</li><li class="listitem">
recursively set the setgid bit on the log directories
</li></ul></div></div></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_spack_installation"></a>4. Spack installation</h2></div></div></div><p>You can also install Darshan via <a class="ulink" href="https://spack.io/" target="_top">Spack</a> as an alternative
to manual download, compilation, and installation. This may be
especially convenient for single-user installs. Darshan is divided
into two separate packages for the command line utilities and runtime
instrumentation. You can install either or both as follows:</p><pre class="screen">spack install darshan-util
spack install darshan-runtime</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>Darshan will generally compile and install fine using a variety of
compilers, but we advise using a gcc compiler in Spack to compile Darshan
(regardless of what compiler you will use for your applications) to
ensure maximum runtime compatibility.</p></div><p>You can use the <code class="literal">spack info darshan-runtime</code> query to view the full list of
variants available for the darshan-runtime Spack package. For example, adding a <code class="literal">+slurm</code> to
the command line (<code class="literal">spack install darshan-runtime+slurm</code>) will cause Darshan
to be compiled with support for gathering job ID information from the Slurm
scheduler.</p><p>The following commands will load the Darshan packages once they have been
installed:</p><pre class="screen">spack load -r darshan-util
spack load -r darshan-runtime</pre><p>Note that the spack install of darshan-runtime will use an environment
variable named <code class="literal">$DARSHAN_LOG_DIR_PATH</code> to indicate where it should store log
files. This variable is set to the user’s home directory by default when
the package is loaded, but it may be overridden.</p><p>On Cray systems, you can also perform an additional step to load a
Cray-specific module file. This will make a module called <code class="literal">darshan</code>
available as described later in this document in the Cray platform recipe.
It enables automatic instrumentation when using the standard Cray compiler
wrappers.</p><pre class="screen">module use `spack location -i darshan-runtime`/share/craype-2.x/modulefiles</pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_instrumenting_applications"></a>5. Instrumenting applications</h2></div></div></div><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>More specific installation "recipes" are provided later in this document for
some platforms. This section of the documentation covers general techniques.</p></div><p>Once Darshan has been installed and a log path has been prepared, the next
step is to actually instrument applications. The preferred method is to
instrument applications at compile time.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_option_1_instrumenting_mpi_applications_at_compile_time"></a>5.1. Option 1: Instrumenting MPI applications at compile time</h3></div></div></div><p>This method is applicable to C, Fortran, and C++ MPI applications
(regardless of whether they are static or dynamically linked) and is the most
straightforward method to apply transparently system-wide. It works by
injecting additional libraries and options into the linker command line to
intercept relevant I/O calls.</p><p>On Cray platforms you can enable the compile time instrumentation by simply
loading the Darshan module. It can then be enabled for all users by placing
that module in the default environment. As of Darshan 3.2.0 this will
instrument both static and dynamic executables, while in previous versions
of Darshan this was only sufficient for static executables. See the Cray
installation recipe for more details.</p><p>For other general MPICH-based MPI implementations, you can generate
Darshan-enabled variants of the standard mpicc/mpicxx/mpif90/mpif77
wrappers using the following commands:</p><pre class="screen">darshan-gen-cc.pl `which mpicc` --output mpicc.darshan
darshan-gen-cxx.pl `which mpicxx` --output mpicxx.darshan
darshan-gen-fortran.pl `which mpif77` --output mpif77.darshan
darshan-gen-fortran.pl `which mpif90` --output mpif90.darshan</pre><p>The resulting *.darshan wrappers will transparently inject Darshan
instrumentation into the link step without any explicit user intervention.
They can be renamed and placed in an appropriate
PATH to enable automatic instrumentation. This method also works correctly
for both static and dynamic executables as of Darshan 3.2.0.</p><p>For other systems you can enable compile-time instrumentation by either
manually adding the appropriate link options to your command line or
modifying your default MPI compiler script. The <code class="literal">darshan-config</code> command
line tool can be used to display the options that you should use:</p><pre class="screen"># Linker options to use for dynamic linking (default on most platforms)
# These arguments should go *before* the MPI libraries in the underlying
# linker command line to ensure that Darshan can be activated. It should
# also ideally go before other libraries that may issue I/O function calls.
darshan-config --dyn-ld-flags
# linker options to use for static linking
# The first set of arguments should go early in the link command line
# (before MPI, while the second set should go at the end of the link command
# line
darshan-config --pre-ld-flags
darshan-config --post-ld-flags</pre><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="_using_a_profile_configuration"></a>Using a profile configuration</h4></div></div></div><p>The MPICH MPI implementation supports the specification of a profiling library
configuration that can be used to insert Darshan instrumentation without
modifying the existing MPI compiler script. You can enable a profiling
configuration using environment variables or command line arguments to the
compiler scripts:</p><p>Example for MPICH 3.1.1 or newer:</p><pre class="screen">export MPICC_PROFILE=$DARSHAN_PREFIX/share/mpi-profile/darshan-cc
export MPICXX_PROFILE=$DARSHAN_PREFIX/share/mpi-profile/darshan-cxx
export MPIFORT_PROFILE=$DARSHAN_PREFIX/share/mpi-profile/darshan-f</pre><p>Examples for command line use:</p><pre class="screen">mpicc -profile=$DARSHAN_PREFIX/share/mpi-profile/darshan-c <args>
mpicxx -profile=$DARSHAN_PREFIX/share/mpi-profile/darshan-cxx <args>
mpif77 -profile=$DARSHAN_PREFIX/share/mpi-profile/darshan-f <args>
mpif90 -profile=$DARSHAN_PREFIX/share/mpi-profile/darshan-f <args></pre><p>Note that unlike the previously described methods in this section, this
method <span class="strong"><strong>will not</strong></span> automatically adapt to static and dynamic linking options.
The example profile configurations show above only support dynamic linking.</p><p>Example profile configurations are also provided with a "-static" suffix if
you need examples for static linking.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_option_2_instrumenting_mpi_applications_at_runtime"></a>5.2. Option 2: Instrumenting MPI applications at runtime</h3></div></div></div><p>This method is applicable to pre-compiled dynamically linked executables
as well as interpreted languages such as Python. You do not need to
change your compile options in any way. This method works by injecting
instrumentation at runtime. It will not work for statically linked
executables.</p><p>To use this mechanism, set the <code class="literal">LD_PRELOAD</code> environment variable to the full
path to the Darshan shared library. The preferred method of inserting Darshan
instrumentation in this case is to set the <code class="literal">LD_PRELOAD</code> variable specifically
for the application of interest. Typically this is possible using
command line arguments offered by the <code class="literal">mpirun</code> or <code class="literal">mpiexec</code> scripts or by
the job scheduler:</p><pre class="screen">mpiexec -n 4 -env LD_PRELOAD /home/carns/darshan-install/lib/libdarshan.so mpi-io-test</pre><pre class="screen">srun -n 4 --export=LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test</pre><p>For sequential invocations of MPI programs, the following will set LD_PRELOAD for process duration only:</p><pre class="screen">env LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test</pre><p>Other environments may have other specific options for controlling this behavior.
Please check your local site documentation for details.</p><p>It is also possible to just export LD_PRELOAD as follows, but it is recommended
against doing that to prevent Darshan and MPI symbols from being pulled into
unrelated binaries:</p><pre class="screen">export LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>For SGI systems running the MPT environment, it may be necessary to set the <code class="literal">MPI_SHEPHERD</code>
environment variable equal to <code class="literal">true</code> to avoid deadlock when preloading the Darshan shared
library.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_option_3_instrumenting_non_mpi_applications_at_runtime"></a>5.3. Option 3: Instrumenting non-MPI applications at runtime</h3></div></div></div><p>Similar to the process described in the previous section, Darshan relies on the
<code class="literal">LD_PRELOAD</code> mechanism for instrumenting dynamically-linked non-MPI applications.
This allows Darshan to instrument dynamically-linked binaries produced by non-MPI
compilers (e.g., gcc or clang), extending Darshan instrumentation to new contexts
(like instrumentation of arbitrary Python programs or instrumenting serial
file transfer utilities like <code class="literal">cp</code> and <code class="literal">scp</code>).</p><p>The only additional step required of Darshan non-MPI users is to also set the
DARSHAN_ENABLE_NONMPI environment variable to signal to Darshan that non-MPI
instrumentation is requested:</p><pre class="screen">export DARSHAN_ENABLE_NONMPI=1</pre><p>As described in the previous section, it may be desirable to users to limit the
scope of Darshan’s instrumentation by only enabling LD_PRELOAD on the target
executable:</p><pre class="screen">env LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so io-test</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>Recall that Darshan instrumentation of non-MPI applications is only possible with
dynamically-linked applications.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_using_other_profiling_tools_at_the_same_time_as_darshan"></a>5.4. Using other profiling tools at the same time as Darshan</h3></div></div></div><p>As of Darshan version 3.2.0, Darshan does not necessarily interfere with
other profiling tools (particularly those using the PMPI profiling
interface). Darshan itself does not use the PMPI interface, and instead
uses dynamic linker symbol interception or --wrap function interception for
static executables.</p><p>As a rule of thumb most profiling tools should appear in the linker command
line <span class="strong"><strong>before</strong></span> -ldarshan if possible.</p></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_using_the_darshan_extended_tracing_dxt_module"></a>6. Using the Darshan eXtended Tracing (DXT) module</h2></div></div></div><p>Darshan’s DXT module provides full tracing of MPI-IO and POSIX read/write APIs.
While the DXT module is able to capture finer-grained details compared to traditional
Darshan instrumentation, it may exhibit higher runtime and memory overheads.
For this reason, DXT support is disabled by default in Darshan, but users can opt-in
to DXT instrumentation at runtime by setting their environment as follows:</p><pre class="screen">export DXT_ENABLE_IO_TRACE=1</pre><p>DXT will trace each I/O operation to files instrumented by Darshan’s MPI-IO and
POSIX modules, using a default memory limit of 2 MiB for each module (DXT_POSIX
and DXT_MPIIO). Memory usage and a number of other aspects of DXT tracing can
be configured as described in section
<a class="ulink" href="darshan-runtime.html#_configuring_darshan_library_at_runtime" target="_top">Configuring Darshan library at runtime</a>.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_using_autoperf_instrumentation_modules"></a>7. Using AutoPerf instrumentation modules</h2></div></div></div><p>AutoPerf offers two additional Darshan instrumentation modules that may be enabled for MPI applications.</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
APMPI: Instrumentation of over 70 MPI-3 communication routines, providing operation counts, datatype sizes, and timing information for each application MPI rank.
</li><li class="listitem">
APXC: Instrumentation of Cray XC environments to provide network and compute counters of interest, via PAPI.
</li></ul></div><p>Users can request Darshan to build the APMPI and APXC modules by passing
<code class="literal">--enable-apmpi-mod</code> and <code class="literal">--enable-apxc-mod</code> options to configure, respectively. Note that these options can be requested independently (i.e., you can build Darshan with APMPI support but not APXC support, and vice versa).</p><p>The only prerequsisite for the APMPI module is that Darsan be configured with a MPI-3 compliant compiler. For APXC, the user must obviously be using a Cray XC system and must make the PAPI interface available to Darshan (i.e., by running <code class="literal">module load papi</code>, before building Darshan).</p><p>If using the APMPI module, users can additionally specify the <code class="literal">--enable-apmpi-coll-sync</code> configure option to force Darshan to synchronize before calling underlying MPI routines and to capture additional timing information on how synchronized processes are. Users should note this option will impose additional overheads, but can be useful to help diagnose whether applications are spending a lot of time synchronizing as part of collective communication calls. For this reason, we do not recommend users setting this particular option for production Darshan deployments.</p><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>The AutoPerf instrumentation modules are provided as Git submodules to Darshan’s main repository, so if building Darshan source that has been cloned from Git, it is necessary to first retrieve the AutoPerf submodules by running the following command:</p><pre class="screen">git submodule update --init</pre></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_configuring_darshan_library_at_runtime"></a>8. Configuring Darshan library at runtime</h2></div></div></div><p>To fine tune Darshan library settings (e.g., internal memory usage, instrumentation
scope, etc.), Darshan provides a couple of mechanisms:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
user environment variable overrides
</li><li class="listitem">
a configuration file, which users must specify the path to using the
<code class="literal">DARSHAN_CONFIG_PATH</code> environment variable
</li></ul></div><p>For settings that are specified via a config file and via an environment variable,
the environment settings will take precedence.</p><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><p>Users of facility-provided Darshan installs should be mindful that these
installs could define their own default Darshan config file. In this case,
users should double check that <code class="literal">DARSHAN_CONFIG_PATH</code> environment variable
is not already set, and if it is, users should consider copying the
default config file as a starting point before applying their own settings.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_darshan_library_config_settings"></a>8.1. Darshan library config settings</h3></div></div></div><p>The Darshan library honors the following settings to modify behavior at runtime:</p><div class="table"><a id="idm273"></a><p class="title"><strong>Table 1. Darshan library config settings</strong></p><div class="table-contents"><table class="table" summary="Darshan library config settings" cellpadding="4px" style="border-collapse: collapse;border-top: 3px solid #527bbd; border-bottom: 3px solid #527bbd; border-left: 3px solid #527bbd; border-right: 3px solid #527bbd; "><colgroup><col class="col_1" /><col class="col_2" /><col class="col_3" /></colgroup><thead><tr><th style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"> environment variable setting </th><th style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"> config file setting </th><th style="border-bottom: 1px solid #527bbd; " align="left" valign="top"> description</th></tr></thead><tbody><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_DISABLE=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Disables Darshan instrumentation.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_ENABLE_NONMPI=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Enables Darshan's non-MPI mode, required for applications that do
not call MPI_Init and MPI_Finalize.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_CONFIG_PATH=<path> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the path to a Darshan config file to load settings from.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_DUMP_CONFIG=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DUMP_CONFIG
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Prints the Darshan configuration to stderr at runtime.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_DISABLE_SHARED_REDUCTION=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DISABLE_SHARED_REDUCTION
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Disables the step in Darshan aggregation in which files that were
accessed by all ranks are collapsed into a single cumulative file
record at rank 0. This option retains more per-process information
at the expense of creating larger log files.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_INTERNAL_TIMING=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> INTERNAL_TIMING
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Enables internal instrumentation that will print the time required
to startup and shutdown Darshan to stderr at runtime.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_MODMEM=<val> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MODMEM <val>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the amount of memory (in MiB) Darshan instrumentation
modules can collectively consume (if not specified, a default 4 MiB
quota is used). Overrides any `--with-mod-mem` configure argument.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_NAMEMEM=<val> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> NAMEMEM <val>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the amount of memory (in MiB) Darshan can consume for
storing record names (if not specified, a default 200 KiB quota is
used). Overrides any `--with-name-mem` configure argument.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_MEMALIGN=<val> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MEMALIGN <val>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a value for system memory alignment. Overrides any
`--with-mem-align` configure argument (default is 8 bytes).</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_JOBID=<string> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> JOBID <string>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the name of the environment variable to use for the job
identifier, such as PBS_JOBID. Overrides `--with-jobid-env` configure
argument.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_LOGHINTS=<string> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> LOGHINTS <string>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the MPI-IO hints to use when storing the Darshan output
file. The format is a semicolon-delimited list of key=value pairs,
for example: hint1=value1;hint2=value2. Overrides any
`--with-log-hints` configure argument.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_LOGPATH=<path> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> LOGPATH <path>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the path to write Darshan log files to. Note that this
directory needs to be formatted using the darshan-mk-log-dirs script.
Overrides any `--with-log-path` configure argument.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_MMAP_LOGPATH=<path> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MMAP_LOGPATH <path>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> If Darshan's mmap log file mechanism is enabled, this variable
specifies what path the mmap log files should be stored in (if not
specified, log files will be stored in `/tmp`).</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_LOGFILE=<path> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the path (directory + Darshan log file name) to write
the output Darshan log to. This overrides the default Darshan
behavior of automatically generating a log file name and adding it
to a log file directory formatted using darshan-mk-log-dirs script.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_MOD_DISABLE=<mod_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MOD_DISABLE <mod_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated Darshan module names to
disable at runtime.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_MOD_ENABLE=<mod_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MOD_ENABLE <mod_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated Darshan module names to
enable at runtime.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_APP_EXCLUDE=<regex_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> APP_EXCLUDE <regex_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated regexes that match
application names that should not be instrumented. This is useful
if Darshan is LD_PRELOADed, in which case logs may be generated
for many unintended applications.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_APP_INCLUDE=<regex_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> APP_INCLUDE <regex_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated regexes that match
application names that should be instrumented. This setting is
used to override any APP_INCLUDE rules.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_RANK_EXCLUDE=<rank_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> RANK_EXCLUDE <rank_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated ranks (or rank ranges) that
should not be instrumented. Rank ranges are formatted like
"start:end" (if start or end are not specified, the first or last
rank is assumed, respectively). Note that the Darshan library will
still run on all processes of an application, this setting just
controls whether specific ranks are capturing instrumentation data.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_RANK_INCLUDE=<rank_csv> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> RANK_INCLUDE <rank_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated ranks (or rank ranges) that
should be instrumented. This setting is used to override any
RANK_INCLUDE rules.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_DXT_SMALL_IO_TRIGGER=<val> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DXT_SMALL_IO_TRIGGER <val> </pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a floating point percentage (i.e., ".8" would be 80%)
indicating a threshold of small I/O operation accesses (defined
as accesses smaller than 10 KiB), with DXT trace data being
discarded for files that exhibit a percentage of small I/O
operations less than this threshold.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_DXT_UNALIGNED_IO_TRIGGER=<val> </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DXT_UNALIGNED_IO_TRIGGER <val> </pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a floating point percentage (i.e., ".8" would be 80%)
indicating a threshold of unaligned I/O operation accesses (defined
as accesses not aligned to the file alignment value determined
by Darshan), with DXT trace data being discarded for files that
exhibit a percentage of unaligned I/O operations less than this
threshold.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> MAX_RECORDS <val> <mod_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies the number of records to pre-allocate for each
instrumentation module given in a comma-separated list.
Most modules default to tracing 1024 file records per-process.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> NAME_EXCLUDE <regex_csv> <mod_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated regexes that match
record names that should not be instrumented for
instrumentation modules given in a comma-separated module list.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> NAME_INCLUDE <regex_csv> <mod_csv>
</pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> Specifies a list of comma-separated regexes that match
record names that should be instrumented for instrumentation
modules given in a comma-separated module list. This setting is
used to override any NAME_EXCLUDE rules.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DXT_ENABLE_IO_TRACE=1 </pre></td><td style="border-right: 1px solid #527bbd; border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A </pre></td><td style="border-bottom: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> (DEPRECATED) Setting this environment variable enables the DXT
(Darshan eXtended Tracing) modules at runtime for all files
instrumented by Darshan. Replaced by MODULE_ENABLE setting.</pre></td></tr><tr><td style="border-right: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> DARSHAN_EXCLUDE_DIRS=<path_csv> </pre></td><td style="border-right: 1px solid #527bbd; " align="left" valign="top"><pre class="literallayout"> N/A
</pre></td><td style="" align="left" valign="top"><pre class="literallayout"> (DEPRECATED) Specifies a list of comma-separated paths that
Darshan will not instrument at runtime (in addition to Darshan's
default exclusion list). Replaced by NAME_EXCLUDE setting.</pre></td></tr></tbody></table></div></div><br class="table-break" /><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Note</h3><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
Config file settings must be specified one per-line, with settings and
their parameters separated by any whitespace.
</li><li class="listitem">
Settings that take a comma-separated list of modules can use "*" as a
wildcard to represent all modules.
</li><li class="listitem">
Some config file settings (specifically, <code class="literal">MOD_DISABLE</code>/<code class="literal">ENABLE</code>,
<code class="literal">APP_EXCLUDE</code>/<code class="literal">INCLUDE</code>, <code class="literal">RANK_EXCLUDE</code>/<code class="literal">INCLUDE</code>, <code class="literal">NAME_EXCLUDE</code>/<code class="literal">INCLUDE</code>,
and <code class="literal">MAX_RECORDS</code>) may be repeated multiple times rather than
providing comma-separated values, to ease readability.
</li><li class="listitem">
Improperly formatted config settings are ignored, with Darshan falling
back to its default configuration.
</li><li class="listitem">
All settings that take regular expressions as input expect them to be
formatted according to the POSIX <code class="literal">regex.h</code> interface — refer to the
<a class="ulink" href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/regex.h.html" target="_top">manpage</a>
for more details on regex syntax.
</li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_example_darshan_configuration"></a>8.2. Example Darshan configuration</h3></div></div></div><p>An example configuration file with annotations is given below
(note that comments are allowed by prefixing a line with <code class="literal">#</code>):</p><pre class="screen"># enable DXT modules, which are off by default
MOD_ENABLE DXT_POSIX,DXT_MPIIO
# allocate 4096 file records for POSIX and MPI-IO modules
# (darshan only allocates 1024 per-module by default)
MAX_RECORDS 4096 POSIX,MPI-IO
# the '*' specifier can be used to apply settings for all modules
# in this case, we want all modules to ignore record names
# prefixed with "/home" (i.e., stored in our home directory),
# with a superseding inclusion for files with a ".out" suffix)
NAME_EXCLUDE ^/home *
NAME_INCLUDE .out$ *
# bump up Darshan's default memory usage to 8 MiB
MODMEM 8
# avoid generating logs for git and ls binaries
APP_EXCLUDE git,ls
# exclude instrumentation for all ranks first
RANK_EXCLUDE 0:
# then selectively re-include ranks 0-3 and 12:15
RANK_INCLUDE 0:3
RANK_INCLUDE 12:15
# only retain DXT traces for files that were accessed
# using small I/O ops 20+% of the time
DXT_SMALL_IO_TRIGGER .2</pre><p>This configuration could be similarly set using environment variables,
though note that both <code class="literal">MAX_RECORDS</code> and <code class="literal">NAME_EXCLUDE</code>/<code class="literal">INCLUDE</code>
settings do not have environment variable counterparts:</p><pre class="screen">export DARSHAN_MOD_ENABLE="DXT_POSIX,DXT_MPIIO"
export DARSHAN_MODMEM=8
export DARSHAN_APP_EXCLUDE="git,ls"
export DARSHAN_RANK_EXCLUDE="0:"
export DARSHAN_RANK_INCLUDE="0:3,12:15"
export DARSHAN_DXT_SMALL_IO_TRIGGER=.2</pre></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_darshan_installation_recipes"></a>9. Darshan installation recipes</h2></div></div></div><p>The following recipes provide examples for prominent HPC systems.
These are intended to be used as a starting point. You will most likely have to adjust paths and options to
reflect the specifics of your system.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_cray_platforms_xe_xc_or_similar"></a>9.1. Cray platforms (XE, XC, or similar)</h3></div></div></div><p>This section describes how to compile and install Darshan,
as well as how to use a software module to enable and disable Darshan
instrumentation on Cray systems.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="_building_and_installing_darshan"></a>Building and installing Darshan</h4></div></div></div><p>Please set your environment to use the GNU programming environment before
configuring or compiling Darshan. Although Darshan can be built with a
variety of compilers, the GNU compiler is recommended because it will
produce a Darshan library that is interoperable with the widest range
of compilers and linkers. On most Cray systems you can enable the GNU
programming environment with a command similar to "module swap PrgEnv-intel
PrgEnv-gnu". Please see your site documentation for information about
how to switch programming environments.</p><p>The following example shows how to configure and build Darshan on a Cray
system using the GNU programming environment. Adjust the
--with-log-path and --prefix arguments to point to the desired log file path
and installation path, respectively.</p><pre class="screen">module swap PrgEnv-pgi PrgEnv-gnu
./configure \
--with-log-path=/shared-file-system/darshan-logs \
--prefix=/soft/darshan-2.2.3 \
--with-jobid-env=PBS_JOBID --disable-cuserid CC=cc
make install
module swap PrgEnv-gnu PrgEnv-pgi</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Rationale</h3><p>The job ID is set to <code class="literal">PBS_JOBID</code> for use with a Torque or PBS based scheduler.
The <code class="literal">CC</code> variable is configured to point the standard MPI compiler.</p><p>The --disable-cuserid argument is used to prevent Darshan from attempting to
use the cuserid() function to retrieve the user name associated with a job.
Darshan automatically falls back to other methods if this function fails,
but on some Cray environments (notably the Beagle XE6 system as of March 2012)
the cuserid() call triggers a segmentation fault. With this option set,
Darshan will typically use the LOGNAME environment variable to determine a
userid.</p></div><p>If instrumentation of the HDF5 library is desired, additionally load an
acceptable HDF5 module (e.g., <code class="literal">module load cray-hdf5-parallel</code>) prior to
building and use the the <code class="literal">--enable-hdf5-mod</code> configure argument.
We additionally recommend that you modify Darshan’s generated Cray software
module to include a dependency on the HDF5 software module used — this is
necessary to ensure Darshan library dependencies are satisfied at
application link and run time.</p><pre class="screen">prereq cray-hdf5-parallel</pre><p>Note that the Darshan-enabled Cray compiler wrappers will always prefer
user-supplied HDF5 libraries over the library used to build Darshan.
However, due to ABI changes in the HDF5 library, the two HDF5 libraries
used must be compatible. Specifically, the HDF5 library versions need to
be either both greater than or equal to 1.10 or both less than 1.10. If
users use an HDF5 version that is incompatible with Darshan, either
link or runtime errors will occur and the user will have to switch
HDF5 versions or unload the Darshan module.</p><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Optional RDTSCP timers for Theta</h3><p>Darshan’s default mechanism (<code class="literal">clock_gettime()</code>) for retrieving timing
information may introduce more overhead than expected for statically linked
executables on some platforms. The Theta system at the ALCF (as of July
2021) is a notable example. It uses static linking by default (which
prevents the use of the standard vDSO optimization for <code class="literal">clock_gettime()</code>
calls), and it’s CPU architecture exhibits relatively high system call
overhead. For Theta and other similar platforms you can explicitly request
that Darshan use the <code class="literal">RDTSCP</code> instruction in place of <code class="literal">clock_gettime()</code> for
timing purposes. <code class="literal">RDTSCP</code> is a non-portable, Intel-specific instruction.
It must be enabled explicitly at configure time, and the base clock
frequency of the compute node CPU must be specified.</p><p>This mechanism can be activated on Theta by adding the
<code class="literal">--enable-rdtscp=1300000000</code> to the configure command line (the KNL CPUs on
Theta have a base frequency of 1.3 GHz).</p><p>Note that timer overhead is unlikely to be a factor in overall performance
unless the application has an edge case workload with frequent sequential
I/O operations, such as small I/O accesses to cached data on a single
process.</p></div><p>As in any Darshan installation, the darshan-mk-log-dirs.pl script can then be
used to create the appropriate directory hierarchy for storing Darshan log
files in the --with-log-path directory.</p><p>Note that Darshan is not currently capable of detecting the stripe size
(and therefore the Darshan FILE_ALIGNMENT value) on Lustre file systems.
If a Lustre file system is detected, then Darshan assumes an optimal
file alignment of 1 MiB.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="_enabling_darshan_instrumentation"></a>Enabling Darshan instrumentation</h4></div></div></div><p>Darshan will automatically install example software module files in the
following locations (depending on how you specified the --prefix option in
the previous section):</p><pre class="screen">/soft/darshan-2.2.3/share/craype-1.x/modulefiles/darshan
/soft/darshan-2.2.3/share/craype-2.x/modulefiles/darshan</pre><p>Select the one that is appropriate for your Cray programming environment
(see the version number of the craype module in <code class="literal">module list</code>).</p><p>If you are using the Cray Programming Environment version 1.x, then you
must modify the corresponding modulefile before using it. Please see
the comments at the end of the file and choose an environment variable
method that is appropriate for your system. If this is not done, then
the compiler may fail to link some applications when the Darshan module
is loaded.</p><p>If you are using the Cray Programming Environment version 2.x then you can
likely use the modulefile as is. Note that it pulls most of its
configuration from the lib/pkgconfig/darshan-runtime.pc file installed with
Darshan.</p><p>The modulefile that you select can be copied to a system location, or the
install location can be added to your local module path with the following
command:</p><pre class="screen">module use /soft/darshan-2.2.3/share/craype-<VERSION>/modulefiles</pre><p>From this point, Darshan instrumentation can be enabled for all future
application compilations by running "module load darshan".</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_linux_clusters_using_mpich"></a>9.2. Linux clusters using MPICH</h3></div></div></div><p>Most MPICH installations produce dynamic executables by default. To
configure Darshan in this environment you can use the following example. We
recommend using mpicc with GNU compilers to compile Darshan.</p><pre class="screen">./configure --with-log-path=/darshan-logs --with-jobid-env=PBS_JOBID CC=mpicc</pre><p>The darshan-gen-* scripts described earlier in this document can be used
to create variants of the standard mpicc/mpicxx/mpif77/mpif90 scripts
that are Darshan enabled. These scripts will work correctly for both
dynamic and statically linked executables.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_linux_clusters_using_intel_mpi"></a>9.3. Linux clusters using Intel MPI</h3></div></div></div><p>Most Intel MPI installations produce dynamic executables by default. To
configure Darshan in this environment you can use the following example:</p><pre class="screen">./configure --with-log-path=/darshan-logs --with-jobid-env=PBS_JOBID CC=mpicc</pre><div class="note" style="margin-left: 0; margin-right: 10%;"><h3 class="title">Rationale</h3><p>There is nothing unusual in this configuration except that you should use
the underlying GNU compilers rather than the Intel ICC compilers to compile
Darshan itself.</p></div><p>You can enable Darshan instrumentation at compile time by adding
<code class="literal">darshan-config --dyn-ld-flags</code> options to your linker command line.</p><p>Alternatively you can use <code class="literal">LD_PRELOAD</code> runtime instrumentation method to
instrument executables that have already been compiled.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_linux_clusters_using_open_mpi"></a>9.4. Linux clusters using Open MPI</h3></div></div></div><p>Follow the generic instructions provided at the top of this document for
compilation, and make sure that the <code class="literal">CC</code> used for compilation is based on a
GNU compiler.</p><p>You can enable Darshan instrumentation at compile time by adding
<code class="literal">darshan-config --dyn-ld-flags</code> options to your linker command line.</p><p>Alternatively you can use <code class="literal">LD_PRELOAD</code> runtime instrumentation method to
instrument executables that have already been compiled.</p></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="_debugging"></a>10. Debugging</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_no_log_file"></a>10.1. No log file</h3></div></div></div><p>In cases where Darshan is not generating a log file for an application, some common things to check are:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
Make sure you are looking in the correct place for logs. Confirm the
location with the <code class="literal">darshan-config --log-path</code> command.
</li><li class="listitem">
Check stderr to ensure Darshan isn’t indicating any internal errors (e.g., invalid log file path)
</li></ul></div><p>For statically linked executables:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
Ensure that Darshan symbols are present in the underlying executable by running <code class="literal">nm</code> on it:
</li></ul></div><pre class="screen">> nm test | grep darshan
0000000000772260 b darshan_core
0000000000404440 t darshan_core_cleanup
00000000004049b0 T darshan_core_initialize
000000000076b660 d darshan_core_mutex
00000000004070a0 T darshan_core_register_module</pre><p>For dynamically linked executables:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
Ensure that the Darshan library is present in the list of shared libraries
to be used by the application, and that it appears before the MPI library:
</li></ul></div><pre class="screen">> ldd mpi-io-test
linux-vdso.so.1 (0x00007ffd83925000)
libdarshan.so => /home/carns/working/install/lib/libdarshan.so (0x00007f0f4a7a6000)
libmpi.so.12 => /home/carns/working/src/spack/opt/spack/linux-ubuntu19.10-skylake/gcc-9.2.1/mpich-3.3.2-h3dybprufq7i5kt4hcyfoyihnrnbaogk/lib/libmpi.so.12 (0x00007f0f4a44f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f4a241000)
...</pre><p>General:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p class="simpara">
Ensure that the linker is correctly linking in Darshan’s runtime libraries:
</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p class="simpara">
A common mistake is to explicitly link in the underlying MPI libraries (e.g., <code class="literal">-lmpich</code> or <code class="literal">-lmpichf90</code>)
in the link command, which can interfere with Darshan’s instrumentation
</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: square; "><li class="listitem">
These libraries are usually linked in automatically by the compiler
</li><li class="listitem">
MPICH’s <code class="literal">mpicc</code> comipler’s <code class="literal">-show</code> flag can be used to examine the invoked link command, for instance
</li></ul></div></li><li class="listitem">
The linker’s <code class="literal">-y</code> option can be used to verify that Darshan is properly intercepting MPI_Init
function (e.g. by setting <code class="literal">CFLAGS='-Wl,-yMPI_Init'</code>), which it uses to initialize its runtime structures
</li></ul></div></li></ul></div><pre class="screen">/usr/common/software/darshan/3.0.0-pre3/lib/libdarshan.a(darshan-core-init-finalize.o): definition of MPI_Init</pre></div></div></div></body></html>