<html lang='en'>
<head>
<meta content='text/html; charset=utf-8' http-equiv='Content-Type'>
<title>
GitLab
</title>
</meta>
</head>
<style>
img {
max-width: 100%;
height: auto;
}
p.details {
font-style:italic;
color:#777
}
.footer p {
font-size:small;
color:#777
}
pre.commit-message {
white-space: pre-wrap;
}
.file-stats a {
text-decoration: none;
}
.file-stats .new-file {
color: #090;
}
.file-stats .deleted-file {
color: #B00;
}
</style>
<body>
<div class='content'>
<h3>Shane Snyder pushed to branch mmap-dev at <a href="https://xgitlab.cels.anl.gov/darshan/darshan">darshan / darshan</a></h3>
<h4>
Commits:
</h4>
<ul>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/1fae7e7c73e7d0da9d5b90317e4e7341d4cc1c6d">1fae7e7c</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-06T16:09:09Z</i>
</div>
<pre class='commit-message'>small fixes for COPYRIGHT, ChangeLog, & comments</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/4150a0f951a78a299cbfb777a2c4c561c1c4e655">4150a0f9</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-12T12:03:16Z</i>
</div>
<pre class='commit-message'>update makefile to install uthash header</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/df02ace59f38349b7f49c67120b2fb7e2324dc36">df02ace5</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-12T16:44:25Z</i>
</div>
<pre class='commit-message'>make uthash header install in correct directory</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/320968568dddcd8f81c466c85c03046a08fbc710">32096856</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-13T09:50:02Z</i>
</div>
<pre class='commit-message'>make install updates to support external tools</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/48352d366d6530b22d3fc4a2821c8b7a7e3ea214">48352d36</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-29T16:13:28Z</i>
</div>
<pre class='commit-message'>make maximum runtime records configurable
use --with-max-records at configure time to change from default
of 2048 records</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/98c93e0f45ef35bfde0da5d2b143e0b0de9f6ce5">98c93e0f</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-02T12:34:38Z</i>
</div>
<pre class='commit-message'>darshan-runtime commits for configurable mem
user can now give a config option or use a runtime environment
variable to control darshan's per module memory requirements.</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/40d1dc03b7e336f5127f70a68c3a8069fd92536c">40d1dc03</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-02T13:31:25Z</i>
</div>
<pre class='commit-message'>darshan-util updates to support partial logs</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/2950f613462a892c1847281e5ce9f4257c082d89">2950f613</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-02T16:45:56Z</i>
</div>
<pre class='commit-message'>fix random typos in comments</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/45eb36b1ff0dede4ad5d27d913a20dbfa3524119">45eb36b1</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-03T13:09:06Z</i>
</div>
<pre class='commit-message'>update darshan version in configure</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/8731ade9ebdf93664a369aee29a6af4e2bb3e02f">8731ade9</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-03T13:09:40Z</i>
</div>
<pre class='commit-message'>update darshan modularization docs</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/e810279e0d23dc0451d25d32abecc643ee3fb243">e810279e</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-11T15:43:27Z</i>
</div>
<pre class='commit-message'>Merge branch 'dev-modular' into mmap-dev
Conflicts:
darshan-runtime/darshan-core.h
darshan-runtime/lib/darshan-core.c</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/799d9890ca7c78387435865c3bb7f1496221523b">799d9890</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-11-24T14:45:31Z</i>
</div>
<pre class='commit-message'>update logutils to allow logs with only job info</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/bdb2a5f0075b548e09a1c0197473deb0315d1e12">bdb2a5f0</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-12-01T12:40:28Z</i>
</div>
<pre class='commit-message'>runtime now mmaps header and job data to tmp log</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/3b0c668b50b84b4407f57b5ea40336ef14d0e9d2">3b0c668b</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-12-02T14:36:19Z</i>
</div>
<pre class='commit-message'>update logutils to read uncompressed log files</pre>
</li>
</ul>
<h4>25 changed files:</h4>
<ul>
<li class='file-stats'>
<a href='#diff-0'>
COPYRIGHT
</a>
</li>
<li class='file-stats'>
<a href='#diff-1'>
ChangeLog
</a>
</li>
<li class='file-stats'>
<a href='#diff-2'>
darshan-log-format.h
</a>
</li>
<li class='file-stats'>
<a href='#diff-3'>
darshan-runtime/Makefile.in
</a>
</li>
<li class='file-stats'>
<a href='#diff-4'>
darshan-runtime/configure
</a>
</li>
<li class='file-stats'>
<a href='#diff-5'>
darshan-runtime/configure.in
</a>
</li>
<li class='file-stats'>
<a href='#diff-6'>
darshan-runtime/darshan-core.h
</a>
</li>
<li class='file-stats'>
<a href='#diff-7'>
darshan-runtime/darshan-runtime-config.h.in
</a>
</li>
<li class='file-stats'>
<a href='#diff-8'>
darshan-runtime/darshan.h
</a>
</li>
<li class='file-stats'>
<a href='#diff-9'>
darshan-runtime/doc/darshan-runtime.txt
</a>
</li>
<li class='file-stats'>
<a href='#diff-10'>
darshan-runtime/lib/darshan-bgq.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-11'>
darshan-runtime/lib/darshan-core.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-12'>
darshan-runtime/lib/darshan-hdf5.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-13'>
darshan-runtime/lib/darshan-mpiio.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-14'>
darshan-runtime/lib/darshan-null.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-15'>
darshan-runtime/lib/darshan-pnetcdf.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-16'>
darshan-runtime/lib/darshan-posix.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-17'>
darshan-util/configure
</a>
</li>
<li class='file-stats'>
<a href='#diff-18'>
darshan-util/configure.in
</a>
</li>
<li class='file-stats'>
<a href='#diff-19'>
darshan-util/darshan-convert.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-20'>
darshan-util/darshan-job-summary/bin/darshan-job-summary.pl.in
</a>
</li>
<li class='file-stats'>
<a href='#diff-21'>
darshan-util/darshan-logutils.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-22'>
darshan-util/darshan-logutils.h
</a>
</li>
<li class='file-stats'>
<a href='#diff-23'>
darshan-util/darshan-parser.c
</a>
</li>
<li class='file-stats'>
<a href='#diff-24'>
doc/darshan-modularization.txt
</a>
</li>
</ul>
<h4>Changes:</h4>
<li id='diff-0'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-0'>
<strong>
COPYRIGHT
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/COPYRIGHT
</span><span style="color: #000000;background-color: #ddffdd">+++ b/COPYRIGHT
</span><span style="color: #aaaaaa">@@ -6,7 +6,7 @@ which must be included in the prologue of the code and in all source listings
</span> of the code.
Copyright Notice
<span style="color: #000000;background-color: #ffdddd">- + 2009 University of Chicago
</span><span style="color: #000000;background-color: #ddffdd">+ + 2015 University of Chicago
</span>
Permission is hereby granted to use, reproduce, prepare derivative works, and
to redistribute to others. This software was authored by:
</code></pre>
<br>
</li>
<li id='diff-1'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-1'>
<strong>
ChangeLog
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/ChangeLog
</span><span style="color: #000000;background-color: #ddffdd">+++ b/ChangeLog
</span><span style="color: #aaaaaa">@@ -2,6 +2,16 @@
</span> Darshan Release Change Log
--------------------------
<span style="color: #000000;background-color: #ddffdd">+Darshan-3.0.0-pre2
+=============
+* add fix to install appropriate headers for linking external
+ applications with darshan-util (reported by Matthieu Dorier)
+* add darshan-util Ruby bindings for the new modularized version
+ of Darshan (3.0) (Matthieu Dorier)
+* add enhancement to darshan-runtime to allow per-module instrumentation
+ memory to be user configurable using a configure option or a runtime
+ environment variable
+
</span> Darshan-3.0.0-pre1
=============
* new version of Darshan with the following features/improvements:
<span style="color: #aaaaaa">@@ -18,6 +28,7 @@ Darshan-3.0.0-pre1
</span> suffixes
- a new instrumentation module for capturing BG/Q-specific parameters
(BG/Q environment is automatically detected at configure time)
<span style="color: #000000;background-color: #ddffdd">+ (implemented by Kevin Harms)
</span> - new darshan-parser and darshan-job-summary output to utilize the
new modularized log format
* updated documentation outlining changes in this release, as well as
</code></pre>
<br>
</li>
<li id='diff-2'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-2'>
<strong>
darshan-log-format.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-log-format.h
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-log-format.h
</span><span style="color: #aaaaaa">@@ -69,21 +69,25 @@ static char * const darshan_module_names[] =
</span> };
#undef X
<span style="color: #000000;background-color: #ddffdd">+/* simple macros for accessing module flag bitfields */
+#define DARSHAN_MOD_FLAG_SET(flags, id) flags = (flags | (1 << id))
+#define DARSHAN_MOD_FLAG_UNSET(flags, id) flags = (flags & ~(1 << id))
+#define DARSHAN_MOD_FLAG_ISSET(flags, id) (flags & (1 << id))
+
</span> /* compression method used on darshan log file */
enum darshan_comp_type
{
DARSHAN_ZLIB_COMP,
DARSHAN_BZIP2_COMP,
<span style="color: #000000;background-color: #ddffdd">+ DARSHAN_NO_COMP,
</span> };
typedef uint64_t darshan_record_id;
/* the darshan_log_map structure is used to indicate the location of
* specific module data in a Darshan log. Note that 'off' and 'len' are
<span style="color: #000000;background-color: #ffdddd">- * the respective offset and length of the data in the file, in *uncompressed*
- * terms -- this is nonintuitive since the data is compressed, but this is
- * done so we can utilize the gzread interface for all Darshan log reading
- * utilities.
</span><span style="color: #000000;background-color: #ddffdd">+ * the respective offset and length of the data in the file, in
+ * *compressed* terms
</span> */
struct darshan_log_map
{
<span style="color: #aaaaaa">@@ -99,7 +103,7 @@ struct darshan_header
</span> char version_string[8];
int64_t magic_nr;
unsigned char comp_type;
<span style="color: #000000;background-color: #ffdddd">- unsigned char partial_flag;
</span><span style="color: #000000;background-color: #ddffdd">+ uint32_t partial_flag;
</span> struct darshan_log_map rec_map;
struct darshan_log_map mod_map[DARSHAN_MAX_MODS];
};
</code></pre>
<br>
</li>
<li id='diff-3'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-3'>
<strong>
darshan-runtime/Makefile.in
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/Makefile.in
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/Makefile.in
</span><span style="color: #aaaaaa">@@ -1,4 +1,5 @@
</span>-all: lib/libdarshan.a lib/libdarshan-stubs.a lib/darshan-null.o
<span style="color: #000000;background-color: #ddffdd">+all: lib/libdarshan.a lib/libdarshan-stubs.a
+#TODO: lib/darshan-null.o
</span>
#TODO: each module provides own makefile with module-specific objects, build options, etc.
</code></pre>
<br>
</li>
<li id='diff-4'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-4'>
<strong>
darshan-runtime/configure
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/configure
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/configure
</span><span style="color: #aaaaaa">@@ -1,6 +1,6 @@
</span> #! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for darshan-runtime 3.0.0-pre1.
<span style="color: #000000;background-color: #ddffdd">+# Generated by GNU Autoconf 2.69 for darshan-runtime 3.0.0-pre2.
</span> #
#
# Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
<span style="color: #aaaaaa">@@ -577,8 +577,8 @@ MAKEFLAGS=
</span> # Identity of this package.
PACKAGE_NAME='darshan-runtime'
PACKAGE_TARNAME='darshan-runtime'
-PACKAGE_VERSION='3.0.0-pre1'
-PACKAGE_STRING='darshan-runtime 3.0.0-pre1'
<span style="color: #000000;background-color: #ddffdd">+PACKAGE_VERSION='3.0.0-pre2'
+PACKAGE_STRING='darshan-runtime 3.0.0-pre2'
</span> PACKAGE_BUGREPORT=''
PACKAGE_URL=''
<span style="color: #aaaaaa">@@ -693,6 +693,7 @@ with_log_hints
</span> with_log_path
with_jobid_env
enable_bgq_mod
<span style="color: #000000;background-color: #ddffdd">+with_mod_mem
</span> '
ac_precious_vars='build_alias
host_alias
<span style="color: #aaaaaa">@@ -1243,7 +1244,7 @@ if test "$ac_init_help" = "long"; then
</span> # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures darshan-runtime 3.0.0-pre1 to adapt to many kinds of systems.
<span style="color: #000000;background-color: #ddffdd">+\`configure' configures darshan-runtime 3.0.0-pre2 to adapt to many kinds of systems.
</span>
Usage: $0 [OPTION]... [VAR=VALUE]...
<span style="color: #aaaaaa">@@ -1304,7 +1305,7 @@ fi
</span>
if test -n "$ac_init_help"; then
case $ac_init_help in
<span style="color: #000000;background-color: #ffdddd">- short | recursive ) echo "Configuration of darshan-runtime 3.0.0-pre1:";;
</span><span style="color: #000000;background-color: #ddffdd">+ short | recursive ) echo "Configuration of darshan-runtime 3.0.0-pre2:";;
</span> esac
cat <<\_ACEOF
<span style="color: #aaaaaa">@@ -1330,6 +1331,7 @@ Optional Packages:
</span> --with-jobid-env=<name> Name of environment variable that stores the jobid
(specify "NONE" if no appropriate environment variable is available:
Darshan will use rank 0's pid instead)
<span style="color: #000000;background-color: #ddffdd">+ --with-mod-mem=<num> Maximum amount of memory (in MiB) for each Darshan module
</span>
Some influential environment variables:
CC C compiler command
<span style="color: #aaaaaa">@@ -1407,7 +1409,7 @@ fi
</span> test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-darshan-runtime configure 3.0.0-pre1
<span style="color: #000000;background-color: #ddffdd">+darshan-runtime configure 3.0.0-pre2
</span> generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
<span style="color: #aaaaaa">@@ -1759,7 +1761,7 @@ cat >config.log <<_ACEOF
</span> This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by darshan-runtime $as_me 3.0.0-pre1, which was
<span style="color: #000000;background-color: #ddffdd">+It was created by darshan-runtime $as_me 3.0.0-pre2, which was
</span> generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
<span style="color: #aaaaaa">@@ -4278,7 +4280,23 @@ fi
</span> rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
fi
-DARSHAN_VERSION="3.0.0-pre1"
<span style="color: #000000;background-color: #ddffdd">+
+# Check whether --with-mod-mem was given.
+if test "${with_mod_mem+set}" = set; then :
+ withval=$with_mod_mem; if test x$withval = xyes; then
+ as_fn_error $? "--with-mod-mem must be given a number" "$LINENO" 5
+ else
+
+cat >>confdefs.h <<_ACEOF
+#define __DARSHAN_MOD_MEM_MAX ${withval}
+_ACEOF
+
+ fi
+
+fi
+
+
+DARSHAN_VERSION="3.0.0-pre2"
</span>
<span style="color: #aaaaaa">@@ -4797,7 +4815,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
</span> # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by darshan-runtime $as_me 3.0.0-pre1, which was
<span style="color: #000000;background-color: #ddffdd">+This file was extended by darshan-runtime $as_me 3.0.0-pre2, which was
</span> generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
<span style="color: #aaaaaa">@@ -4859,7 +4877,7 @@ _ACEOF
</span> cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-darshan-runtime config.status 3.0.0-pre1
<span style="color: #000000;background-color: #ddffdd">+darshan-runtime config.status 3.0.0-pre2
</span> configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"
</code></pre>
<br>
</li>
<li id='diff-5'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-5'>
<strong>
darshan-runtime/configure.in
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/configure.in
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/configure.in
</span><span style="color: #aaaaaa">@@ -5,7 +5,7 @@ dnl Process this file with autoconf to produce a configure script.
</span> dnl You may need to use autoheader as well if changing any DEFINEs
dnl sanity checks, output header, location of scripts used here
-AC_INIT([darshan-runtime], [3.0.0-pre1])
<span style="color: #000000;background-color: #ddffdd">+AC_INIT([darshan-runtime], [3.0.0-pre2])
</span> AC_CONFIG_SRCDIR([darshan.h])
AC_CONFIG_AUX_DIR(../maint/config)
AC_CONFIG_HEADER(darshan-runtime-config.h)
<span style="color: #aaaaaa">@@ -300,6 +300,15 @@ if test x$enable_bgq_mod != xno; then
</span> []))
fi
<span style="color: #000000;background-color: #ddffdd">+AC_ARG_WITH(mod-mem,
+[ --with-mod-mem=<num> Maximum amount of memory (in MiB) for each Darshan module],
+ if test x$withval = xyes; then
+ AC_MSG_ERROR(--with-mod-mem must be given a number)
+ else
+ AC_DEFINE_UNQUOTED(__DARSHAN_MOD_MEM_MAX, ${withval}, Maximum memory (in MiB) for each Darshan module)
+ fi
+)
+
</span> DARSHAN_VERSION="AC_PACKAGE_VERSION"
AC_SUBST(darshan_lib_path)
</code></pre>
<br>
</li>
<li id='diff-6'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-6'>
<strong>
darshan-runtime/darshan-core.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/darshan-core.h
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/darshan-core.h
</span><span style="color: #aaaaaa">@@ -26,33 +26,32 @@
</span> /* Environment variable to override __DARSHAN_MEM_ALIGNMENT */
#define DARSHAN_MEM_ALIGNMENT_OVERRIDE "DARSHAN_MEMALIGN"
-/* maximum number of records that can be tracked on a single process */
-#define DARSHAN_CORE_MAX_RECORDS 2048
<span style="color: #000000;background-color: #ddffdd">+/* Environment variable to override memory per module */
+#define DARSHAN_MOD_MEM_OVERRIDE "DARSHAN_MODMEM"
</span>
-/* TODO: revisit this default size if we change memory per module */
-#define DARSHAN_CORE_COMP_BUF_SIZE (2 * 1024 * 1024)
<span style="color: #000000;background-color: #ddffdd">+/* Maximum amount of memory per instrumentation module in MiB */
+#ifdef __DARSHAN_MOD_MEM_MAX
+#define DARSHAN_MOD_MEM_MAX (__DARSHAN_MOD_MEM_MAX * 1024 * 1024)
+#else
+#define DARSHAN_MOD_MEM_MAX (2 * 1024 * 1024) /* 2 MiB default */
+#endif
</span>
-/* this controls the maximum mmapped memory each module can use */
-#define DARSHAN_MMAP_CHUNK_SIZE (4 * 1024)
-
-#define DARSHAN_CORE_MOD_SET(flags, id) (flags | (1 << id))
-#define DARSHAN_CORE_MOD_UNSET(flags, id) (flags & ~(1 << id))
-#define DARSHAN_CORE_MOD_ISSET(flags, id) (flags & (1 << id))
<span style="color: #000000;background-color: #ddffdd">+/* Default runtime compression buffer size */
+#define DARSHAN_COMP_BUF_SIZE DARSHAN_MOD_MEM_MAX
</span>
/* in memory structure to keep up with job level data */
struct darshan_core_runtime
{
<span style="color: #000000;background-color: #ffdddd">- /* XXX-MMAP */
</span><span style="color: #000000;background-color: #ddffdd">+ struct darshan_header *log_hdr_p;
+ struct darshan_job *log_job_p;
+ char *log_exemnt_p;
+ /* XXX: MMAP */
</span> void *mmap_p;
<span style="color: #000000;background-color: #ffdddd">- struct darshan_job *mmap_job_p;
- char *mmap_exe_mnt_p;
- void *mmap_mod_p;
- /* XXX-MMAP */
</span>
struct darshan_core_record_ref *rec_hash;
int rec_count;
struct darshan_core_module* mod_array[DARSHAN_MAX_MODS];
<span style="color: #000000;background-color: #ffdddd">- char comp_buf[DARSHAN_CORE_COMP_BUF_SIZE];
</span><span style="color: #000000;background-color: #ddffdd">+ char comp_buf[DARSHAN_COMP_BUF_SIZE]; /* TODO: why is this allocated statically? */
</span> double wtime_offset;
};
</code></pre>
<br>
</li>
<li id='diff-7'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-7'>
<strong>
darshan-runtime/darshan-runtime-config.h.in
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/darshan-runtime-config.h.in
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/darshan-runtime-config.h.in
</span><span style="color: #aaaaaa">@@ -105,5 +105,8 @@
</span> /* Memory alignment in bytes */
#undef __DARSHAN_MEM_ALIGNMENT
<span style="color: #000000;background-color: #ddffdd">+/* Maximum memory (in MiB) for each Darshan module */
+#undef __DARSHAN_MOD_MEM_MAX
+
</span> /* Generalized request type for MPI-IO */
#undef __D_MPI_REQUEST
</code></pre>
<br>
</li>
<li id='diff-8'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-8'>
<strong>
darshan-runtime/darshan.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/darshan.h
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/darshan.h
</span><span style="color: #aaaaaa">@@ -98,10 +98,9 @@ struct darshan_module_funcs
</span> void darshan_core_register_module(
darshan_module_id mod_id,
struct darshan_module_funcs *funcs,
<span style="color: #000000;background-color: #ddffdd">+ void **mod_buf,
+ int *mod_buf_size,
</span> int *my_rank,
<span style="color: #000000;background-color: #ffdddd">- int *mod_mem_limit,
- void **mmap_buf,
- int *mmap_buf_size,
</span> int *sys_mem_alignment);
/* darshan_core_unregister_module()
<span style="color: #aaaaaa">@@ -117,18 +116,21 @@ void darshan_core_unregister_module(
</span> * Register the Darshan record given by 'name' with the darshan-core
* runtime, allowing it to be properly tracked and (potentially)
* correlated with records from other modules. 'len' is the size of
<span style="color: #000000;background-color: #ffdddd">- * the name pointer (string length for string names), 'printable_flag'
- * indicates whether the name is a string, and 'mod_id' is the identifier
- * of the calling module. 'rec_id' is an output pointer storing the
- * correspoing Darshan record identifier and 'file_alignment' is an
- * output pointer storing the file system alignment value for the given
- * record.
</span><span style="color: #000000;background-color: #ddffdd">+ * the name pointer (string length for string names), and 'printable_flag'
+ * indicates whether the name is a string. 'mod_limit_flag' is set if
+ * the calling module is out of memory (to prevent darshan-core from
+ * creating new records and to just search existing records) and 'mod_id'
+ * is the identifier of the calling module. 'rec_id' is an output pointer
+ * storing the correspoing Darshan record identifier and 'file_alignment'
+ * is an output pointer storing the file system alignment value for the
+ * given record.
</span> */
void darshan_core_register_record(
void *name,
int len,
<span style="color: #000000;background-color: #ffdddd">- int printable_flag,
</span> darshan_module_id mod_id,
<span style="color: #000000;background-color: #ddffdd">+ int printable_flag,
+ int mod_limit_flag,
</span> darshan_record_id *rec_id,
int *file_alignment);
</code></pre>
<br>
</li>
<li id='diff-9'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-9'>
<strong>
darshan-runtime/doc/darshan-runtime.txt
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/doc/darshan-runtime.txt
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/doc/darshan-runtime.txt
</span><span style="color: #aaaaaa">@@ -57,6 +57,8 @@ will be placed.
</span> determine the log path at run time.
* `--with-log-hints=`: specifies hints to use when writing the Darshan log
file. See `./configure --help` for details.
<span style="color: #000000;background-color: #ddffdd">+* `--with-mod-mem=`: specifies the maximum amount of memory (in MiB) that
+each Darshan module can consume.
</span> * `--with-zlib=`: specifies an alternate location for the zlib development
header and library.
* `CC=`: specifies the MPI C compiler to use for compilation.
<span style="color: #aaaaaa">@@ -474,3 +476,4 @@ behavior at runtime:
</span> * DARSHAN_DISABLE_SHARED_REDUCTION: disables the step in Darshan aggregation in which files that were accessed by all ranks are collapsed into a single cumulative file record at rank 0. This option retains more per-process information at the expense of creating larger log files. Note that it is up to individual instrumentation module implementations whether this environment variable is actually honored.
* DARSHAN_LOGPATH: specifies the path to write Darshan log files to. Note that this directory needs to be formatted using the darshan-mk-log-dirs script.
* DARSHAN_LOGFILE: specifies the path (directory + Darshan log file name) to write the output Darshan log to. This overrides the default Darshan behavior of automatically generating a log file name and adding it to a log file directory formatted using darshan-mk-log-dirs script.
<span style="color: #000000;background-color: #ddffdd">+* DARSHAN_MODMEM: specifies the maximum amount of memory (in MiB) a Darshan instrumentation module can consume at runtime.
</span></code></pre>
<br>
</li>
<li id='diff-10'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-10'>
<strong>
darshan-runtime/lib/darshan-bgq.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-bgq.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-bgq.c
</span><span style="color: #aaaaaa">@@ -163,8 +163,9 @@ void bgq_runtime_initialize()
</span> darshan_core_register_record(
recname,
strlen(recname),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_BGQ_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ 0,
</span> &bgq_runtime->record.f_id,
&bgq_runtime->record.alignment);
</code></pre>
<br>
</li>
<li id='diff-11'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-11'>
<strong>
darshan-runtime/lib/darshan-core.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-core.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-core.c
</span><span style="color: #aaaaaa">@@ -100,7 +100,8 @@ static void darshan_get_exe_and_mounts(
</span> static void darshan_block_size_from_path(
const char *path, int *block_size);
static void darshan_get_shared_records(
<span style="color: #000000;background-color: #ffdddd">- struct darshan_core_runtime *core, darshan_record_id *shared_recs);
</span><span style="color: #000000;background-color: #ddffdd">+ struct darshan_core_runtime *core, darshan_record_id **shared_recs,
+ int *shared_rec_cnt);
</span> static int darshan_log_open_all(
char *logfile_name, MPI_File *log_fh);
static int darshan_deflate_buffer(
<span style="color: #aaaaaa">@@ -199,11 +200,8 @@ void darshan_core_initialize(int argc, char **argv)
</span> sys_page_size = sysconf(_SC_PAGESIZE);
assert(sys_page_size > 0);
<span style="color: #000000;background-color: #ffdddd">- /* set the size of the mmap, making sure to round up to the
- * nearest page size. One mmap chunk is used for the job-level
- * metadata, and the rest are statically assigned to modules
- */
- mmap_size = (1 + DARSHAN_MAX_MODS) * DARSHAN_MMAP_CHUNK_SIZE;
</span><span style="color: #000000;background-color: #ddffdd">+ /* XXX: MMAP */
+ mmap_size = sizeof(struct darshan_header) + DARSHAN_JOB_RECORD_SIZE + DARSHAN_MOD_MEM_MAX;
</span> if(mmap_size % sys_page_size)
mmap_size = ((mmap_size / sys_page_size) + 1) * sys_page_size;
<span style="color: #aaaaaa">@@ -223,6 +221,7 @@ void darshan_core_initialize(int argc, char **argv)
</span> return;
}
<span style="color: #000000;background-color: #ddffdd">+ /* TODO: what's more expensive? truncate or write zeros? perf test this call and later accesses */
</span> /* allocate the necessary space in the log file */
ret = ftruncate(mmap_fd, mmap_size);
if(ret < 0)
<span style="color: #aaaaaa">@@ -253,18 +252,27 @@ void darshan_core_initialize(int argc, char **argv)
</span> /* close darshan log file (this does *not* unmap the log file) */
close(mmap_fd);
<span style="color: #000000;background-color: #ffdddd">- /* set the pointers for each log file region */
- init_core->mmap_job_p = (struct darshan_job *)(init_core->mmap_p);
- init_core->mmap_exe_mnt_p =
- (char *)(((char *)init_core->mmap_p) + sizeof(struct darshan_job));
- init_core->mmap_mod_p =
- (void *)(((char *)init_core->mmap_p) + DARSHAN_MMAP_CHUNK_SIZE);
</span><span style="color: #000000;background-color: #ddffdd">+ /* set the memory pointers for each log file region */
+ init_core->log_hdr_p = (struct darshan_header *)
+ (init_core->mmap_p);
+ init_core->log_job_p = (struct darshan_job *)
+ (init_core->log_hdr_p + sizeof(struct darshan_header));
+ init_core->log_exemnt_p = (char *)
+ (((char *)init_core->log_job_p) + sizeof(struct darshan_job));
+ /* TODO: file hash & module memory */
+
+ /* XXX: MMAP */
+
+ /* set known header fields for the log file */
+ strcpy(init_core->log_hdr_p->version_string, DARSHAN_LOG_VERSION);
+ init_core->log_hdr_p->magic_nr = DARSHAN_MAGIC_NR;
+ init_core->log_hdr_p->comp_type = DARSHAN_NO_COMP;
</span>
<span style="color: #000000;background-color: #ffdddd">- /* set known job-level metadata files for the log file */
- init_core->mmap_job_p->uid = getuid();
- init_core->mmap_job_p->start_time = time(NULL);
- init_core->mmap_job_p->nprocs = nprocs;
- init_core->mmap_job_p->jobid = (int64_t)jobid;
</span><span style="color: #000000;background-color: #ddffdd">+ /* set known job-level metadata fields for the log file */
+ init_core->log_job_p->uid = getuid();
+ init_core->log_job_p->start_time = time(NULL);
+ init_core->log_job_p->nprocs = nprocs;
+ init_core->log_job_p->jobid = (int64_t)jobid;
</span>
/* if we are using any hints to write the log file, then record those
* hints with the darshan job information
<span style="color: #aaaaaa">@@ -274,9 +282,7 @@ void darshan_core_initialize(int argc, char **argv)
</span> /* collect information about command line and mounted file systems */
darshan_get_exe_and_mounts(init_core, argc, argv);
<span style="color: #000000;background-color: #ffdddd">- /* TODO: what would be needed in a termination routine? set job end time? */
</span>-
<span style="color: #000000;background-color: #ffdddd">- /* maybe bootstrap modules with static initializers */
</span><span style="color: #000000;background-color: #ddffdd">+ /* bootstrap any modules with static initialization routines */
</span> i = 0;
while(mod_static_init_fns[i])
{
<span style="color: #aaaaaa">@@ -336,7 +342,7 @@ void darshan_core_shutdown()
</span> }
DARSHAN_CORE_UNLOCK();
<span style="color: #000000;background-color: #ffdddd">- final_core->mmap_job_p->end_time = time(NULL);
</span><span style="color: #000000;background-color: #ddffdd">+ final_core->log_job_p->end_time = time(NULL);
</span>
darshan_core_cleanup(final_core);
<span style="color: #aaaaaa">@@ -518,11 +524,11 @@ static void darshan_get_logfile_name(char* logfile_name, int jobid, struct tm* s
</span> return;
}
-/* record any hints used to write the darshan log in the log header */
<span style="color: #000000;background-color: #ddffdd">+/* record any hints used to write the darshan log in the job data */
</span> static void darshan_log_record_hints_and_ver(struct darshan_core_runtime* core)
{
char* hints;
<span style="color: #000000;background-color: #ffdddd">- char* header_hints;
</span><span style="color: #000000;background-color: #ddffdd">+ char* job_hints;
</span> int meta_remain = 0;
char* m;
<span style="color: #aaaaaa">@@ -538,28 +544,28 @@ static void darshan_log_record_hints_and_ver(struct darshan_core_runtime* core)
</span> if(!hints || strlen(hints) < 1)
return;
<span style="color: #000000;background-color: #ffdddd">- header_hints = strdup(hints);
- if(!header_hints)
</span><span style="color: #000000;background-color: #ddffdd">+ job_hints = strdup(hints);
+ if(!job_hints)
</span> return;
meta_remain = DARSHAN_JOB_METADATA_LEN -
<span style="color: #000000;background-color: #ffdddd">- strlen(core->mmap_job_p->metadata) - 1;
</span><span style="color: #000000;background-color: #ddffdd">+ strlen(core->log_job_p->metadata) - 1;
</span> if(meta_remain >= (strlen(PACKAGE_VERSION) + 9))
{
<span style="color: #000000;background-color: #ffdddd">- sprintf(core->mmap_job_p->metadata, "lib_ver=%s\n", PACKAGE_VERSION);
</span><span style="color: #000000;background-color: #ddffdd">+ sprintf(core->log_job_p->metadata, "lib_ver=%s\n", PACKAGE_VERSION);
</span> meta_remain -= (strlen(PACKAGE_VERSION) + 9);
}
<span style="color: #000000;background-color: #ffdddd">- if(meta_remain >= (3 + strlen(header_hints)))
</span><span style="color: #000000;background-color: #ddffdd">+ if(meta_remain >= (3 + strlen(job_hints)))
</span> {
<span style="color: #000000;background-color: #ffdddd">- m = core->mmap_job_p->metadata + strlen(core->mmap_job_p->metadata);
</span><span style="color: #000000;background-color: #ddffdd">+ m = core->log_job_p->metadata + strlen(core->log_job_p->metadata);
</span> /* We have room to store the hints in the metadata portion of
<span style="color: #000000;background-color: #ffdddd">- * the job header. We just prepend an h= to the hints list. The
</span><span style="color: #000000;background-color: #ddffdd">+ * the job structure. We just prepend an h= to the hints list. The
</span> * metadata parser will ignore = characters that appear in the value
* portion of the metadata key/value pair.
*/
<span style="color: #000000;background-color: #ffdddd">- sprintf(m, "h=%s\n", header_hints);
</span><span style="color: #000000;background-color: #ddffdd">+ sprintf(m, "h=%s\n", job_hints);
</span> }
<span style="color: #000000;background-color: #ffdddd">- free(header_hints);
</span><span style="color: #000000;background-color: #ddffdd">+ free(job_hints);
</span>
return;
}
<span style="color: #aaaaaa">@@ -603,7 +609,7 @@ static void add_entry(char* buf, int* space_left, struct mntent *entry)
</span> else
mnt_data_array[mnt_data_count].block_size = 4096;
<span style="color: #000000;background-color: #ffdddd">- /* store mount information for use in header of darshan log */
</span><span style="color: #000000;background-color: #ddffdd">+ /* store mount information with the job-level metadata in darshan log */
</span> ret = snprintf(tmp_mnt, 256, "\n%s\t%s",
entry->mnt_type, entry->mnt_dir);
if(ret < 256 && strlen(tmp_mnt) <= (*space_left))
<span style="color: #aaaaaa">@@ -619,7 +625,7 @@ static void add_entry(char* buf, int* space_left, struct mntent *entry)
</span> /* darshan_get_exe_and_mounts_root()
*
* collects command line and list of mounted file systems into a string that
<span style="color: #000000;background-color: #ffdddd">- * will be stored with the job header
</span><span style="color: #000000;background-color: #ddffdd">+ * will be stored with the job-level metadata
</span> */
static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
int argc, char **argv)
<span style="color: #aaaaaa">@@ -655,12 +661,12 @@ static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
</span> /* record exe and arguments */
for(i=0; i<argc; i++)
{
<span style="color: #000000;background-color: #ffdddd">- strncat(core->mmap_exe_mnt_p, argv[i], space_left);
- space_left = DARSHAN_EXE_LEN-strlen(core->mmap_exe_mnt_p);
</span><span style="color: #000000;background-color: #ddffdd">+ strncat(core->log_exemnt_p, argv[i], space_left);
+ space_left = DARSHAN_EXE_LEN-strlen(core->log_exemnt_p);
</span> if(i < (argc-1))
{
<span style="color: #000000;background-color: #ffdddd">- strncat(core->mmap_exe_mnt_p, " ", space_left);
- space_left = DARSHAN_EXE_LEN-strlen(core->mmap_exe_mnt_p);
</span><span style="color: #000000;background-color: #ddffdd">+ strncat(core->log_exemnt_p, " ", space_left);
+ space_left = DARSHAN_EXE_LEN-strlen(core->log_exemnt_p);
</span> }
}
<span style="color: #aaaaaa">@@ -669,21 +675,20 @@ static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
</span> */
if(argc == 0)
{
<span style="color: #000000;background-color: #ffdddd">- strncat(core->mmap_exe_mnt_p, __progname_full, space_left);
- space_left = DARSHAN_EXE_LEN-strlen(core->mmap_exe_mnt_p);
- strncat(core->mmap_exe_mnt_p, " <unknown args>", space_left);
- space_left = DARSHAN_EXE_LEN-strlen(core->mmap_exe_mnt_p);
</span><span style="color: #000000;background-color: #ddffdd">+ strncat(core->log_exemnt_p, __progname_full, space_left);
+ space_left = DARSHAN_EXE_LEN-strlen(core->log_exemnt_p);
+ strncat(core->log_exemnt_p, " <unknown args>", space_left);
+ space_left = DARSHAN_EXE_LEN-strlen(core->log_exemnt_p);
</span> }
if(space_left == 0)
{
/* we ran out of room; mark that string was truncated */
truncate_offset = DARSHAN_EXE_LEN - strlen(truncate_string);
<span style="color: #000000;background-color: #ffdddd">- sprintf(&core->mmap_exe_mnt_p[truncate_offset], "%s",
</span><span style="color: #000000;background-color: #ddffdd">+ sprintf(&(core->log_exemnt_p[truncate_offset]), "%s",
</span> truncate_string);
}
-
/* we make two passes through mounted file systems; in the first pass we
* grab any non-nfs mount points, then on the second pass we grab nfs
* mount points
<span style="color: #aaaaaa">@@ -711,7 +716,7 @@ static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
</span> if(skip || (strcmp(entry->mnt_type, "nfs") == 0))
continue;
<span style="color: #000000;background-color: #ffdddd">- add_entry(core->mmap_exe_mnt_p, &space_left, entry);
</span><span style="color: #000000;background-color: #ddffdd">+ add_entry(core->log_exemnt_p, &space_left, entry);
</span> }
endmntent(tab);
<span style="color: #aaaaaa">@@ -724,7 +729,7 @@ static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
</span> if(strcmp(entry->mnt_type, "nfs") != 0)
continue;
<span style="color: #000000;background-color: #ffdddd">- add_entry(core->mmap_exe_mnt_p, &space_left, entry);
</span><span style="color: #000000;background-color: #ddffdd">+ add_entry(core->log_exemnt_p, &space_left, entry);
</span> }
endmntent(tab);
<span style="color: #aaaaaa">@@ -739,7 +744,7 @@ static void darshan_get_exe_and_mounts_root(struct darshan_core_runtime *core,
</span> /* darshan_get_exe_and_mounts()
*
* collects command line and list of mounted file systems into a string that
<span style="color: #000000;background-color: #ffdddd">- * will be stored with the job header
</span><span style="color: #000000;background-color: #ddffdd">+ * will be stored with the job-level metadata
</span> */
static void darshan_get_exe_and_mounts(struct darshan_core_runtime *core,
int argc, char **argv)
<span style="color: #aaaaaa">@@ -777,32 +782,45 @@ static void darshan_block_size_from_path(const char *path, int *block_size)
</span> }
static void darshan_get_shared_records(struct darshan_core_runtime *core,
<span style="color: #000000;background-color: #ffdddd">- darshan_record_id *shared_recs)
</span><span style="color: #000000;background-color: #ddffdd">+ darshan_record_id **shared_recs, int *shared_rec_cnt)
</span> {
<span style="color: #000000;background-color: #ffdddd">- int i;
- int ndx;
</span><span style="color: #000000;background-color: #ddffdd">+ int i, j;
+ int tmp_cnt = core->rec_count;
</span> struct darshan_core_record_ref *tmp, *ref;
<span style="color: #000000;background-color: #ffdddd">- darshan_record_id id_array[DARSHAN_CORE_MAX_RECORDS] = {0};
- uint64_t mod_flags[DARSHAN_CORE_MAX_RECORDS] = {0};
- uint64_t global_mod_flags[DARSHAN_CORE_MAX_RECORDS] = {0};
</span><span style="color: #000000;background-color: #ddffdd">+ darshan_record_id *id_array;
+ uint64_t *mod_flags;
+ uint64_t *global_mod_flags;
+
+ /* broadcast root's number of records to all other processes */
+ DARSHAN_MPI_CALL(PMPI_Bcast)(&tmp_cnt, 1, MPI_INT, 0, MPI_COMM_WORLD);
+
+ /* use root record count to allocate data structures */
+ id_array = malloc(tmp_cnt * sizeof(darshan_record_id));
+ mod_flags = malloc(tmp_cnt * sizeof(uint64_t));
+ global_mod_flags = malloc(tmp_cnt * sizeof(uint64_t));
+ *shared_recs = malloc(tmp_cnt * sizeof(darshan_record_id));
+ assert(id_array && mod_flags && global_mod_flags && *shared_recs);
+
+ memset(mod_flags, 0, tmp_cnt * sizeof(uint64_t));
+ memset(global_mod_flags, 0, tmp_cnt * sizeof(uint64_t));
+ memset(*shared_recs, 0, tmp_cnt * sizeof(darshan_record_id));
</span>
/* first, determine list of records root process has opened */
if(my_rank == 0)
{
<span style="color: #000000;background-color: #ffdddd">- ndx = 0;
</span><span style="color: #000000;background-color: #ddffdd">+ i = 0;
</span> HASH_ITER(hlink, core->rec_hash, ref, tmp)
{
<span style="color: #000000;background-color: #ffdddd">- id_array[ndx++] = ref->rec.id;
</span><span style="color: #000000;background-color: #ddffdd">+ id_array[i++] = ref->rec.id;
</span> }
}
/* broadcast root's list of records to all other processes */
<span style="color: #000000;background-color: #ffdddd">- DARSHAN_MPI_CALL(PMPI_Bcast)(id_array,
- (DARSHAN_CORE_MAX_RECORDS * sizeof(darshan_record_id)),
</span><span style="color: #000000;background-color: #ddffdd">+ DARSHAN_MPI_CALL(PMPI_Bcast)(id_array, (tmp_cnt * sizeof(darshan_record_id)),
</span> MPI_BYTE, 0, MPI_COMM_WORLD);
/* everyone looks to see if they opened the same records as root */
<span style="color: #000000;background-color: #ffdddd">- for(i=0; (i<DARSHAN_CORE_MAX_RECORDS && id_array[i] != 0); i++)
</span><span style="color: #000000;background-color: #ddffdd">+ for(i=0; i<tmp_cnt; i++)
</span> {
HASH_FIND(hlink, core->rec_hash, &id_array[i], sizeof(darshan_record_id), ref);
if(ref)
<span style="color: #aaaaaa">@@ -815,15 +833,15 @@ static void darshan_get_shared_records(struct darshan_core_runtime *core,
</span> /* now allreduce so everyone agrees which files are shared and
* which modules accessed them collectively
*/
<span style="color: #000000;background-color: #ffdddd">- DARSHAN_MPI_CALL(PMPI_Allreduce)(mod_flags, global_mod_flags,
- DARSHAN_CORE_MAX_RECORDS, MPI_UINT64_T, MPI_BAND, MPI_COMM_WORLD);
</span><span style="color: #000000;background-color: #ddffdd">+ DARSHAN_MPI_CALL(PMPI_Allreduce)(mod_flags, global_mod_flags, tmp_cnt,
+ MPI_UINT64_T, MPI_BAND, MPI_COMM_WORLD);
</span>
<span style="color: #000000;background-color: #ffdddd">- ndx = 0;
- for(i=0; (i<DARSHAN_CORE_MAX_RECORDS && id_array[i] != 0); i++)
</span><span style="color: #000000;background-color: #ddffdd">+ j = 0;
+ for(i=0; i<tmp_cnt; i++)
</span> {
if(global_mod_flags[i] != 0)
{
<span style="color: #000000;background-color: #ffdddd">- shared_recs[ndx++] = id_array[i];
</span><span style="color: #000000;background-color: #ddffdd">+ (*shared_recs)[j++] = id_array[i];
</span>
/* set global_mod_flags so we know which modules collectively
* accessed this module. we need this info to support shared
<span style="color: #aaaaaa">@@ -834,6 +852,7 @@ static void darshan_get_shared_records(struct darshan_core_runtime *core,
</span> ref->global_mod_flags = global_mod_flags[i];
}
}
<span style="color: #000000;background-color: #ddffdd">+ *shared_rec_cnt = j;
</span>
return;
}
<span style="color: #aaaaaa">@@ -938,7 +957,7 @@ static int darshan_deflate_buffer(void **pointers, int *lengths, int count,
</span> }
tmp_stream.next_out = (unsigned char *)comp_buf;
<span style="color: #000000;background-color: #ffdddd">- tmp_stream.avail_out = DARSHAN_CORE_COMP_BUF_SIZE;
</span><span style="color: #000000;background-color: #ddffdd">+ tmp_stream.avail_out = DARSHAN_COMP_BUF_SIZE;
</span>
/* loop over the input pointers */
for(i = 0; i < count; i++)
<span style="color: #aaaaaa">@@ -999,9 +1018,9 @@ static int darshan_log_write_record_hash(MPI_File log_fh, struct darshan_core_ru
</span> char *hash_buf;
char *hash_buf_off;
<span style="color: #000000;background-color: #ffdddd">- /* allocate a buffer to store at most 64 bytes for each of a max number of records */
</span><span style="color: #000000;background-color: #ddffdd">+ /* allocate a buffer to store at most 64 bytes for each registered record */
</span> /* NOTE: this buffer may be reallocated if estimate is too small */
<span style="color: #000000;background-color: #ffdddd">- hash_buf_sz = DARSHAN_CORE_MAX_RECORDS * 64;
</span><span style="color: #000000;background-color: #ddffdd">+ hash_buf_sz = core->rec_count * 64;
</span> hash_buf = malloc(hash_buf_sz);
if(!hash_buf)
{
<span style="color: #aaaaaa">@@ -1165,38 +1184,37 @@ static void darshan_core_cleanup(struct darshan_core_runtime* core)
</span> void darshan_core_register_module(
darshan_module_id mod_id,
struct darshan_module_funcs *funcs,
<span style="color: #000000;background-color: #ddffdd">+ void **mod_buf,
+ int *mod_buf_size,
</span> int *my_rank,
<span style="color: #000000;background-color: #ffdddd">- int *mod_mem_limit,
- void **mmap_buf,
- int *mmap_buf_size,
</span> int *sys_mem_alignment)
{
<span style="color: #000000;background-color: #ddffdd">+ int ret;
+ int tmpval;
</span> struct darshan_core_module* mod;
<span style="color: #000000;background-color: #ffdddd">- *mod_mem_limit = 0;
</span><span style="color: #000000;background-color: #ddffdd">+ char *mod_mem_str = NULL;
+
+ *mod_buf_size = 0;
+ *mod_buf = NULL;
</span>
if(!darshan_core || (mod_id >= DARSHAN_MAX_MODS))
return;
<span style="color: #000000;background-color: #ddffdd">+ /* XXX */
+ return;
+ /* XXX how do we assign size and address */
+
</span> if(sys_mem_alignment)
*sys_mem_alignment = darshan_mem_alignment;
/* get the calling process's rank */
DARSHAN_MPI_CALL(PMPI_Comm_rank)(MPI_COMM_WORLD, my_rank);
<span style="color: #000000;background-color: #ffdddd">- /* pass back the mmap buffer this module can use to persist
- * some module data (mmap_buf_size at max) even in the case
- * where darshan is not finalized
- */
- *mmap_buf =
- (void *)(((char *)darshan_core->mmap_mod_p) + (mod_id * DARSHAN_MMAP_CHUNK_SIZE));
- *mmap_buf_size = DARSHAN_MMAP_CHUNK_SIZE;
</span>-
/* see if this module is already registered */
DARSHAN_CORE_LOCK();
if(darshan_core->mod_array[mod_id])
{
/* if module is already registered just return */
<span style="color: #000000;background-color: #ffdddd">- /* NOTE: we do not recalculate memory limit here, just set to 0 */
</span> DARSHAN_CORE_UNLOCK();
return;
}
<span style="color: #aaaaaa">@@ -1215,9 +1233,8 @@ void darshan_core_register_module(
</span> /* register module with darshan */
darshan_core->mod_array[mod_id] = mod;
<span style="color: #000000;background-color: #ffdddd">- /* TODO: something smarter than just 2 MiB per module */
- *mod_mem_limit = 2 * 1024 * 1024;
</span>-
<span style="color: #000000;background-color: #ddffdd">+ /* get the calling process's rank */
+ DARSHAN_MPI_CALL(PMPI_Comm_rank)(MPI_COMM_WORLD, my_rank);
</span> DARSHAN_CORE_UNLOCK();
return;
<span style="color: #aaaaaa">@@ -1251,8 +1268,9 @@ void darshan_core_unregister_module(
</span> void darshan_core_register_record(
void *name,
int len,
<span style="color: #000000;background-color: #ffdddd">- int printable_flag,
</span> darshan_module_id mod_id,
<span style="color: #000000;background-color: #ddffdd">+ int printable_flag,
+ int mod_limit_flag,
</span> darshan_record_id *rec_id,
int *file_alignment)
{
<span style="color: #aaaaaa">@@ -1274,19 +1292,19 @@ void darshan_core_register_record(
</span> HASH_FIND(hlink, darshan_core->rec_hash, &tmp_rec_id, sizeof(darshan_record_id), ref);
if(!ref)
{
<span style="color: #000000;background-color: #ffdddd">- /* record not found -- add it to the hash if we aren't already tracking the
- * maximum number of records
</span><span style="color: #000000;background-color: #ddffdd">+ /* record not found -- add it to the hash if this module has not already used
+ * all of its memory
</span> */
<span style="color: #000000;background-color: #ffdddd">-
- if(darshan_core->rec_count >= DARSHAN_CORE_MAX_RECORDS)
</span><span style="color: #000000;background-color: #ddffdd">+
+#if 0
+ if(mod_limit_flag)
</span> {
<span style="color: #000000;background-color: #ffdddd">- /* if we are already tracking the max records, set a flag to indicate
- * that this log file has partial results
- */
- //darshan_core->logfile_header_p->partial_flag = 1;
</span><span style="color: #000000;background-color: #ddffdd">+ /* if this module is OOM, set a flag in the header to indicate this */
+ DARSHAN_MOD_FLAG_SET(darshan_core->log_header.partial_flag, mod_id);
</span> DARSHAN_CORE_UNLOCK();
return;
}
<span style="color: #000000;background-color: #ddffdd">+#endif
</span>
ref = malloc(sizeof(struct darshan_core_record_ref));
if(ref)
<span style="color: #aaaaaa">@@ -1301,7 +1319,7 @@ void darshan_core_register_record(
</span> darshan_core->rec_count++;
}
}
<span style="color: #000000;background-color: #ffdddd">- ref->mod_flags = DARSHAN_CORE_MOD_SET(ref->mod_flags, mod_id);
</span><span style="color: #000000;background-color: #ddffdd">+ DARSHAN_MOD_FLAG_SET(ref->mod_flags, mod_id);
</span> DARSHAN_CORE_UNLOCK();
if(file_alignment)
<span style="color: #aaaaaa">@@ -1326,7 +1344,7 @@ void darshan_core_unregister_record(
</span> assert(ref);
/* disassociate this module from the given record id */
<span style="color: #000000;background-color: #ffdddd">- ref->mod_flags = DARSHAN_CORE_MOD_UNSET(ref->mod_flags, mod_id);
</span><span style="color: #000000;background-color: #ddffdd">+ DARSHAN_MOD_FLAG_UNSET(ref->mod_flags, mod_id);
</span> if(!(ref->mod_flags))
{
/* if no other modules are associated with this rec, delete it */
</code></pre>
<br>
</li>
<li id='diff-12'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-12'>
<strong>
darshan-runtime/lib/darshan-hdf5.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-hdf5.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-hdf5.c
</span><span style="color: #aaaaaa">@@ -263,6 +263,7 @@ static struct hdf5_file_runtime* hdf5_file_by_name(const char *name)
</span> struct hdf5_file_runtime *file = NULL;
char *newname = NULL;
darshan_record_id file_id;
<span style="color: #000000;background-color: #ddffdd">+ int limit_flag;
</span>
if(!hdf5_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #aaaaaa">@@ -271,12 +272,15 @@ static struct hdf5_file_runtime* hdf5_file_by_name(const char *name)
</span> if(!newname)
newname = (char*)name;
<span style="color: #000000;background-color: #ddffdd">+ limit_flag = (hdf5_runtime->file_array_ndx >= hdf5_runtime->file_array_size);
+
</span> /* get a unique id for this file from darshan core */
darshan_core_register_record(
(void*)newname,
strlen(newname),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_HDF5_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ limit_flag,
</span> &file_id,
NULL);
<span style="color: #aaaaaa">@@ -299,19 +303,15 @@ static struct hdf5_file_runtime* hdf5_file_by_name(const char *name)
</span> return(file);
}
<span style="color: #000000;background-color: #ffdddd">- if(hdf5_runtime->file_array_ndx < hdf5_runtime->file_array_size);
- {
- /* no existing record, assign a new file record from the global array */
- file = &(hdf5_runtime->file_runtime_array[hdf5_runtime->file_array_ndx]);
- file->file_record = &(hdf5_runtime->file_record_array[hdf5_runtime->file_array_ndx]);
- file->file_record->f_id = file_id;
- file->file_record->rank = my_rank;
</span>-
<span style="color: #000000;background-color: #ffdddd">- /* add new record to file hash table */
- HASH_ADD(hlink, hdf5_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
</span><span style="color: #000000;background-color: #ddffdd">+ /* no existing record, assign a new file record from the global array */
+ file = &(hdf5_runtime->file_runtime_array[hdf5_runtime->file_array_ndx]);
+ file->file_record = &(hdf5_runtime->file_record_array[hdf5_runtime->file_array_ndx]);
+ file->file_record->f_id = file_id;
+ file->file_record->rank = my_rank;
</span>
<span style="color: #000000;background-color: #ffdddd">- hdf5_runtime->file_array_ndx++;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* add new record to file hash table */
+ HASH_ADD(hlink, hdf5_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
+ hdf5_runtime->file_array_ndx++;
</span>
if(newname != name)
free(newname);
</code></pre>
<br>
</li>
<li id='diff-13'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-13'>
<strong>
darshan-runtime/lib/darshan-mpiio.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-mpiio.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-mpiio.c
</span><span style="color: #aaaaaa">@@ -884,6 +884,7 @@ static struct mpiio_file_runtime* mpiio_file_by_name(const char *name)
</span> struct mpiio_file_runtime *file = NULL;
char *newname = NULL;
darshan_record_id file_id;
<span style="color: #000000;background-color: #ddffdd">+ int limit_flag;
</span>
if(!mpiio_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #aaaaaa">@@ -892,17 +893,20 @@ static struct mpiio_file_runtime* mpiio_file_by_name(const char *name)
</span> if(!newname)
newname = (char*)name;
<span style="color: #000000;background-color: #ddffdd">+ limit_flag = (mpiio_runtime->file_array_ndx >= mpiio_runtime->file_array_size);
+
</span> /* get a unique id for this file from darshan core */
darshan_core_register_record(
(void*)newname,
strlen(newname),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_MPIIO_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ limit_flag,
</span> &file_id,
NULL);
<span style="color: #000000;background-color: #ffdddd">- /* if record is set to 0, darshan-core is out of space and will not
- * track this record, so we should avoid tracking it, too
</span><span style="color: #000000;background-color: #ddffdd">+ /* the file record id is set to 0 if no memory is available for tracking
+ * new records -- just fall through and ignore this record
</span> */
if(file_id == 0)
{
<span style="color: #aaaaaa">@@ -920,19 +924,15 @@ static struct mpiio_file_runtime* mpiio_file_by_name(const char *name)
</span> return(file);
}
<span style="color: #000000;background-color: #ffdddd">- if(mpiio_runtime->file_array_ndx < mpiio_runtime->file_array_size);
- {
- /* no existing record, assign a new file record from the global array */
- file = &(mpiio_runtime->file_runtime_array[mpiio_runtime->file_array_ndx]);
- file->file_record = &(mpiio_runtime->file_record_array[mpiio_runtime->file_array_ndx]);
- file->file_record->f_id = file_id;
- file->file_record->rank = my_rank;
</span>-
<span style="color: #000000;background-color: #ffdddd">- /* add new record to file hash table */
- HASH_ADD(hlink, mpiio_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
</span><span style="color: #000000;background-color: #ddffdd">+ /* no existing record, assign a new file record from the global array */
+ file = &(mpiio_runtime->file_runtime_array[mpiio_runtime->file_array_ndx]);
+ file->file_record = &(mpiio_runtime->file_record_array[mpiio_runtime->file_array_ndx]);
+ file->file_record->f_id = file_id;
+ file->file_record->rank = my_rank;
</span>
<span style="color: #000000;background-color: #ffdddd">- mpiio_runtime->file_array_ndx++;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* add new record to file hash table */
+ HASH_ADD(hlink, mpiio_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
+ mpiio_runtime->file_array_ndx++;
</span>
if(newname != name)
free(newname);
</code></pre>
<br>
</li>
<li id='diff-14'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-14'>
<strong>
darshan-runtime/lib/darshan-null.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-null.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-null.c
</span><span style="color: #aaaaaa">@@ -275,6 +275,7 @@ static struct null_record_runtime* null_record_by_name(const char *name)
</span> {
struct null_record_runtime *rec = NULL;
darshan_record_id rec_id;
<span style="color: #000000;background-color: #ddffdd">+ int limit_flag;
</span>
/* Don't search for a record if the "NULL" module is not initialized or
* if instrumentation has been toggled off.
<span style="color: #aaaaaa">@@ -282,15 +283,27 @@ static struct null_record_runtime* null_record_by_name(const char *name)
</span> if(!null_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #000000;background-color: #ddffdd">+ /* stop tracking new records if we are tracking our maximum count */
+ limit_flag = (null_runtime->rec_array_ndx >= null_runtime->rec_array_size);
+
</span> /* get a unique record identifier for this record from darshan-core */
darshan_core_register_record(
(void*)name,
strlen(name),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_NULL_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ limit_flag,
</span> &rec_id,
NULL);
<span style="color: #000000;background-color: #ddffdd">+ /* the file record id is set to 0 if no memory is available for tracking
+ * new records -- just fall through and ignore this record
+ */
+ if(rec_id == 0)
+ {
+ return(NULL);
+ }
+
</span> /* search the hash table for this file record, and return if found */
HASH_FIND(hlink, null_runtime->record_hash, &rec_id, sizeof(darshan_record_id), rec);
if(rec)
<span style="color: #aaaaaa">@@ -298,21 +311,17 @@ static struct null_record_runtime* null_record_by_name(const char *name)
</span> return(rec);
}
<span style="color: #000000;background-color: #ffdddd">- if(null_runtime->rec_array_ndx < null_runtime->rec_array_size);
- {
- /* no existing record, assign a new one from the global array */
- rec = &(null_runtime->runtime_record_array[null_runtime->rec_array_ndx]);
- rec->record_p = &(null_runtime->record_array[null_runtime->rec_array_ndx]);
</span><span style="color: #000000;background-color: #ddffdd">+ /* no existing record, assign a new one from the global array */
+ rec = &(null_runtime->runtime_record_array[null_runtime->rec_array_ndx]);
+ rec->record_p = &(null_runtime->record_array[null_runtime->rec_array_ndx]);
</span>
<span style="color: #000000;background-color: #ffdddd">- /* set the darshan record id and corresponding process rank for this record */
- rec->record_p->f_id = rec_id;
- rec->record_p->rank = my_rank;
</span><span style="color: #000000;background-color: #ddffdd">+ /* set the darshan record id and corresponding process rank for this record */
+ rec->record_p->f_id = rec_id;
+ rec->record_p->rank = my_rank;
</span>
<span style="color: #000000;background-color: #ffdddd">- /* add new record to file hash table */
- HASH_ADD(hlink, null_runtime->record_hash, record_p->f_id, sizeof(darshan_record_id), rec);
</span>-
<span style="color: #000000;background-color: #ffdddd">- null_runtime->rec_array_ndx++;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* add new record to file hash table */
+ HASH_ADD(hlink, null_runtime->record_hash, record_p->f_id, sizeof(darshan_record_id), rec);
+ null_runtime->rec_array_ndx++;
</span>
return(rec);
}
</code></pre>
<br>
</li>
<li id='diff-15'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-15'>
<strong>
darshan-runtime/lib/darshan-pnetcdf.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-pnetcdf.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-pnetcdf.c
</span><span style="color: #aaaaaa">@@ -275,6 +275,7 @@ static struct pnetcdf_file_runtime* pnetcdf_file_by_name(const char *name)
</span> struct pnetcdf_file_runtime *file = NULL;
char *newname = NULL;
darshan_record_id file_id;
<span style="color: #000000;background-color: #ddffdd">+ int limit_flag;
</span>
if(!pnetcdf_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #aaaaaa">@@ -283,17 +284,20 @@ static struct pnetcdf_file_runtime* pnetcdf_file_by_name(const char *name)
</span> if(!newname)
newname = (char*)name;
<span style="color: #000000;background-color: #ddffdd">+ limit_flag = (pnetcdf_runtime->file_array_ndx >= pnetcdf_runtime->file_array_size);
+
</span> /* get a unique id for this file from darshan core */
darshan_core_register_record(
(void*)newname,
strlen(newname),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_PNETCDF_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ limit_flag,
</span> &file_id,
NULL);
<span style="color: #000000;background-color: #ffdddd">- /* if record is set to 0, darshan-core is out of space and will not
- * track this record, so we should avoid tracking it, too
</span><span style="color: #000000;background-color: #ddffdd">+ /* the file record id is set to 0 if no memory is available for tracking
+ * new records -- just fall through and ignore this record
</span> */
if(file_id == 0)
{
<span style="color: #aaaaaa">@@ -311,19 +315,15 @@ static struct pnetcdf_file_runtime* pnetcdf_file_by_name(const char *name)
</span> return(file);
}
<span style="color: #000000;background-color: #ffdddd">- if(pnetcdf_runtime->file_array_ndx < pnetcdf_runtime->file_array_size);
- {
- /* no existing record, assign a new file record from the global array */
- file = &(pnetcdf_runtime->file_runtime_array[pnetcdf_runtime->file_array_ndx]);
- file->file_record = &(pnetcdf_runtime->file_record_array[pnetcdf_runtime->file_array_ndx]);
- file->file_record->f_id = file_id;
- file->file_record->rank = my_rank;
</span>-
<span style="color: #000000;background-color: #ffdddd">- /* add new record to file hash table */
- HASH_ADD(hlink, pnetcdf_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
</span><span style="color: #000000;background-color: #ddffdd">+ /* no existing record, assign a new file record from the global array */
+ file = &(pnetcdf_runtime->file_runtime_array[pnetcdf_runtime->file_array_ndx]);
+ file->file_record = &(pnetcdf_runtime->file_record_array[pnetcdf_runtime->file_array_ndx]);
+ file->file_record->f_id = file_id;
+ file->file_record->rank = my_rank;
</span>
<span style="color: #000000;background-color: #ffdddd">- pnetcdf_runtime->file_array_ndx++;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* add new record to file hash table */
+ HASH_ADD(hlink, pnetcdf_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
+ pnetcdf_runtime->file_array_ndx++;
</span>
if(newname != name)
free(newname);
</code></pre>
<br>
</li>
<li id='diff-16'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-16'>
<strong>
darshan-runtime/lib/darshan-posix.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-runtime/lib/darshan-posix.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-runtime/lib/darshan-posix.c
</span><span style="color: #aaaaaa">@@ -177,8 +177,6 @@ struct posix_runtime
</span> int file_array_ndx;
struct posix_file_runtime* file_hash;
struct posix_file_runtime_ref* fd_hash;
-
<span style="color: #000000;background-color: #ffdddd">- struct posix_file_runtime agg_file_runtime;
</span> };
static struct posix_runtime *posix_runtime = NULL;
<span style="color: #aaaaaa">@@ -1625,9 +1623,8 @@ static void posix_runtime_initialize()
</span> .get_output_data = &posix_get_output_data,
.shutdown = &posix_shutdown
};
<span style="color: #000000;background-color: #ffdddd">- int mem_limit;
- void *mmap_buf;
- int mmap_buf_size;
</span><span style="color: #000000;background-color: #ddffdd">+ void *psx_buf;
+ int psx_buf_size;
</span>
/* don't do anything if already initialized or instrumenation is disabled */
if(posix_runtime || instrumentation_disabled)
<span style="color: #aaaaaa">@@ -1637,14 +1634,13 @@ static void posix_runtime_initialize()
</span> darshan_core_register_module(
DARSHAN_POSIX_MOD,
&posix_mod_fns,
<span style="color: #000000;background-color: #ddffdd">+ &psx_buf,
+ &psx_buf_size,
</span> &my_rank,
<span style="color: #000000;background-color: #ffdddd">- &mem_limit,
- &mmap_buf,
- &mmap_buf_size,
</span> &darshan_mem_alignment);
<span style="color: #000000;background-color: #ffdddd">- /* return if no memory assigned by darshan core */
- if(mem_limit == 0)
</span><span style="color: #000000;background-color: #ddffdd">+ /* return if no memory assigned by darshan-core */
+ if(psx_buf_size == 0)
</span> return;
posix_runtime = malloc(sizeof(*posix_runtime));
<span style="color: #aaaaaa">@@ -1654,35 +1650,22 @@ static void posix_runtime_initialize()
</span>
/* set maximum number of file records according to max memory limit */
/* NOTE: maximum number of records is based on the size of a posix file record */
<span style="color: #000000;background-color: #ffdddd">- /* TODO: should we base memory usage off file record or total runtime structure sizes? */
- posix_runtime->file_array_size = mem_limit / sizeof(struct darshan_posix_file);
</span><span style="color: #000000;background-color: #ddffdd">+ posix_runtime->file_array_size = psx_buf_size / sizeof(struct darshan_posix_file);
</span> posix_runtime->file_array_ndx = 0;
/* allocate array of runtime file records */
posix_runtime->file_runtime_array = malloc(posix_runtime->file_array_size *
sizeof(struct posix_file_runtime));
<span style="color: #000000;background-color: #ffdddd">- posix_runtime->file_record_array = malloc(posix_runtime->file_array_size *
- sizeof(struct darshan_posix_file));
- if(!posix_runtime->file_runtime_array || !posix_runtime->file_record_array)
</span><span style="color: #000000;background-color: #ddffdd">+ if(!posix_runtime->file_runtime_array)
</span> {
posix_runtime->file_array_size = 0;
return;
}
memset(posix_runtime->file_runtime_array, 0, posix_runtime->file_array_size *
sizeof(struct posix_file_runtime));
<span style="color: #000000;background-color: #ffdddd">- memset(posix_runtime->file_record_array, 0, posix_runtime->file_array_size *
- sizeof(struct darshan_posix_file));
</span>
<span style="color: #000000;background-color: #ffdddd">- /* XXX-MMAP */
- if(mmap_buf_size >= sizeof(struct darshan_posix_file))
- {
- memset(&(posix_runtime->agg_file_runtime), 0,
- sizeof(struct posix_file_runtime));
- posix_runtime->agg_file_runtime.file_record =
- (struct darshan_posix_file *)mmap_buf;
- posix_runtime->agg_file_runtime.file_record->f_id = DARSHAN_POSIX_MOD;
- posix_runtime->agg_file_runtime.file_record->rank = my_rank;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* store pointer to POSIX record buffer given by darshan-core */
+ posix_runtime->file_record_array = (struct darshan_posix_file *)psx_buf;
</span>
return;
}
<span style="color: #aaaaaa">@@ -1694,27 +1677,29 @@ static struct posix_file_runtime* posix_file_by_name(const char *name)
</span> char *newname = NULL;
darshan_record_id file_id;
int file_alignment;
<span style="color: #000000;background-color: #ddffdd">+ int limit_flag;
</span>
if(!posix_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #000000;background-color: #ffdddd">- return(&(posix_runtime->agg_file_runtime));
</span>-#if 0
newname = darshan_clean_file_path(name);
if(!newname)
newname = (char*)name;
<span style="color: #000000;background-color: #ddffdd">+ limit_flag = (posix_runtime->file_array_ndx >= posix_runtime->file_array_size);
+
</span> /* get a unique id for this file from darshan core */
darshan_core_register_record(
(void*)newname,
strlen(newname),
<span style="color: #000000;background-color: #ffdddd">- 1,
</span> DARSHAN_POSIX_MOD,
<span style="color: #000000;background-color: #ddffdd">+ 1,
+ limit_flag,
</span> &file_id,
&file_alignment);
<span style="color: #000000;background-color: #ffdddd">- /* if record is set to 0, darshan-core is out of space and will not
- * track this record, so we should avoid tracking it, too
</span><span style="color: #000000;background-color: #ddffdd">+ /* the file record id is set to 0 if no memory is available for tracking
+ * new records -- just fall through and ignore this record
</span> */
if(file_id == 0)
{
<span style="color: #aaaaaa">@@ -1732,26 +1717,21 @@ static struct posix_file_runtime* posix_file_by_name(const char *name)
</span> return(file);
}
<span style="color: #000000;background-color: #ffdddd">- if(posix_runtime->file_array_ndx < posix_runtime->file_array_size);
- {
- /* no existing record, assign a new file record from the global array */
- file = &(posix_runtime->file_runtime_array[posix_runtime->file_array_ndx]);
- file->file_record = &(posix_runtime->file_record_array[posix_runtime->file_array_ndx]);
- file->file_record->f_id = file_id;
- file->file_record->rank = my_rank;
- file->file_record->counters[POSIX_MEM_ALIGNMENT] = darshan_mem_alignment;
- file->file_record->counters[POSIX_FILE_ALIGNMENT] = file_alignment;
</span>-
<span style="color: #000000;background-color: #ffdddd">- /* add new record to file hash table */
- HASH_ADD(hlink, posix_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
</span>-
<span style="color: #000000;background-color: #ffdddd">- posix_runtime->file_array_ndx++;
- }
</span><span style="color: #000000;background-color: #ddffdd">+ /* no existing record, assign a new file record from the global array */
+ file = &(posix_runtime->file_runtime_array[posix_runtime->file_array_ndx]);
+ file->file_record = &(posix_runtime->file_record_array[posix_runtime->file_array_ndx]);
+ file->file_record->f_id = file_id;
+ file->file_record->rank = my_rank;
+ file->file_record->counters[POSIX_MEM_ALIGNMENT] = darshan_mem_alignment;
+ file->file_record->counters[POSIX_FILE_ALIGNMENT] = file_alignment;
+
+ /* add new record to file hash table */
+ HASH_ADD(hlink, posix_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
+ posix_runtime->file_array_ndx++;
</span>
if(newname != name)
free(newname);
return(file);
-#endif
}
/* get a POSIX file record for the given file path, and also create a
<span style="color: #aaaaaa">@@ -1768,7 +1748,6 @@ static struct posix_file_runtime* posix_file_by_name_setfd(const char* name, int
</span> /* find file record by name first */
file = posix_file_by_name(name);
-#if 0
if(!file)
return(NULL);
<span style="color: #aaaaaa">@@ -1794,7 +1773,6 @@ static struct posix_file_runtime* posix_file_by_name_setfd(const char* name, int
</span> ref->file = file;
ref->fd = fd;
HASH_ADD(hlink, posix_runtime->fd_hash, fd, sizeof(int), ref);
-#endif
return(file);
}
<span style="color: #aaaaaa">@@ -1807,16 +1785,12 @@ static struct posix_file_runtime* posix_file_by_fd(int fd)
</span> if(!posix_runtime || instrumentation_disabled)
return(NULL);
<span style="color: #000000;background-color: #ffdddd">- return(posix_file_by_name(NULL));
</span>-
-#if 0
/* search hash table for existing file ref for this fd */
HASH_FIND(hlink, posix_runtime->fd_hash, &fd, sizeof(int), ref);
if(ref)
return(ref->file);
return(NULL);
-#endif
}
/* free up reference data structures for the given file descriptor */
<span style="color: #aaaaaa">@@ -1827,7 +1801,6 @@ static void posix_file_close_fd(int fd)
</span> if(!posix_runtime || instrumentation_disabled)
return;
-#if 0
/* search hash table for this fd */
HASH_FIND(hlink, posix_runtime->fd_hash, &fd, sizeof(int), ref);
if(ref)
<span style="color: #aaaaaa">@@ -1836,7 +1809,6 @@ static void posix_file_close_fd(int fd)
</span> HASH_DELETE(hlink, posix_runtime->fd_hash, ref);
free(ref);
}
-#endif
return;
}
</code></pre>
<br>
</li>
<li id='diff-17'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-17'>
<strong>
darshan-util/configure
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/configure
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/configure
</span><span style="color: #aaaaaa">@@ -1,6 +1,6 @@
</span> #! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for darshan-util 3.0.0-pre1.
<span style="color: #000000;background-color: #ddffdd">+# Generated by GNU Autoconf 2.69 for darshan-util 3.0.0-pre2.
</span> #
#
# Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
<span style="color: #aaaaaa">@@ -577,8 +577,8 @@ MAKEFLAGS=
</span> # Identity of this package.
PACKAGE_NAME='darshan-util'
PACKAGE_TARNAME='darshan-util'
-PACKAGE_VERSION='3.0.0-pre1'
-PACKAGE_STRING='darshan-util 3.0.0-pre1'
<span style="color: #000000;background-color: #ddffdd">+PACKAGE_VERSION='3.0.0-pre2'
+PACKAGE_STRING='darshan-util 3.0.0-pre2'
</span> PACKAGE_BUGREPORT=''
PACKAGE_URL=''
<span style="color: #aaaaaa">@@ -1236,7 +1236,7 @@ if test "$ac_init_help" = "long"; then
</span> # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures darshan-util 3.0.0-pre1 to adapt to many kinds of systems.
<span style="color: #000000;background-color: #ddffdd">+\`configure' configures darshan-util 3.0.0-pre2 to adapt to many kinds of systems.
</span>
Usage: $0 [OPTION]... [VAR=VALUE]...
<span style="color: #aaaaaa">@@ -1297,7 +1297,7 @@ fi
</span>
if test -n "$ac_init_help"; then
case $ac_init_help in
<span style="color: #000000;background-color: #ffdddd">- short | recursive ) echo "Configuration of darshan-util 3.0.0-pre1:";;
</span><span style="color: #000000;background-color: #ddffdd">+ short | recursive ) echo "Configuration of darshan-util 3.0.0-pre2:";;
</span> esac
cat <<\_ACEOF
<span style="color: #aaaaaa">@@ -1393,7 +1393,7 @@ fi
</span> test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-darshan-util configure 3.0.0-pre1
<span style="color: #000000;background-color: #ddffdd">+darshan-util configure 3.0.0-pre2
</span> generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
<span style="color: #aaaaaa">@@ -1758,7 +1758,7 @@ cat >config.log <<_ACEOF
</span> This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by darshan-util $as_me 3.0.0-pre1, which was
<span style="color: #000000;background-color: #ddffdd">+It was created by darshan-util $as_me 3.0.0-pre2, which was
</span> generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
<span style="color: #aaaaaa">@@ -4105,7 +4105,7 @@ fi
</span> done
-DARSHAN_UTIL_VERSION="3.0.0-pre1"
<span style="color: #000000;background-color: #ddffdd">+DARSHAN_UTIL_VERSION="3.0.0-pre2"
</span>
<span style="color: #aaaaaa">@@ -4621,7 +4621,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
</span> # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by darshan-util $as_me 3.0.0-pre1, which was
<span style="color: #000000;background-color: #ddffdd">+This file was extended by darshan-util $as_me 3.0.0-pre2, which was
</span> generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
<span style="color: #aaaaaa">@@ -4683,7 +4683,7 @@ _ACEOF
</span> cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-darshan-util config.status 3.0.0-pre1
<span style="color: #000000;background-color: #ddffdd">+darshan-util config.status 3.0.0-pre2
</span> configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"
</code></pre>
<br>
</li>
<li id='diff-18'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-18'>
<strong>
darshan-util/configure.in
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/configure.in
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/configure.in
</span><span style="color: #aaaaaa">@@ -5,7 +5,7 @@ dnl Process this file with autoconf to produce a configure script.
</span> dnl You may need to use autoheader as well if changing any DEFINEs
dnl sanity checks, output header, location of scripts used here
-AC_INIT([darshan-util], [3.0.0-pre1])
<span style="color: #000000;background-color: #ddffdd">+AC_INIT([darshan-util], [3.0.0-pre2])
</span> AC_CONFIG_SRCDIR([darshan-logutils.h])
AC_CONFIG_AUX_DIR(../maint/config)
AC_CONFIG_HEADER(darshan-util-config.h)
</code></pre>
<br>
</li>
<li id='diff-19'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-19'>
<strong>
darshan-util/darshan-convert.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/darshan-convert.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/darshan-convert.c
</span><span style="color: #aaaaaa">@@ -250,7 +250,7 @@ int main(int argc, char **argv)
</span> if(!infile)
return(-1);
<span style="color: #000000;background-color: #ffdddd">- comp_type = bzip2 ? comp_type = DARSHAN_BZIP2_COMP : DARSHAN_ZLIB_COMP;
</span><span style="color: #000000;background-color: #ddffdd">+ comp_type = bzip2 ? DARSHAN_BZIP2_COMP : DARSHAN_ZLIB_COMP;
</span> outfile = darshan_log_create(outfile_name, comp_type, infile->partial_flag);
if(!outfile)
{
</code></pre>
<br>
</li>
<li id='diff-20'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-20'>
<strong>
darshan-util/darshan-job-summary/bin/darshan-job-summary.pl.in
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/darshan-job-summary/bin/darshan-job-summary.pl.in
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/darshan-job-summary/bin/darshan-job-summary.pl.in
</span><span style="color: #aaaaaa">@@ -97,7 +97,6 @@ while($line = <PARSE_OUT>)
</span> $f_save = "";
($junk, $cmdline) = split(':', $line, 2);
<span style="color: #000000;background-color: #ffdddd">- print("PRE: $cmdline\n");
</span> # add escape characters if needed for special characters in
# command line
if ($cmdline =~ /<unknown args>/)
<span style="color: #aaaaaa">@@ -108,7 +107,6 @@ while($line = <PARSE_OUT>)
</span> $cmdline = substr($cmdline, 0, -14);
}
$cmdline = encode('latex', $cmdline) . $f_save;
<span style="color: #000000;background-color: #ffdddd">- print("POST: $cmdline\n");
</span> }
elsif ($line =~ /^# nprocs: /)
{
<span style="color: #aaaaaa">@@ -135,7 +133,7 @@ while($line = <PARSE_OUT>)
</span> ($junk, $version) = split(':', $line, 2);
$version =~ s/^\s+//;
}
<span style="color: #000000;background-color: #ffdddd">- elsif ($line =~ /^# \*WARNING\*: This Darshan log contains incomplete data!/)
</span><span style="color: #000000;background-color: #ddffdd">+ elsif ($line =~ /^# \*WARNING\*: .* contains incomplete data!/)
</span> {
$partial_flag = 1;
}
</code></pre>
<br>
</li>
<li id='diff-21'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-21'>
<strong>
darshan-util/darshan-logutils.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/darshan-logutils.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/darshan-logutils.c
</span><span style="color: #aaaaaa">@@ -31,16 +31,18 @@
</span>
struct darshan_dz_state
{
<span style="color: #000000;background-color: #ffdddd">- /* (libz/bzip2) stream data structure for managing
- * compression and decompression state */
- void *strm;
</span><span style="color: #000000;background-color: #ddffdd">+ /* pointer to arbitrary data structure used for managing
+ * compression/decompression state (e.g., z_stream
+ * structure needed for libz)
+ */
+ void *comp_dat;
</span> /* buffer for staging compressed data to/from log file */
unsigned char *buf;
/* size of staging buffer */
<span style="color: #000000;background-color: #ffdddd">- int size;
</span><span style="color: #000000;background-color: #ddffdd">+ unsigned int size;
</span> /* for reading logs, flag indicating end of log file region */
int eor;
<span style="color: #000000;background-color: #ffdddd">- /* the region we last tried reading/writing */
</span><span style="color: #000000;background-color: #ddffdd">+ /* the region id we last tried reading/writing */
</span> int prev_reg_id;
};
<span style="color: #aaaaaa">@@ -53,8 +55,6 @@ struct darshan_fd_int_state
</span> int64_t pos;
/* flag indicating whether log file was created (and written) */
int creat_flag;
<span style="color: #000000;background-color: #ffdddd">- /* compression type used on log file (libz or bzip2) */
- enum darshan_comp_type comp_type;
</span> /* log file path name */
char logfile_path[PATH_MAX];
/* pointer to exe & mount data in darshan job data structure */
<span style="color: #aaaaaa">@@ -62,7 +62,7 @@ struct darshan_fd_int_state
</span> /* whether previous file operations have failed */
int err;
<span style="color: #000000;background-color: #ffdddd">- /* compression/decompression state */
</span><span style="color: #000000;background-color: #ddffdd">+ /* compression/decompression stream read/write state */
</span> struct darshan_dz_state dz;
};
<span style="color: #aaaaaa">@@ -71,20 +71,26 @@ static int darshan_log_putheader(darshan_fd fd);
</span> static int darshan_log_seek(darshan_fd fd, off_t offset);
static int darshan_log_read(darshan_fd fd, void *buf, int len);
static int darshan_log_write(darshan_fd fd, void *buf, int len);
-static int darshan_log_dzinit(struct darshan_fd_int_state *state);
-static void darshan_log_dzdestroy(struct darshan_fd_int_state *state);
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_dzinit(darshan_fd fd);
+static void darshan_log_dzdestroy(darshan_fd fd);
</span> static int darshan_log_dzread(darshan_fd fd, int region_id, void *buf, int len);
static int darshan_log_dzwrite(darshan_fd fd, int region_id, void *buf, int len);
-static int darshan_log_libz_read(darshan_fd fd, int region_id, void *buf, int len);
-static int darshan_log_libz_write(darshan_fd fd, int region_id, void *buf, int len);
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_libz_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_strm_flag);
+static int darshan_log_libz_write(darshan_fd fd, struct darshan_log_map *map_p,
+ void *buf, int len, int flush_strm_flag);
</span> static int darshan_log_libz_flush(darshan_fd fd, int region_id);
#ifdef HAVE_LIBBZ2
-static int darshan_log_bzip2_read(darshan_fd fd, int region_id, void *buf, int len);
-static int darshan_log_bzip2_write(darshan_fd fd, int region_id, void *buf, int len);
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_bzip2_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_strm_flag);
+static int darshan_log_bzip2_write(darshan_fd fd, struct darshan_log_map *map_p,
+ void *buf, int len, int flush_strm_flag);
</span> static int darshan_log_bzip2_flush(darshan_fd fd, int region_id);
#endif
static int darshan_log_dzload(darshan_fd fd, struct darshan_log_map map);
static int darshan_log_dzunload(darshan_fd fd, struct darshan_log_map *map_p);
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_noz_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_strm_flag);
</span>
/* each module's implementation of the darshan logutil functions */
#define X(a, b, c) c,
<span style="color: #aaaaaa">@@ -140,7 +146,7 @@ darshan_fd darshan_log_open(const char *name)
</span> }
/* initialize compression data structures */
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_dzinit(tmp_fd->state);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_dzinit(tmp_fd);
</span> if(ret < 0)
{
fprintf(stderr, "Error: failed to initialize decompression data structures.\n");
<span style="color: #aaaaaa">@@ -177,6 +183,7 @@ darshan_fd darshan_log_create(const char *name, enum darshan_comp_type comp_type
</span> return(NULL);
}
memset(tmp_fd->state, 0, sizeof(struct darshan_fd_int_state));
<span style="color: #000000;background-color: #ddffdd">+ tmp_fd->comp_type = comp_type;
</span>
/* create the log for writing, making sure to not overwrite existing log */
tmp_fd->state->fildes = creat(name, 0400);
<span style="color: #aaaaaa">@@ -188,7 +195,6 @@ darshan_fd darshan_log_create(const char *name, enum darshan_comp_type comp_type
</span> return(NULL);
}
tmp_fd->state->creat_flag = 1;
<span style="color: #000000;background-color: #ffdddd">- tmp_fd->state->comp_type = comp_type;
</span> tmp_fd->partial_flag = partial_flag;
strncpy(tmp_fd->state->logfile_path, name, PATH_MAX);
<span style="color: #aaaaaa">@@ -208,7 +214,7 @@ darshan_fd darshan_log_create(const char *name, enum darshan_comp_type comp_type
</span> }
/* initialize compression data structures */
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_dzinit(tmp_fd->state);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_dzinit(tmp_fd);
</span> if(ret < 0)
{
fprintf(stderr, "Error: failed to initialize compression data structures.\n");
<span style="color: #aaaaaa">@@ -240,7 +246,7 @@ int darshan_log_getjob(darshan_fd fd, struct darshan_job *job)
</span>
/* read the compressed job data from the log file */
ret = darshan_log_dzread(fd, DARSHAN_JOB_REGION_ID, job_buf, job_buf_sz);
<span style="color: #000000;background-color: #ffdddd">- if(ret <= sizeof(*job))
</span><span style="color: #000000;background-color: #ddffdd">+ if(ret <= (int)sizeof(*job))
</span> {
fprintf(stderr, "Error: failed to read darshan log file job data.\n");
return(-1);
<span style="color: #aaaaaa">@@ -774,7 +780,7 @@ void darshan_log_close(darshan_fd fd)
</span> if(state->creat_flag)
{
/* flush the last region of the log to file */
<span style="color: #000000;background-color: #ffdddd">- switch(state->comp_type)
</span><span style="color: #000000;background-color: #ddffdd">+ switch(fd->comp_type)
</span> {
case DARSHAN_ZLIB_COMP:
ret = darshan_log_libz_flush(fd, state->dz.prev_reg_id);
<span style="color: #aaaaaa">@@ -812,7 +818,7 @@ void darshan_log_close(darshan_fd fd)
</span> unlink(state->logfile_path);
}
<span style="color: #000000;background-color: #ffdddd">- darshan_log_dzdestroy(state);
</span><span style="color: #000000;background-color: #ddffdd">+ darshan_log_dzdestroy(fd);
</span> if(state->exe_mnt_data)
free(state->exe_mnt_data);
free(state);
<span style="color: #aaaaaa">@@ -830,7 +836,6 @@ void darshan_log_close(darshan_fd fd)
</span> */
static int darshan_log_getheader(darshan_fd fd)
{
<span style="color: #000000;background-color: #ffdddd">- struct darshan_fd_int_state *state = fd->state;
</span> struct darshan_header header;
int i;
int ret;
<span style="color: #aaaaaa">@@ -844,7 +849,7 @@ static int darshan_log_getheader(darshan_fd fd)
</span>
/* read uncompressed header from log file */
ret = darshan_log_read(fd, &header, sizeof(header));
<span style="color: #000000;background-color: #ffdddd">- if(ret != sizeof(header))
</span><span style="color: #000000;background-color: #ddffdd">+ if(ret != (int)sizeof(header))
</span> {
fprintf(stderr, "Error: failed to read darshan log file header.\n");
return(-1);
<span style="color: #aaaaaa">@@ -883,15 +888,42 @@ static int darshan_log_getheader(darshan_fd fd)
</span> }
}
<span style="color: #000000;background-color: #ffdddd">- state->comp_type = header.comp_type;
</span><span style="color: #000000;background-color: #ddffdd">+ fd->comp_type = header.comp_type;
</span> fd->partial_flag = header.partial_flag;
/* save the mapping of data within log file to this file descriptor */
<span style="color: #000000;background-color: #ffdddd">- fd->job_map.off = sizeof(struct darshan_header);
- fd->job_map.len = header.rec_map.off - fd->job_map.off;
</span> memcpy(&fd->rec_map, &(header.rec_map), sizeof(struct darshan_log_map));
memcpy(&fd->mod_map, &(header.mod_map), DARSHAN_MAX_MODS * sizeof(struct darshan_log_map));
<span style="color: #000000;background-color: #ddffdd">+ /* there may be nothing following the job data, so safety check map */
+ fd->job_map.off = sizeof(struct darshan_header);
+ if(fd->rec_map.off == 0)
+ {
+ for(i = 0; i < DARSHAN_MAX_MODS; i++)
+ {
+ if(fd->mod_map[i].off != 0)
+ {
+ fd->job_map.len = fd->mod_map[i].off - fd->job_map.off;
+ break;
+ }
+ }
+
+ if(fd->job_map.len == 0)
+ {
+ struct stat sbuf;
+ if(fstat(fd->state->fildes, &sbuf) != 0)
+ {
+ fprintf(stderr, "Error: unable to stat darshan log file.\n");
+ return(-1);
+ }
+ fd->job_map.len = sbuf.st_size - fd->job_map.off;
+ }
+ }
+ else
+ {
+ fd->job_map.len = fd->rec_map.off - fd->job_map.off;
+ }
+
</span> return(0);
}
<span style="color: #aaaaaa">@@ -901,7 +933,6 @@ static int darshan_log_getheader(darshan_fd fd)
</span> */
static int darshan_log_putheader(darshan_fd fd)
{
<span style="color: #000000;background-color: #ffdddd">- struct darshan_fd_int_state *state = fd->state;
</span> struct darshan_header header;
int ret;
<span style="color: #aaaaaa">@@ -915,7 +946,7 @@ static int darshan_log_putheader(darshan_fd fd)
</span> memset(&header, 0, sizeof(header));
strcpy(header.version_string, DARSHAN_LOG_VERSION);
header.magic_nr = DARSHAN_MAGIC_NR;
<span style="color: #000000;background-color: #ffdddd">- header.comp_type = state->comp_type;
</span><span style="color: #000000;background-color: #ddffdd">+ header.comp_type = fd->comp_type;
</span> header.partial_flag = fd->partial_flag;
/* copy the mapping information to the header */
<span style="color: #aaaaaa">@@ -924,7 +955,7 @@ static int darshan_log_putheader(darshan_fd fd)
</span>
/* write header to file */
ret = darshan_log_write(fd, &header, sizeof(header));
<span style="color: #000000;background-color: #ffdddd">- if(ret != sizeof(header))
</span><span style="color: #000000;background-color: #ddffdd">+ if(ret != (int)sizeof(header))
</span> {
fprintf(stderr, "Error: failed to write Darshan log file header.\n");
return(-1);
<span style="color: #aaaaaa">@@ -959,13 +990,20 @@ static int darshan_log_read(darshan_fd fd, void* buf, int len)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
<span style="color: #000000;background-color: #ddffdd">+ unsigned int read_so_far = 0;
</span>
<span style="color: #000000;background-color: #ffdddd">- /* read data from the log file using the given map */
- ret = read(state->fildes, buf, len);
- if(ret > 0)
- state->pos += ret;
</span><span style="color: #000000;background-color: #ddffdd">+ do
+ {
+ ret = read(state->fildes, buf + read_so_far, len - read_so_far);
+ if(ret <= 0)
+ break;
+ read_so_far += ret;
+ } while(read_so_far < len);
+ if(ret < 0)
+ return(-1);
</span>
<span style="color: #000000;background-color: #ffdddd">- return(ret);
</span><span style="color: #000000;background-color: #ddffdd">+ state->pos += read_so_far;
+ return(read_so_far);
</span> }
/* return amount written on success, -1 on failure.
<span style="color: #aaaaaa">@@ -974,26 +1012,37 @@ static int darshan_log_write(darshan_fd fd, void* buf, int len)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
<span style="color: #000000;background-color: #ddffdd">+ unsigned int wrote_so_far = 0;
</span>
<span style="color: #000000;background-color: #ffdddd">- ret = write(state->fildes, buf, len);
- if(ret > 0)
- state->pos += ret;
</span><span style="color: #000000;background-color: #ddffdd">+ do
+ {
+ ret = write(state->fildes, buf + wrote_so_far, len - wrote_so_far);
+ if(ret <= 0)
+ break;
+ wrote_so_far += ret;
+ } while(wrote_so_far < len);
+ if(ret < 0)
+ return(-1);
</span>
<span style="color: #000000;background-color: #ffdddd">- return(ret);
</span><span style="color: #000000;background-color: #ddffdd">+ state->pos += wrote_so_far;
+ return(wrote_so_far);
</span> }
-static int darshan_log_dzinit(struct darshan_fd_int_state *state)
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_dzinit(darshan_fd fd)
</span> {
<span style="color: #000000;background-color: #ddffdd">+ struct darshan_fd_int_state *state = fd->state;
</span> int ret;
<span style="color: #000000;background-color: #ffdddd">- /* initialize buffers for staging compressed data to/from log file */
</span><span style="color: #000000;background-color: #ddffdd">+ /* initialize buffers for staging compressed data
+ * to/from log file
+ */
</span> state->dz.buf = malloc(DARSHAN_DEF_COMP_BUF_SZ);
if(state->dz.buf == NULL)
return(-1);
-
<span style="color: #000000;background-color: #ddffdd">+ state->dz.size = 0;
</span> state->dz.prev_reg_id = DARSHAN_HEADER_REGION_ID;
<span style="color: #000000;background-color: #ffdddd">- switch(state->comp_type)
</span><span style="color: #000000;background-color: #ddffdd">+ switch(fd->comp_type)
</span> {
case DARSHAN_ZLIB_COMP:
{
<span style="color: #aaaaaa">@@ -1028,7 +1077,7 @@ static int darshan_log_dzinit(struct darshan_fd_int_state *state)
</span> free(state->dz.buf);
return(-1);
}
<span style="color: #000000;background-color: #ffdddd">- state->dz.strm = tmp_zstrm;
</span><span style="color: #000000;background-color: #ddffdd">+ state->dz.comp_dat = tmp_zstrm;
</span> break;
}
#ifdef HAVE_LIBBZ2
<span style="color: #aaaaaa">@@ -1044,9 +1093,9 @@ static int darshan_log_dzinit(struct darshan_fd_int_state *state)
</span> tmp_bzstrm->bzfree = NULL;
tmp_bzstrm->opaque = NULL;
tmp_bzstrm->avail_in = 0;
<span style="color: #000000;background-color: #ffdddd">- tmp_bzstrm->next_in = Z_NULL;
</span><span style="color: #000000;background-color: #ddffdd">+ tmp_bzstrm->next_in = NULL;
</span>
<span style="color: #000000;background-color: #ffdddd">- if(state->creat_flag)
</span><span style="color: #000000;background-color: #ddffdd">+ if(!(state->creat_flag))
</span> {
/* read only file, init decompress algorithm */
ret = BZ2_bzDecompressInit(tmp_bzstrm, 1, 0);
<span style="color: #aaaaaa">@@ -1064,10 +1113,18 @@ static int darshan_log_dzinit(struct darshan_fd_int_state *state)
</span> free(state->dz.buf);
return(-1);
}
<span style="color: #000000;background-color: #ffdddd">- state->dz.strm = tmp_bzstrm;
</span><span style="color: #000000;background-color: #ddffdd">+ state->dz.comp_dat = tmp_bzstrm;
</span> break;
}
#endif
<span style="color: #000000;background-color: #ddffdd">+ case DARSHAN_NO_COMP:
+ {
+ /* we just track an offset into the staging buffers for no_comp */
+ int *buf_off = malloc(sizeof(int));
+ *buf_off = 0;
+ state->dz.comp_dat = buf_off;
+ break;
+ }
</span> default:
fprintf(stderr, "Error: invalid compression type.\n");
return(-1);
<span style="color: #aaaaaa">@@ -1076,30 +1133,36 @@ static int darshan_log_dzinit(struct darshan_fd_int_state *state)
</span> return(0);
}
-static void darshan_log_dzdestroy(struct darshan_fd_int_state *state)
<span style="color: #000000;background-color: #ddffdd">+static void darshan_log_dzdestroy(darshan_fd fd)
</span> {
<span style="color: #000000;background-color: #ffdddd">- switch(state->comp_type)
</span><span style="color: #000000;background-color: #ddffdd">+ struct darshan_fd_int_state *state = fd->state;
+
+ switch(fd->comp_type)
</span> {
case DARSHAN_ZLIB_COMP:
if(!(state->creat_flag))
<span style="color: #000000;background-color: #ffdddd">- inflateEnd(state->dz.strm);
</span><span style="color: #000000;background-color: #ddffdd">+ inflateEnd((z_stream *)state->dz.comp_dat);
</span> else
<span style="color: #000000;background-color: #ffdddd">- deflateEnd(state->dz.strm);
- free(state->dz.strm);
</span><span style="color: #000000;background-color: #ddffdd">+ deflateEnd((z_stream *)state->dz.comp_dat);
</span> break;
#ifdef HAVE_LIBBZ2
case DARSHAN_BZIP2_COMP:
if(!(state->creat_flag))
<span style="color: #000000;background-color: #ffdddd">- BZ2_bzDecompressEnd(state->dz.strm);
</span><span style="color: #000000;background-color: #ddffdd">+ BZ2_bzDecompressEnd((bz_stream *)state->dz.comp_dat);
</span> else
<span style="color: #000000;background-color: #ffdddd">- BZ2_bzCompressEnd(state->dz.strm);
- free(state->dz.strm);
</span><span style="color: #000000;background-color: #ddffdd">+ BZ2_bzCompressEnd((bz_stream *)state->dz.comp_dat);
</span> break;
#endif
<span style="color: #000000;background-color: #ddffdd">+ case DARSHAN_NO_COMP:
+ {
+ /* do nothing */
+ break;
+ }
</span> default:
fprintf(stderr, "Error: invalid compression type.\n");
}
<span style="color: #000000;background-color: #ddffdd">+ free(state->dz.comp_dat);
</span> free(state->dz.buf);
return;
}
<span style="color: #aaaaaa">@@ -1107,74 +1170,111 @@ static void darshan_log_dzdestroy(struct darshan_fd_int_state *state)
</span> static int darshan_log_dzread(darshan_fd fd, int region_id, void *buf, int len)
{
struct darshan_fd_int_state *state = fd->state;
<span style="color: #000000;background-color: #ddffdd">+ struct darshan_log_map map;
+ int reset_strm_flag = 0;
</span> int ret;
<span style="color: #000000;background-color: #ffdddd">- switch(state->comp_type)
</span><span style="color: #000000;background-color: #ddffdd">+ /* if new log region, we reload buffers and clear eor flag */
+ if(region_id != state->dz.prev_reg_id)
+ {
+ state->dz.eor = 0;
+ reset_strm_flag = 1; /* reset libz/bzip2 streams */
+ }
+
+ if(region_id == DARSHAN_JOB_REGION_ID)
+ map = fd->job_map;
+ else if(region_id == DARSHAN_REC_MAP_REGION_ID)
+ map = fd->rec_map;
+ else
+ map = fd->mod_map[region_id];
+
+ switch(fd->comp_type)
</span> {
case DARSHAN_ZLIB_COMP:
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_libz_read(fd, region_id, buf, len);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_libz_read(fd, map, buf, len, reset_strm_flag);
</span> break;
#ifdef HAVE_LIBBZ2
case DARSHAN_BZIP2_COMP:
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_bzip2_read(fd, region_id, buf, len);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_bzip2_read(fd, map, buf, len, reset_strm_flag);
</span> break;
#endif
<span style="color: #000000;background-color: #ddffdd">+ case DARSHAN_NO_COMP:
+ {
+ ret = darshan_log_noz_read(fd, map, buf, len, reset_strm_flag);
+ break;
+ }
</span> default:
fprintf(stderr, "Error: invalid compression type.\n");
return(-1);
}
<span style="color: #000000;background-color: #ddffdd">+ state->dz.prev_reg_id = region_id;
</span> return(ret);
}
static int darshan_log_dzwrite(darshan_fd fd, int region_id, void *buf, int len)
{
struct darshan_fd_int_state *state = fd->state;
<span style="color: #000000;background-color: #ddffdd">+ struct darshan_log_map *map_p;
+ int flush_strm_flag = 0;
</span> int ret;
<span style="color: #000000;background-color: #ffdddd">- switch(state->comp_type)
</span><span style="color: #000000;background-color: #ddffdd">+ /* if new log region, finish prev region's zstream and flush to log file */
+ if(region_id != state->dz.prev_reg_id)
+ {
+ /* error out if the region we are writing to precedes the previous
+ * region we wrote -- we shouldn't be moving backwards in the log
+ */
+ if(region_id < state->dz.prev_reg_id)
+ return(-1);
+
+ if(state->dz.prev_reg_id != DARSHAN_HEADER_REGION_ID)
+ flush_strm_flag = 1;
+ }
+
+ if(region_id == DARSHAN_JOB_REGION_ID)
+ map_p = &(fd->job_map);
+ else if(region_id == DARSHAN_REC_MAP_REGION_ID)
+ map_p = &(fd->rec_map);
+ else
+ map_p = &(fd->mod_map[region_id]);
+
+ switch(fd->comp_type)
</span> {
case DARSHAN_ZLIB_COMP:
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_libz_write(fd, region_id, buf, len);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_libz_write(fd, map_p, buf, len, flush_strm_flag);
</span> break;
#ifdef HAVE_LIBBZ2
case DARSHAN_BZIP2_COMP:
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_bzip2_write(fd, region_id, buf, len);
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_bzip2_write(fd, map_p, buf, len, flush_strm_flag);
</span> break;
#endif
<span style="color: #000000;background-color: #ddffdd">+ case DARSHAN_NO_COMP:
+ fprintf(stderr,
+ "Error: uncompressed writing of log files is not supported.\n");
+ return(-1);
</span> default:
fprintf(stderr, "Error: invalid compression type.\n");
return(-1);
}
<span style="color: #000000;background-color: #ddffdd">+ state->dz.prev_reg_id = region_id;
</span> return(ret);
}
-static int darshan_log_libz_read(darshan_fd fd, int region_id, void *buf, int len)
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_libz_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_stream_flag)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
int total_bytes = 0;
int tmp_out_bytes;
<span style="color: #000000;background-color: #ffdddd">- struct darshan_log_map map;
- z_stream *z_strmp = (z_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ z_stream *z_strmp = (z_stream *)state->dz.comp_dat;
</span>
assert(z_strmp);
<span style="color: #000000;background-color: #ffdddd">- /* if new log region, we reload buffers and clear eor flag */
- if(region_id != state->dz.prev_reg_id)
- {
</span><span style="color: #000000;background-color: #ddffdd">+ if(reset_stream_flag)
</span> z_strmp->avail_in = 0;
<span style="color: #000000;background-color: #ffdddd">- state->dz.eor = 0;
- state->dz.prev_reg_id = region_id;
- }
</span>-
<span style="color: #000000;background-color: #ffdddd">- if(region_id == DARSHAN_JOB_REGION_ID)
- map = fd->job_map;
- else if(region_id == DARSHAN_REC_MAP_REGION_ID)
- map = fd->rec_map;
- else
- map = fd->mod_map[region_id];
</span>
z_strmp->avail_out = len;
z_strmp->next_out = buf;
<span style="color: #aaaaaa">@@ -1223,44 +1323,26 @@ static int darshan_log_libz_read(darshan_fd fd, int region_id, void *buf, int le
</span> return(total_bytes);
}
-static int darshan_log_libz_write(darshan_fd fd, int region_id, void *buf, int len)
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_libz_write(darshan_fd fd, struct darshan_log_map *map_p,
+ void *buf, int len, int flush_strm_flag)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
int total_bytes = 0;
int tmp_in_bytes;
int tmp_out_bytes;
<span style="color: #000000;background-color: #ffdddd">- struct darshan_log_map *map_p;
- z_stream *z_strmp = (z_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ z_stream *z_strmp = (z_stream *)state->dz.comp_dat;
</span>
assert(z_strmp);
<span style="color: #000000;background-color: #ffdddd">- /* if new log region, finish prev region's zstream and flush to log file */
- if(region_id != state->dz.prev_reg_id)
</span><span style="color: #000000;background-color: #ddffdd">+ /* flush compressed output buffer if we are moving to a new log region */
+ if(flush_strm_flag)
</span> {
<span style="color: #000000;background-color: #ffdddd">- /* error out if the region we are writing to precedes the previous
- * region we wrote -- we shouldn't be moving backwards in the log
- */
- if(region_id < state->dz.prev_reg_id)
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_libz_flush(fd, state->dz.prev_reg_id);
+ if(ret < 0)
</span> return(-1);
-
<span style="color: #000000;background-color: #ffdddd">- if(state->dz.prev_reg_id != DARSHAN_HEADER_REGION_ID)
- {
- ret = darshan_log_libz_flush(fd, state->dz.prev_reg_id);
- if(ret < 0)
- return(-1);
- }
</span>-
<span style="color: #000000;background-color: #ffdddd">- state->dz.prev_reg_id = region_id;
</span> }
<span style="color: #000000;background-color: #ffdddd">- if(region_id == DARSHAN_JOB_REGION_ID)
- map_p = &(fd->job_map);
- else if(region_id == DARSHAN_REC_MAP_REGION_ID)
- map_p = &(fd->rec_map);
- else
- map_p = &(fd->mod_map[region_id]);
</span>-
z_strmp->avail_in = len;
z_strmp->next_in = buf;
<span style="color: #aaaaaa">@@ -1301,7 +1383,7 @@ static int darshan_log_libz_flush(darshan_fd fd, int region_id)
</span> int ret;
int tmp_out_bytes;
struct darshan_log_map *map_p;
<span style="color: #000000;background-color: #ffdddd">- z_stream *z_strmp = (z_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ z_stream *z_strmp = (z_stream *)state->dz.comp_dat;
</span>
assert(z_strmp);
<span style="color: #aaaaaa">@@ -1342,32 +1424,19 @@ static int darshan_log_libz_flush(darshan_fd fd, int region_id)
</span> }
#ifdef HAVE_LIBBZ2
-
-static int darshan_log_bzip2_read(darshan_fd fd, int region_id, void *buf, int len)
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_bzip2_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_strm_flag)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
int total_bytes = 0;
int tmp_out_bytes;
<span style="color: #000000;background-color: #ffdddd">- struct darshan_log_map map;
- bz_stream *bz_strmp = (bz_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ bz_stream *bz_strmp = (bz_stream *)state->dz.comp_dat;
</span>
assert(bz_strmp);
<span style="color: #000000;background-color: #ffdddd">- /* if new log region, we reload buffers and clear eor flag */
- if(region_id != state->dz.prev_reg_id)
- {
</span><span style="color: #000000;background-color: #ddffdd">+ if(reset_strm_flag)
</span> bz_strmp->avail_in = 0;
<span style="color: #000000;background-color: #ffdddd">- state->dz.eor = 0;
- state->dz.prev_reg_id = region_id;
- }
</span>-
<span style="color: #000000;background-color: #ffdddd">- if(region_id == DARSHAN_JOB_REGION_ID)
- map = fd->job_map;
- else if(region_id == DARSHAN_REC_MAP_REGION_ID)
- map = fd->rec_map;
- else
- map = fd->mod_map[region_id];
</span>
bz_strmp->avail_out = len;
bz_strmp->next_out = buf;
<span style="color: #aaaaaa">@@ -1419,44 +1488,26 @@ static int darshan_log_bzip2_read(darshan_fd fd, int region_id, void *buf, int l
</span> return(total_bytes);
}
-static int darshan_log_bzip2_write(darshan_fd fd, int region_id, void *buf, int len)
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_bzip2_write(darshan_fd fd, struct darshan_log_map *map_p,
+ void *buf, int len, int flush_strm_flag)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
int total_bytes = 0;
int tmp_in_bytes;
int tmp_out_bytes;
<span style="color: #000000;background-color: #ffdddd">- struct darshan_log_map *map_p;
- bz_stream *bz_strmp = (bz_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ bz_stream *bz_strmp = (bz_stream *)state->dz.comp_dat;
</span>
assert(bz_strmp);
<span style="color: #000000;background-color: #ffdddd">- /* if new log region, finish prev region's zstream and flush to log file */
- if(region_id != state->dz.prev_reg_id)
</span><span style="color: #000000;background-color: #ddffdd">+ /* flush compressed output buffer if we are moving to a new log region */
+ if(flush_strm_flag)
</span> {
<span style="color: #000000;background-color: #ffdddd">- /* error out if the region we are writing to precedes the previous
- * region we wrote -- we shouldn't be moving backwards in the log
- */
- if(region_id < state->dz.prev_reg_id)
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_bzip2_flush(fd, state->dz.prev_reg_id);
+ if(ret < 0)
</span> return(-1);
-
<span style="color: #000000;background-color: #ffdddd">- if(state->dz.prev_reg_id != DARSHAN_HEADER_REGION_ID)
- {
- ret = darshan_log_bzip2_flush(fd, state->dz.prev_reg_id);
- if(ret < 0)
- return(-1);
- }
</span>-
<span style="color: #000000;background-color: #ffdddd">- state->dz.prev_reg_id = region_id;
</span> }
<span style="color: #000000;background-color: #ffdddd">- if(region_id == DARSHAN_JOB_REGION_ID)
- map_p = &(fd->job_map);
- else if(region_id == DARSHAN_REC_MAP_REGION_ID)
- map_p = &(fd->rec_map);
- else
- map_p = &(fd->mod_map[region_id]);
</span>-
bz_strmp->avail_in = len;
bz_strmp->next_in = buf;
<span style="color: #aaaaaa">@@ -1497,7 +1548,7 @@ static int darshan_log_bzip2_flush(darshan_fd fd, int region_id)
</span> int ret;
int tmp_out_bytes;
struct darshan_log_map *map_p;
<span style="color: #000000;background-color: #ffdddd">- bz_stream *bz_strmp = (bz_stream *)state->dz.strm;
</span><span style="color: #000000;background-color: #ddffdd">+ bz_stream *bz_strmp = (bz_stream *)state->dz.comp_dat;
</span>
assert(bz_strmp);
<span style="color: #aaaaaa">@@ -1532,22 +1583,65 @@ static int darshan_log_bzip2_flush(darshan_fd fd, int region_id)
</span> bz_strmp->next_out = (char *)state->dz.buf;
}
} while (ret != BZ_STREAM_END);
-
BZ2_bzCompressEnd(bz_strmp);
BZ2_bzCompressInit(bz_strmp, 9, 1, 30);
return(0);
}
-
#endif
<span style="color: #000000;background-color: #ddffdd">+static int darshan_log_noz_read(darshan_fd fd, struct darshan_log_map map,
+ void *buf, int len, int reset_strm_flag)
+{
+ struct darshan_fd_int_state *state = fd->state;
+ int ret;
+ int total_bytes = 0;
+ int cp_size;
+ int *buf_off = (int *)state->dz.comp_dat;
+
+ if(reset_strm_flag)
+ *buf_off = state->dz.size;
+
+ /* we just decompress until the output buffer is full, assuming there
+ * is enough compressed data in file to satisfy the request size.
+ */
+ while(total_bytes < len)
+ {
+ /* check if we need more compressed data */
+ if(*buf_off == state->dz.size)
+ {
+ /* if the eor flag is set, clear it and return -- future
+ * reads of this log region will restart at the beginning
+ */
+ if(state->dz.eor)
+ {
+ state->dz.eor = 0;
+ break;
+ }
+
+ /* read more data from input file */
+ ret = darshan_log_dzload(fd, map);
+ if(ret < 0)
+ return(-1);
+ assert(state->dz.size > 0);
+ }
+
+ cp_size = (len > (state->dz.size - *buf_off)) ?
+ state->dz.size - *buf_off : len;
+ memcpy(buf, state->dz.buf + *buf_off, cp_size);
+ total_bytes += cp_size;
+ *buf_off += cp_size;
+ }
+
+ return(total_bytes);
+}
+
</span> static int darshan_log_dzload(darshan_fd fd, struct darshan_log_map map)
{
struct darshan_fd_int_state *state = fd->state;
int ret;
unsigned int remaining;
unsigned int read_size;
<span style="color: #000000;background-color: #ffdddd">- unsigned int read_so_far = 0;
</span>
state->dz.size = 0;
<span style="color: #aaaaaa">@@ -1566,24 +1660,18 @@ static int darshan_log_dzload(darshan_fd fd, struct darshan_log_map map)
</span> remaining = (map.off + map.len) - state->pos;
read_size = (remaining > DARSHAN_DEF_COMP_BUF_SZ) ?
DARSHAN_DEF_COMP_BUF_SZ : remaining;
<span style="color: #000000;background-color: #ffdddd">- do
- {
- ret = darshan_log_read(fd, state->dz.buf + read_so_far,
- read_size - read_so_far);
- if(ret <= 0)
- break;
- read_so_far += ret;
- } while(read_so_far < read_size);
- if(ret < 0)
</span><span style="color: #000000;background-color: #ddffdd">+
+ ret = darshan_log_read(fd, state->dz.buf, read_size);
+ if(ret < (int)read_size)
</span> {
fprintf(stderr, "Error: unable to read compressed data from file.\n");
return(-1);
}
<span style="color: #000000;background-color: #ffdddd">- if((read_size == remaining) || (ret == 0))
</span><span style="color: #000000;background-color: #ddffdd">+
+ if(ret == (int)remaining)
</span> {
state->dz.eor = 1;
}
-
state->dz.size = read_size;
return(0);
}
<span style="color: #aaaaaa">@@ -1592,24 +1680,18 @@ static int darshan_log_dzunload(darshan_fd fd, struct darshan_log_map *map_p)
</span> {
struct darshan_fd_int_state *state = fd->state;
int ret;
<span style="color: #000000;background-color: #ffdddd">- unsigned int write_so_far = 0;
</span>
/* initialize map structure for this log region */
if(map_p->off == 0)
map_p->off = state->pos;
/* write more compressed data from staging buffer to file */
<span style="color: #000000;background-color: #ffdddd">- do
</span><span style="color: #000000;background-color: #ddffdd">+ ret = darshan_log_write(fd, state->dz.buf, state->dz.size);
+ if(ret < (int)state->dz.size)
</span> {
<span style="color: #000000;background-color: #ffdddd">- ret = darshan_log_write(fd, state->dz.buf + write_so_far,
- state->dz.size - write_so_far);
- if(ret <= 0)
- {
- fprintf(stderr, "Error: unable to write compressed data to file.\n");
- return(-1);
- }
- write_so_far += ret;
- } while(write_so_far < state->dz.size);
</span><span style="color: #000000;background-color: #ddffdd">+ fprintf(stderr, "Error: unable to write compressed data to file.\n");
+ return(-1);
+ }
</span>
map_p->len += state->dz.size;
state->dz.size = 0;
</code></pre>
<br>
</li>
<li id='diff-22'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-22'>
<strong>
darshan-util/darshan-logutils.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/darshan-logutils.h
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/darshan-logutils.h
</span><span style="color: #aaaaaa">@@ -29,6 +29,8 @@ struct darshan_fd_s
</span> int swap_flag;
/* flag indicating whether a log file contains partial data */
int partial_flag;
<span style="color: #000000;background-color: #ddffdd">+ /* compression type used on log file */
+ enum darshan_comp_type comp_type;
</span> /* log file offset/length maps for each log file region */
struct darshan_log_map job_map;
struct darshan_log_map rec_map;
</code></pre>
<br>
</li>
<li id='diff-23'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-23'>
<strong>
darshan-util/darshan-parser.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/darshan-parser.c
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/darshan-parser.c
</span><span style="color: #aaaaaa">@@ -199,6 +199,7 @@ int main(int argc, char **argv)
</span> int mask;
int i, j;
char *filename;
<span style="color: #000000;background-color: #ddffdd">+ char *comp_str;
</span> char tmp_string[4096] = {0};
darshan_fd fd;
struct darshan_job job;
<span style="color: #aaaaaa">@@ -263,8 +264,18 @@ int main(int argc, char **argv)
</span> return(-1);
}
<span style="color: #000000;background-color: #ddffdd">+ if(fd->comp_type == DARSHAN_ZLIB_COMP)
+ comp_str = "ZLIB";
+ else if (fd->comp_type == DARSHAN_BZIP2_COMP)
+ comp_str = "BZIP2";
+ else if (fd->comp_type == DARSHAN_NO_COMP)
+ comp_str = "NONE";
+ else
+ comp_str = "UNKNOWN";
+
</span> /* print job summary */
printf("# darshan log version: %s\n", fd->version);
<span style="color: #000000;background-color: #ddffdd">+ printf("# compression method: %s\n", comp_str);
</span> printf("# exe: %s\n", tmp_string);
printf("# uid: %" PRId64 "\n", job.uid);
printf("# jobid: %" PRId64 "\n", job.jobid);
<span style="color: #aaaaaa">@@ -322,11 +333,6 @@ int main(int argc, char **argv)
</span> }
/* warn user if this log file is incomplete */
<span style="color: #000000;background-color: #ffdddd">- if(fd->partial_flag)
- printf("\n# *WARNING*: This Darshan log contains incomplete data!\n"
- "# This happens when an application creates\n"
- "# more records than Darshan can track.\n");
</span>-
pdata.rank_cumul_io_time = malloc(sizeof(double)*job.nprocs);
pdata.rank_cumul_md_time = malloc(sizeof(double)*job.nprocs);
if (!pdata.rank_cumul_io_time || !pdata.rank_cumul_md_time)
<span style="color: #aaaaaa">@@ -370,6 +376,13 @@ int main(int argc, char **argv)
</span> printf("# %s module data\n", darshan_module_names[i]);
printf("# *******************************************************\n");
<span style="color: #000000;background-color: #ddffdd">+ /* print warning if this module only stored partial data */
+ if(DARSHAN_MOD_FLAG_ISSET(fd->partial_flag, i))
+ printf("\n# *WARNING*: The %s module contains incomplete data!\n"
+ "# This happens when a module runs out of\n"
+ "# memory to store new record data.\n",
+ darshan_module_names[i]);
+
</span> if(mask & OPTION_BASE)
{
/* TODO: does each module print header of what each counter means??? */
</code></pre>
<br>
</li>
<li id='diff-24'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2#diff-24'>
<strong>
doc/darshan-modularization.txt
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/doc/darshan-modularization.txt
</span><span style="color: #000000;background-color: #ddffdd">+++ b/doc/darshan-modularization.txt
</span><span style="color: #aaaaaa">@@ -224,9 +224,10 @@ a module developer must provide to interface with the darshan-core runtime.
</span> * _my_rank_ is a pointer to an integer to store the calling process's application MPI rank in
* _mod_mem_limit_ is a pointer to an integer which will store the amount of memory Darshan
-allows this module to use at runtime. Currently, `darshan-core` will hardcode this value to 2 MiB,
-but in the future this may be changed to optimize Darshan's memory footprint. Note that Darshan
-does not allocate any memory for modules, it just informs a module how much memory it can use.
<span style="color: #000000;background-color: #ddffdd">+allows this module to use at runtime. Darshan's default module memory limit is currently set to
+2 MiB, but the user can choose a different value at configure time (using the `--with-mod-mem`
+configure option) or at runtime (using the DARSHAN_MODMEM environment variable). Note that Darshan
+does not allocate any memory for modules; it just informs a module how much memory it can use.
</span>
* _sys_mem_alignment_ is a pointer to an integer which will store the system memory alignment value
Darshan was configured with. This parameter may be set to `NULL` if a module is not concerned with the
<span style="color: #aaaaaa">@@ -248,8 +249,9 @@ any I/O data.
</span> void darshan_core_register_record(
void *name,
int len,
<span style="color: #000000;background-color: #ffdddd">- int printable_flag,
</span> darshan_module_id mod_id,
<span style="color: #000000;background-color: #ddffdd">+ int printable_flag,
+ int mod_limit_flag,
</span> darshan_record_id *rec_id,
int *file_alignment);
<span style="color: #aaaaaa">@@ -269,9 +271,13 @@ been previously registered.
</span> string length, but for nonprintable record names (e.g., an integer object identifier), this
is the size of the record name type.
<span style="color: #000000;background-color: #ddffdd">+* _mod_id_ is the identifier for the module attempting to register this record.
+
</span> * _printable_flag_ indicates whether the input record name is a printable ASCII string.
-* _mod_id_ is the identifier for the module attempting to register this record.
<span style="color: #000000;background-color: #ddffdd">+* _mod_limit_flag_ indicates whether the calling module is out of memory to instrument new
+records or not. If this flag is set, darshan-core will not create new records and instead just
+search existing records for one corresponding to input _name_.
</span>
* _rec_id_ is a pointer to a variable which will store the unique record identifier generated
by Darshan.
</code></pre>
<br>
</li>
</div>
<div class='footer' style='margin-top: 10px;'>
<p>
—
<br>
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2">View it on GitLab</a>
<script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","action":{"@type":"ViewAction","name":["merge_requests","issues","commit"],"url":"https://xgitlab.cels.anl.gov/darshan/darshan/compare/c3c5e59770b89b2d00aeb45a1dccbbf132a0514d...3b0c668b50b84b4407f57b5ea40336ef14d0e9d2"}}</script>
</p>
</div>
</body>
</html>