[Darshan-commits] [Git][darshan/darshan][mmap-dev] 25 commits: rework header org. for module-specific versions
Shane Snyder
xgitlab at cels.anl.gov
Tue Jan 19 17:45:03 CST 2016
Shane Snyder pushed to branch mmap-dev at darshan / darshan
Commits:
ba451e9a by Shane Snyder at 2015-11-24T22:40:51Z
rework header org. for module-specific versions
- - - - -
66181457 by Shane Snyder at 2015-11-25T10:18:12Z
modify runtime code to store each module's version
- - - - -
173d38b8 by Shane Snyder at 2015-11-25T12:41:33Z
update util interfaces to support mod-specific ver
- - - - -
b07e6220 by Phil Carns at 2015-12-01T10:44:14Z
support MPICH 3.1.1 in profile conf test
fixes #174
- - - - -
26c4ff22 by Shane Snyder at 2015-12-08T11:55:49Z
bug fix in bzip2 init in darshan logutils
- - - - -
d097c5f1 by Shane Snyder at 2015-12-08T14:00:48Z
move compression buffer alloc to shutdown time
- - - - -
c371cdea by Shane Snyder at 2015-12-08T14:25:51Z
fix possible race cond in module locking
- - - - -
1c1a9baa by Shane Snyder at 2015-12-11T13:02:32Z
cleanup a bunch of code and stale comments
- - - - -
31e0fb03 by Shane Snyder at 2015-12-11T14:27:31Z
update runtime, util, and modularization docs
- - - - -
9b021c1b by Shane Snyder at 2015-12-11T16:20:12Z
update modules to print counter descriptions
- - - - -
cb6a2525 by Shane Snyder at 2015-12-11T16:30:55Z
updated changelog prior to pre3 release
- - - - -
34e447c0 by Shane Snyder at 2015-12-11T16:34:42Z
update configure versions for pre3
- - - - -
be6d404d by Shane Snyder at 2015-12-16T15:55:01Z
forgotten header install
- - - - -
7d455c98 by Shane Snyder at 2015-12-17T17:07:03Z
back out commit for fixing races -- not needed
- - - - -
31079257 by Phil Carns at 2015-12-18T09:41:40Z
loosen PMPI symbol check
- prevents inadverdent disabling of Darshan on some MPICH builds
- fixes #94
- - - - -
2feaff78 by Shane Snyder at 2016-01-04T09:19:52Z
update runtime docs to give info on upgrading
- - - - -
64978d11 by Shane Snyder at 2016-01-04T09:50:41Z
bug fix for resolving mpi_gather in shared lib
- - - - -
86533a49 by Shane Snyder at 2016-01-04T12:46:33Z
fix typos in counter descriptions
- - - - -
d4413b4e by Shane Snyder at 2016-01-04T13:36:29Z
better error handling for opening old log files
- - - - -
72937c9a by Shane Snyder at 2016-01-04T15:05:02Z
more doc updates
- - - - -
b5bba287 by Shane Snyder at 2016-01-08T16:50:47Z
add the darshan-diff utility back
- - - - -
4bbe8e64 by Shane Snyder at 2016-01-08T17:07:01Z
add diff routines to remaining modules
- - - - -
b770409e by Shane Snyder at 2016-01-13T13:29:12Z
minor bug fix in darshan-diff tool
- - - - -
265bdaba by Shane Snyder at 2016-01-19T16:18:22Z
Merge branch 'dev-modular' into mmap-dev
Conflicts:
darshan-runtime/Makefile.in
darshan-runtime/darshan-core.h
darshan-runtime/lib/darshan-core.c
darshan-util/Makefile.in
darshan-util/darshan-bgq-logutils.c
darshan-util/darshan-diff.c
darshan-util/darshan-hdf5-logutils.c
darshan-util/darshan-logutils.c
darshan-util/darshan-mpiio-logutils.c
darshan-util/darshan-null-logutils.c
darshan-util/darshan-parser.c
darshan-util/darshan-pnetcdf-logutils.c
darshan-util/darshan-posix-logutils.c
- - - - -
96801d4a by Shane Snyder at 2016-01-19T17:44:11Z
fix post-merge build errors
dev-modular should be completely merged into mmap-dev branch
- - - - -
50 changed files:
- ChangeLog
- darshan-bgq-log-format.h
- darshan-hdf5-log-format.h
- darshan-log-format.h
- darshan-mpiio-log-format.h
- darshan-null-log-format.h
- darshan-pnetcdf-log-format.h
- darshan-posix-log-format.h
- darshan-runtime/Makefile.in
- darshan-runtime/configure
- darshan-runtime/configure.in
- darshan-runtime/darshan-core.h
- darshan-runtime/darshan-dynamic.h
- darshan-runtime/darshan-gen-cc.pl.in
- darshan-runtime/darshan-gen-cxx.pl.in
- darshan-runtime/darshan-gen-fortran.pl.in
- darshan-runtime/darshan.h
- darshan-runtime/doc/darshan-runtime.txt
- darshan-runtime/lib/darshan-bgq.c
- darshan-runtime/lib/darshan-core-init-finalize.c
- darshan-runtime/lib/darshan-core.c
- darshan-runtime/lib/darshan-hdf5.c
- darshan-runtime/lib/darshan-mpiio.c
- darshan-runtime/lib/darshan-null.c
- darshan-runtime/lib/darshan-pnetcdf.c
- darshan-runtime/lib/darshan-posix.c
- darshan-test/regression/workstation-profile-conf/env.sh
- darshan-util/Makefile.in
- darshan-util/configure
- darshan-util/configure.in
- darshan-util/darshan-bgq-logutils.c
- darshan-util/darshan-bgq-logutils.h
- darshan-util/darshan-convert.c
- darshan-util/darshan-diff.c
- darshan-util/darshan-hdf5-logutils.c
- darshan-util/darshan-hdf5-logutils.h
- darshan-util/darshan-logutils.c
- darshan-util/darshan-logutils.h
- darshan-util/darshan-mpiio-logutils.c
- darshan-util/darshan-mpiio-logutils.h
- darshan-util/darshan-null-logutils.c
- darshan-util/darshan-null-logutils.h
- darshan-util/darshan-parser.c
- darshan-util/darshan-pnetcdf-logutils.c
- darshan-util/darshan-pnetcdf-logutils.h
- darshan-util/darshan-posix-logutils.c
- darshan-util/darshan-posix-logutils.h
- darshan-util/darshan-stitch-logs.c
- darshan-util/doc/darshan-util.txt
- doc/darshan-modularization.txt
Changes:
=====================================
ChangeLog
=====================================
--- a/ChangeLog
+++ b/ChangeLog
@@ -2,6 +2,20 @@
Darshan Release Change Log
--------------------------
+Darshan-3.0.0-pre3
+=============
+* add module-specific version fields to header to allow utilities
+ to handle different versions of a module's I/O data for backwards
+ compatibility -- NOTE: this breaks the log file parsing for logs
+ obtained using Darshan-3.0.0-pre2 & Darshan-3.0.0-pre1
+* bug fix in regression test scripts for setting proper environment
+ variables to use MPI profiling configuration for Fortran apps
+* bug fix in bzip2 log writing implementation in darshan-logutils
+* possible race conditions resolved in each module's shutdown code
+* general code, comment, and documentation cleanup
+* addition of module-specific counter descriptions printed prior
+ to parsing a modules I/O data in darshan-parser
+
Darshan-3.0.0-pre2
=============
* add fix to install appropriate headers for linking external
=====================================
darshan-bgq-log-format.h
=====================================
--- a/darshan-bgq-log-format.h
+++ b/darshan-bgq-log-format.h
@@ -7,28 +7,42 @@
#ifndef __DARSHAN_BGQ_LOG_FORMAT_H
#define __DARSHAN_BGQ_LOG_FORMAT_H
-#include "darshan-log-format.h"
-
+/* current BGQ log format version */
+#define DARSHAN_BGQ_VER 1
#define BGQ_COUNTERS \
- X(BGQ_CSJOBID, "control system jobid") \
- X(BGQ_NNODES, "number of BGQ compute nodes") \
- X(BGQ_RANKSPERNODE, "number of MPI ranks per node") \
- X(BGQ_DDRPERNODE, "size in MB of DDR3 per node") \
- X(BGQ_INODES, "number of i/o nodes") \
- X(BGQ_ANODES, "dimension of A torus") \
- X(BGQ_BNODES, "dimension of B torus") \
- X(BGQ_CNODES, "dimension of C torus") \
- X(BGQ_DNODES, "dimension of D torus") \
- X(BGQ_ENODES, "dimension of E torus") \
- X(BGQ_TORUSENABLED, "which dimensions are torus") \
- X(BGQ_NUM_INDICES, "end of counters")
+ /* control system jobid*/\
+ X(BGQ_CSJOBID) \
+ /* number of BGQ compute nodes */\
+ X(BGQ_NNODES) \
+ /* number of MPI ranks per node */\
+ X(BGQ_RANKSPERNODE) \
+ /* size in MB of DDR3 per node */\
+ X(BGQ_DDRPERNODE) \
+ /* number of i/o nodes */\
+ X(BGQ_INODES) \
+ /* dimension of A torus */\
+ X(BGQ_ANODES) \
+ /* dimension of B torus */\
+ X(BGQ_BNODES) \
+ /* dimension of C torus */\
+ X(BGQ_CNODES) \
+ /* dimension of D torus */\
+ X(BGQ_DNODES) \
+ /* dimension of E torus */\
+ X(BGQ_ENODES) \
+ /* which dimensions are torus */\
+ X(BGQ_TORUSENABLED) \
+ /* end of counters */\
+ X(BGQ_NUM_INDICES)
#define BGQ_F_COUNTERS \
- X(BGQ_F_TIMESTAMP, "timestamp when data was collected") \
- X(BGQ_F_NUM_INDICES, "end of counters")
+ /* timestamp when data was collected */\
+ X(BGQ_F_TIMESTAMP) \
+ /* end of counters */\
+ X(BGQ_F_NUM_INDICES)
-#define X(a, b) a,
+#define X(a) a,
/* integer counters for the "BGQ" example module */
enum darshan_bgq_indices
{
=====================================
darshan-hdf5-log-format.h
=====================================
--- a/darshan-hdf5-log-format.h
+++ b/darshan-hdf5-log-format.h
@@ -7,7 +7,8 @@
#ifndef __DARSHAN_HDF5_LOG_FORMAT_H
#define __DARSHAN_HDF5_LOG_FORMAT_H
-#include "darshan-log-format.h"
+/* current HDF5 log format version */
+#define DARSHAN_HDF5_VER 1
#define HDF5_COUNTERS \
/* count of HDF5 opens */\
=====================================
darshan-log-format.h
=====================================
--- a/darshan-log-format.h
+++ b/darshan-log-format.h
@@ -31,44 +31,9 @@
/* max length of exe string within job record (not counting '\0') */
#define DARSHAN_EXE_LEN (DARSHAN_JOB_RECORD_SIZE - sizeof(struct darshan_job) - 1)
+/* max number of modules that can be used in a darshan log */
#define DARSHAN_MAX_MODS 16
-/* X-macro for keeping module ordering consistent */
-/* NOTE: first val used to define module enum values,
- * second val used to define module name strings, and
- * third val is used to provide the name of a
- * corresponding logutils structure for parsing module
- * data out of the log file (only used in darshan-util
- * component -- NULL can be passed if there are no
- * logutil definitions)
- */
-#define DARSHAN_MODULE_IDS \
- X(DARSHAN_NULL_MOD, "NULL", NULL) \
- X(DARSHAN_POSIX_MOD, "POSIX", &posix_logutils) \
- X(DARSHAN_MPIIO_MOD, "MPI-IO", &mpiio_logutils) \
- X(DARSHAN_HDF5_MOD, "HDF5", &hdf5_logutils) \
- X(DARSHAN_PNETCDF_MOD, "PNETCDF", &pnetcdf_logutils) \
- X(DARSHAN_BGQ_MOD, "BG/Q", &bgq_logutils)
-
-/* unique identifiers to distinguish between available darshan modules */
-/* NOTES: - valid ids range from [0...DARSHAN_MAX_MODS-1]
- * - order of ids control module shutdown order (and consequently, order in log file)
- */
-#define X(a, b, c) a,
-typedef enum
-{
- DARSHAN_MODULE_IDS
-} darshan_module_id;
-#undef X
-
-/* module name strings */
-#define X(a, b, c) b,
-static char * const darshan_module_names[] =
-{
- DARSHAN_MODULE_IDS
-};
-#undef X
-
/* simple macros for accessing module flag bitfields */
#define DARSHAN_MOD_FLAG_SET(flags, id) flags = (flags | (1 << id))
#define DARSHAN_MOD_FLAG_UNSET(flags, id) flags = (flags & ~(1 << id))
@@ -106,6 +71,7 @@ struct darshan_header
uint32_t partial_flag;
struct darshan_log_map rec_map;
struct darshan_log_map mod_map[DARSHAN_MAX_MODS];
+ uint32_t mod_ver[DARSHAN_MAX_MODS];
};
/* job-level metadata stored for this application */
@@ -126,4 +92,61 @@ struct darshan_base_record
int64_t rank;
};
+
+/************************************************
+ *** module-specific includes and definitions ***
+ ************************************************/
+
+#include "darshan-null-log-format.h"
+#include "darshan-posix-log-format.h"
+#include "darshan-mpiio-log-format.h"
+#include "darshan-hdf5-log-format.h"
+#include "darshan-pnetcdf-log-format.h"
+#include "darshan-bgq-log-format.h"
+
+/* X-macro for keeping module ordering consistent */
+/* NOTE: first val used to define module enum values,
+ * second val used to define module name strings,
+ * third val is the log format version for the module,
+ * and fourth val is used to provide the name of a
+ * corresponding logutils structure for parsing module
+ * data out of the log file (only used in darshan-util
+ * component -- NULL can be passed if there are no
+ * logutil definitions)
+ */
+#define DARSHAN_MODULE_IDS \
+ X(DARSHAN_NULL_MOD, "NULL", DARSHAN_NULL_VER, NULL) \
+ X(DARSHAN_POSIX_MOD, "POSIX", DARSHAN_POSIX_VER, &posix_logutils) \
+ X(DARSHAN_MPIIO_MOD, "MPI-IO", DARSHAN_MPIIO_VER, &mpiio_logutils) \
+ X(DARSHAN_HDF5_MOD, "HDF5", DARSHAN_HDF5_VER, &hdf5_logutils) \
+ X(DARSHAN_PNETCDF_MOD, "PNETCDF", DARSHAN_PNETCDF_VER, &pnetcdf_logutils) \
+ X(DARSHAN_BGQ_MOD, "BG/Q", DARSHAN_BGQ_VER, &bgq_logutils)
+
+/* unique identifiers to distinguish between available darshan modules */
+/* NOTES: - valid ids range from [0...DARSHAN_MAX_MODS-1]
+ * - order of ids control module shutdown order (and consequently, order in log file)
+ */
+#define X(a, b, c, d) a,
+typedef enum
+{
+ DARSHAN_MODULE_IDS
+} darshan_module_id;
+#undef X
+
+/* module name strings */
+#define X(a, b, c, d) b,
+static char * const darshan_module_names[] =
+{
+ DARSHAN_MODULE_IDS
+};
+#undef X
+
+/* module version numbers */
+#define X(a, b, c, d) c,
+static const int darshan_module_versions[] =
+{
+ DARSHAN_MODULE_IDS
+};
+#undef X
+
#endif /* __DARSHAN_LOG_FORMAT_H */
=====================================
darshan-mpiio-log-format.h
=====================================
--- a/darshan-mpiio-log-format.h
+++ b/darshan-mpiio-log-format.h
@@ -7,7 +7,8 @@
#ifndef __DARSHAN_MPIIO_LOG_FORMAT_H
#define __DARSHAN_MPIIO_LOG_FORMAT_H
-#include "darshan-log-format.h"
+/* current MPI-IO log format version */
+#define DARSHAN_MPIIO_VER 1
/* TODO: maybe use a counter to track cases in which a derived datatype is used? */
=====================================
darshan-null-log-format.h
=====================================
--- a/darshan-null-log-format.h
+++ b/darshan-null-log-format.h
@@ -7,7 +7,8 @@
#ifndef __DARSHAN_NULL_LOG_FORMAT_H
#define __DARSHAN_NULL_LOG_FORMAT_H
-#include "darshan-log-format.h"
+/* current log format version, to support backwards compatibility */
+#define DARSHAN_NULL_VER 1
#define NULL_COUNTERS \
/* count of number of 'bar' function calls */\
=====================================
darshan-pnetcdf-log-format.h
=====================================
--- a/darshan-pnetcdf-log-format.h
+++ b/darshan-pnetcdf-log-format.h
@@ -7,7 +7,8 @@
#ifndef __DARSHAN_PNETCDF_LOG_FORMAT_H
#define __DARSHAN_PNETCDF_LOG_FORMAT_H
-#include "darshan-log-format.h"
+/* current PNETCDF log format version */
+#define DARSHAN_PNETCDF_VER 1
#define PNETCDF_COUNTERS \
/* count of PNETCDF independent opens */\
=====================================
darshan-posix-log-format.h
=====================================
--- a/darshan-posix-log-format.h
+++ b/darshan-posix-log-format.h
@@ -6,7 +6,8 @@
#ifndef __DARSHAN_POSIX_LOG_FORMAT_H
#define __DARSHAN_POSIX_LOG_FORMAT_H
-#include "darshan-log-format.h"
+/* current POSIX log format version */
+#define DARSHAN_POSIX_VER 1
#define POSIX_COUNTERS \
/* count of posix opens */\
@@ -175,11 +176,4 @@ struct darshan_posix_file
double fcounters[POSIX_F_NUM_INDICES];
};
-/* This macro can be used to identify files that have been opened using
- * pnetcdf, hdf5, or mpi-io, but were never opened at the posix level. As a
- * result the record will not necessarily have all of the expected fields
- * populated.
- */
-#define POSIX_FILE_PARTIAL(__file)((((__file)->counters[POSIX_OPENS] || (__file)->counters[POSIX_FOPENS] || (__file)->counters[POSIX_STATS]) ? 0 : 1))
-
#endif /* __DARSHAN_POSIX_LOG_FORMAT_H */
=====================================
darshan-runtime/Makefile.in
=====================================
--- a/darshan-runtime/Makefile.in
+++ b/darshan-runtime/Makefile.in
@@ -35,8 +35,15 @@ CFLAGS_SHARED = -DDARSHAN_CONFIG_H=\"darshan-runtime-config.h\" -I . -I$(srcdir)
LIBS = -lz @LIBBZ2@
-static-mod-objs = lib/darshan-posix.o lib/darshan-mpiio.o lib/darshan-hdf5.o lib/darshan-pnetcdf.o
-dynamic-mod-objs = lib/darshan-posix.po lib/darshan-mpiio.po lib/darshan-hdf5.po lib/darshan-pnetcdf.po
+DARSHAN_STATIC_MOD_OBJS = lib/darshan-posix.o lib/darshan-mpiio.o lib/darshan-hdf5.o lib/darshan-pnetcdf.o
+DARSHAN_DYNAMIC_MOD_OBJS = lib/darshan-posix.po lib/darshan-mpiio.po lib/darshan-hdf5.po lib/darshan-pnetcdf.po
+
+ifdef DARSHAN_USE_BGQ
+DARSHAN_STATIC_MOD_OBJS += lib/darshan-bgq.o
+DARSHAN_DYNAMIC_MOD_OBJS += lib/darshan-bgq.po
+CFLAGS += -DDARSHAN_BGQ
+CFLAGS_SHARED += -DDARSHAN_BGQ
+endif
lib::
@mkdir -p $@
@@ -113,10 +120,10 @@ lib/lookup8.o: lib/lookup8.c
lib/lookup8.po: lib/lookup8.c
$(CC) $(CFLAGS_SHARED) -c $< -o $@
-lib/libdarshan.a: lib/darshan-core-init-finalize.o lib/darshan-core.o lib/darshan-common.o $(static-mod-objs) lib/lookup3.o lib/lookup8.o
+lib/libdarshan.a: lib/darshan-core-init-finalize.o lib/darshan-core.o lib/darshan-common.o $(DARSHAN_STATIC_MOD_OBJS) lib/lookup3.o lib/lookup8.o
ar rcs $@ $^
-lib/libdarshan.so: lib/darshan-core-init-finalize.po lib/darshan-core.po lib/darshan-common.po $(dynamic-mod-objs) lib/lookup3.po lib/lookup8.po
+lib/libdarshan.so: lib/darshan-core-init-finalize.po lib/darshan-core.po lib/darshan-common.po $(DARSHAN_DYNAMIC_MOD_OBJS) lib/lookup3.po lib/lookup8.po
$(CC) $(CFLAGS_SHARED) $(LDFLAGS) -o $@ $^ -lpthread -lrt -lz -ldl
lib/libdarshan-stubs.a: lib/darshan-hdf5-stubs.o lib/darshan-pnetcdf-stubs.o
=====================================
darshan-runtime/configure
=====================================
--- a/darshan-runtime/configure
+++ b/darshan-runtime/configure
@@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for darshan-runtime 3.0.0-pre2.
+# Generated by GNU Autoconf 2.69 for darshan-runtime 3.0.0-pre3.
#
#
# Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
@@ -577,8 +577,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='darshan-runtime'
PACKAGE_TARNAME='darshan-runtime'
-PACKAGE_VERSION='3.0.0-pre2'
-PACKAGE_STRING='darshan-runtime 3.0.0-pre2'
+PACKAGE_VERSION='3.0.0-pre3'
+PACKAGE_STRING='darshan-runtime 3.0.0-pre3'
PACKAGE_BUGREPORT=''
PACKAGE_URL=''
@@ -1244,7 +1244,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures darshan-runtime 3.0.0-pre2 to adapt to many kinds of systems.
+\`configure' configures darshan-runtime 3.0.0-pre3 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@@ -1305,7 +1305,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
- short | recursive ) echo "Configuration of darshan-runtime 3.0.0-pre2:";;
+ short | recursive ) echo "Configuration of darshan-runtime 3.0.0-pre3:";;
esac
cat <<\_ACEOF
@@ -1409,7 +1409,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-darshan-runtime configure 3.0.0-pre2
+darshan-runtime configure 3.0.0-pre3
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@@ -1761,7 +1761,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by darshan-runtime $as_me 3.0.0-pre2, which was
+It was created by darshan-runtime $as_me 3.0.0-pre3, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@@ -4296,7 +4296,7 @@ _ACEOF
fi
-DARSHAN_VERSION="3.0.0-pre2"
+DARSHAN_VERSION="3.0.0-pre3"
@@ -4815,7 +4815,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by darshan-runtime $as_me 3.0.0-pre2, which was
+This file was extended by darshan-runtime $as_me 3.0.0-pre3, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@@ -4877,7 +4877,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-darshan-runtime config.status 3.0.0-pre2
+darshan-runtime config.status 3.0.0-pre3
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"
=====================================
darshan-runtime/configure.in
=====================================
--- a/darshan-runtime/configure.in
+++ b/darshan-runtime/configure.in
@@ -5,7 +5,7 @@ dnl Process this file with autoconf to produce a configure script.
dnl You may need to use autoheader as well if changing any DEFINEs
dnl sanity checks, output header, location of scripts used here
-AC_INIT([darshan-runtime], [3.0.0-pre2])
+AC_INIT([darshan-runtime], [3.0.0-pre3])
AC_CONFIG_SRCDIR([darshan.h])
AC_CONFIG_AUX_DIR(../maint/config)
AC_CONFIG_HEADER(darshan-runtime-config.h)
=====================================
darshan-runtime/darshan-core.h
=====================================
--- a/darshan-runtime/darshan-core.h
+++ b/darshan-runtime/darshan-core.h
@@ -56,7 +56,7 @@ struct darshan_core_runtime
int rec_hash_cnt;
struct darshan_core_module* mod_array[DARSHAN_MAX_MODS];
int mod_mem_used;
- char comp_buf[DARSHAN_COMP_BUF_SIZE];
+ char *comp_buf;
double wtime_offset;
};
=====================================
darshan-runtime/darshan-dynamic.h
=====================================
--- a/darshan-runtime/darshan-dynamic.h
+++ b/darshan-runtime/darshan-dynamic.h
@@ -114,10 +114,22 @@ DARSHAN_EXTERN_DECL(PMPI_Type_get_envelope, int, (MPI_Datatype datatype, int *nu
DARSHAN_EXTERN_DECL(PMPI_Type_size, int, (MPI_Datatype datatype, int *size));
DARSHAN_EXTERN_DECL(PMPI_Op_create, int, (MPI_User_function *function, int commute, MPI_Op *op));
DARSHAN_EXTERN_DECL(PMPI_Op_free, int, (MPI_Op *op));
+#ifdef HAVE_MPIIO_CONST
+DARSHAN_EXTERN_DECL(PMPI_Reduce, int, (const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm));
+#else
DARSHAN_EXTERN_DECL(PMPI_Reduce, int, (void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm));
+#endif
+#ifdef HAVE_MPIIO_CONST
+DARSHAN_EXTERN_DECL(PMPI_Send, int, (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm));
+#else
DARSHAN_EXTERN_DECL(PMPI_Send, int, (void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm));
+#endif
DARSHAN_EXTERN_DECL(PMPI_Recv, int, (void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status));
+#ifdef HAVE_MPIIO_CONST
DARSHAN_EXTERN_DECL(PMPI_Gather, int, (const void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm));
+#else
+DARSHAN_EXTERN_DECL(PMPI_Gather, int, (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm));
+#endif
#endif
=====================================
darshan-runtime/darshan-gen-cc.pl.in
=====================================
--- a/darshan-runtime/darshan-gen-cc.pl.in
+++ b/darshan-runtime/darshan-gen-cc.pl.in
@@ -261,8 +261,8 @@ print OUTPUT<<"EOF";
# is MPI in there?
grep MPI \$tmpfile >& /dev/null
rc_mpi=\$?
- # is PMPI being used for any init, finalize, or mpi-io calls?
- grep -E \\(PMPI_File_\\)\\|\\(PMPI_Init\\)\\|\\(PMPI_Finalize\\) \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
+ # is PMPI being used for PMPI_File_open?
+ grep -E PMPI_File_open \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
rc_pmpi=\$?
rm \$tmpfile >& /dev/null
=====================================
darshan-runtime/darshan-gen-cxx.pl.in
=====================================
--- a/darshan-runtime/darshan-gen-cxx.pl.in
+++ b/darshan-runtime/darshan-gen-cxx.pl.in
@@ -263,8 +263,8 @@ print OUTPUT<<"EOF";
grep MPI \$tmpfile >& /dev/null
rc_mpi=\$?
- # is PMPI being used for any init, finalize, or mpi-io calls?
- grep -E \\(PMPI_File_\\)\\|\\(PMPI_Init\\)\\|\\(PMPI_Finalize\\) \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
+ # is PMPI being used for PMPI_File_open?
+ grep -E PMPI_File_open \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
rc_pmpi=\$?
CXXMPICH=-lmpichcxx
=====================================
darshan-runtime/darshan-gen-fortran.pl.in
=====================================
--- a/darshan-runtime/darshan-gen-fortran.pl.in
+++ b/darshan-runtime/darshan-gen-fortran.pl.in
@@ -271,8 +271,8 @@ print OUTPUT<<"EOF";
grep -i MPI \$tmpfile >& /dev/null
rc_mpi=\$?
- # is PMPI being used for any init, finalize, or mpi-io calls?
- grep -E \\(PMPI_File_\\)\\|\\(PMPI_Init\\)\\|\\(PMPI_Finalize\\) \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
+ # is PMPI being used for PMPI_File_open?
+ grep -E PMPI_File_open \$tmpfile | grep -v -E \\(mpich.*\\.a\\) |grep \\(PMPI >& /dev/null
rc_pmpi=\$?
=====================================
darshan-runtime/darshan.h
=====================================
--- a/darshan-runtime/darshan.h
+++ b/darshan-runtime/darshan.h
@@ -125,17 +125,17 @@ void darshan_core_lookup_record(
/* darshan_core_register_record()
*
- * Register the Darshan record given by 'name' with the darshan-core
- * runtime, allowing it to be properly tracked and (potentially)
- * correlated with records from other modules. 'len' is the size of
- * the name pointer (string length for string names), and 'printable_flag'
- * indicates whether the name is a string. 'mod_limit_flag' is set if
- * the calling module is out of memory (to prevent darshan-core from
- * creating new records and to just search existing records) and 'mod_id'
- * is the identifier of the calling module. 'rec_id' is an output pointer
- * storing the correspoing Darshan record identifier and 'file_alignment'
- * is an output pointer storing the file system alignment value for the
- * given record.
+ * Register a record with the darshan-core runtime, allowing it to be
+ * properly tracked and (potentially) correlated with records from other
+ * modules. 'name' is the the name of the Darshan record (e.g., the full
+ * file path) and 'len' is the size of the name pointer (string length
+ * for string names). 'mod_id' is the identifier of the calling module,
+ * 'printable_flag' indicates whether the name is a string, and
+ * 'mod_limit_flag' is set if the calling module is out of memory (to
+ * prevent darshan-core from creating new records and to just search
+ * through existing records). 'rec_id' is an output pointer storing the
+ * correspoing Darshan record identifier and 'file_alignment' is an output
+ * pointer storing the file system alignment value for the given record.
*/
int darshan_core_register_record(
darshan_record_id rec_id,
=====================================
darshan-runtime/doc/darshan-runtime.txt
=====================================
--- a/darshan-runtime/doc/darshan-runtime.txt
+++ b/darshan-runtime/doc/darshan-runtime.txt
@@ -16,7 +16,10 @@ used by the application.
The darshan-runtime instrumentation only instruments MPI applications (the
application must at least call `MPI_Init()` and `MPI_Finalize()`). However,
it captures both MPI-IO and POSIX file access. It also captures limited
-information about HDF5 and PnetCDF access.
+information about HDF5 and PnetCDF access. Darshan also exposes an API that
+can be used to develop and add new instrumentation modules (for other I/O library
+interfaces or to gather system-specific data, for instance), as detailed in
+http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-modularization.html[this document].
This document provides generic installation instructions, but "recipes" for
several common HPC systems are provided at the end of the document as well.
@@ -311,7 +314,7 @@ Please set your environment to use the GNU programming environment before
configuring or compiling Darshan. Although Darshan can be built with a
variety of compilers, the GNU compilers are recommended because it will
produce a Darshan library that is interoperable with the widest range
-of compmilers and linkers. On most Cray systems you can enable the GNU
+of compilers and linkers. On most Cray systems you can enable the GNU
programming environment with a command similar to "module swap PrgEnv-pgi
PrgEnv-gnu". Please see your site documentation for information about
how to switch programming environments.
@@ -463,6 +466,40 @@ older versions of Open MPI, please refer to the following mailing list discussio
http://www.open-mpi.org/community/lists/devel/2013/01/11907.php
+== Upgrading to Darshan 3.x from 2.x
+
+Beginning with Darshan 3.0.0, Darshan has been rewritten to modularize its runtime environment
+and log file format to simplify the addition of new I/O characterization data. The process of
+compiling and installing the Darshan 3.x source code should essentially be identical to this
+process on Darshan 2.x. Therefore, the installation recipes given in the previous section
+should work irrespective of the Darshan version being used. Similarly, the manner in which
+Darshan is used should be the same across versions -- the sections in this document regarding
+Darshan link:darshan-runtime.html#_environment_preparation[environment preparation],
+instrumenting link:darshan-runtime.html#_instrumenting_statically_linked_applications[statically
+linked applications] and link:darshan-runtime.html#_instrumenting_dynamically_linked_applications[
+dynamically linked applications], and using link:darshan-runtime.html#_runtime_environment_variables[
+runtime environment variables] are equally applicable to both versions.
+
+However, we do provide some suggestions and expectations for system administrators to keep in
+mind when upgrading to Darshan 3.x:
+
+* Log file compatibility was broken in the upgrade, and thus 3.x log utilities do not
+work directly with logs generated by 2.x versions of Darshan (and vice versa).
+ - There is currently no tool for converting 2.x logs into the 3.x log format.
+ - The `darshan-logutils` library will provide error messages to indicate whether a given
+log file is incompatible with the correspnonding library version.
+
+* The darshan log file extension has been changed from `.darshan.gz` (or `.darshan.bz2` for
+log files converted to use bzip2 compression) to `.darshan`.
+ - A field in the Darshan log header is used to indicate whether a log is compressed using
+libz or bzip2 compression.
+
+* We encourage administrators to use the same log file directory for version 3.x as had been
+used for version 2.x.
+ - Within this directory, the determination on which set of log utilities (version 2.x
+or version 3.x) to use can be based on the file extension for a given log (as explained
+above).
+
== Runtime environment variables
The Darshan library honors the following environment variables to modify
=====================================
darshan-runtime/lib/darshan-bgq.c
=====================================
--- a/darshan-runtime/lib/darshan-bgq.c
+++ b/darshan-runtime/lib/darshan-bgq.c
@@ -16,7 +16,6 @@
#include "uthash.h"
#include "darshan.h"
-#include "darshan-bgq-log-format.h"
#include "darshan-dynamic.h"
#include <mpix.h>
@@ -54,7 +53,7 @@ static int instrumentation_disabled = 0;
static int my_rank = -1;
static int darshan_mem_alignment = 1;
-/* internal helper functions for the "NULL" module */
+/* internal helper functions for the BGQ module */
void bgq_runtime_initialize(void);
/* forward declaration for module functions needed to interface with darshan-core */
@@ -62,7 +61,7 @@ static void bgq_begin_shutdown(void);
static void bgq_get_output_data(MPI_Comm mod_comm, darshan_record_id *shared_recs, int shared_rec_count, void **buffer, int *size);
static void bgq_shutdown(void);
-/* macros for obtaining/releasing the "NULL" module lock */
+/* macros for obtaining/releasing the BGQ module lock */
#define BGQ_LOCK() pthread_mutex_lock(&bgq_runtime_mutex)
#define BGQ_UNLOCK() pthread_mutex_unlock(&bgq_runtime_mutex)
@@ -141,13 +140,15 @@ void bgq_runtime_initialize()
if(mem_limit == 0)
{
instrumentation_disabled = 1;
+ BGQ_UNLOCK();
return;
}
- /* no enough memory to fit bgq module */
+ /* not enough memory to fit bgq module */
if (mem_limit < sizeof(*bgq_runtime))
{
instrumentation_disabled = 1;
+ BGQ_UNLOCK();
return;
}
@@ -156,6 +157,7 @@ void bgq_runtime_initialize()
if(!bgq_runtime)
{
instrumentation_disabled = 1;
+ BGQ_UNLOCK();
return;
}
memset(bgq_runtime, 0, sizeof(*bgq_runtime));
@@ -188,7 +190,7 @@ void bgq_runtime_initialize()
return;
}
-/* Perform any necessary steps prior to shutting down for the "NULL" module. */
+/* Perform any necessary steps prior to shutting down for the BGQ module. */
static void bgq_begin_shutdown()
{
BGQ_LOCK();
@@ -212,7 +214,7 @@ static int cmpr(const void *p1, const void *p2)
return ((*a == *b) ? 0 : ((*a < *b) ? -1 : 1));
}
-/* Pass output data for the "BGQ" module back to darshan-core to log to file. */
+/* Pass output data for the BGQ module back to darshan-core to log to file. */
static void bgq_get_output_data(
MPI_Comm mod_comm,
darshan_record_id *shared_recs,
@@ -220,8 +222,7 @@ static void bgq_get_output_data(
void **buffer,
int *size)
{
-
- /* Just set the output buffer to point at the array of the "BGQ" module's
+ /* Just set the output buffer to point at the array of the BGQ module's
* I/O records, and set the output size according to the number of records
* currently being tracked.
*/
@@ -279,7 +280,7 @@ static void bgq_get_output_data(
return;
}
-/* Shutdown the "BGQ" module by freeing up all data structures. */
+/* Shutdown the BGQ module by freeing up all data structures. */
static void bgq_shutdown()
{
if (bgq_runtime)
=====================================
darshan-runtime/lib/darshan-core-init-finalize.c
=====================================
--- a/darshan-runtime/lib/darshan-core-init-finalize.c
+++ b/darshan-runtime/lib/darshan-core-init-finalize.c
@@ -120,9 +120,22 @@ DARSHAN_FORWARD_DECL(PMPI_Type_get_envelope, int, (MPI_Datatype datatype, int *n
DARSHAN_FORWARD_DECL(PMPI_Type_size, int, (MPI_Datatype datatype, int *size));
DARSHAN_FORWARD_DECL(PMPI_Op_create, int, (MPI_User_function *function, int commute, MPI_Op *op));
DARSHAN_FORWARD_DECL(PMPI_Op_free, int, (MPI_Op *op));
+#ifdef HAVE_MPIIO_CONST
+DARSHAN_FORWARD_DECL(PMPI_Reduce, int, (const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm));
+#else
DARSHAN_FORWARD_DECL(PMPI_Reduce, int, (void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm));
+#endif
+#ifdef HAVE_MPIIO_CONST
+DARSHAN_FORWARD_DECL(PMPI_Send, int, (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm));
+#else
DARSHAN_FORWARD_DECL(PMPI_Send, int, (void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm));
+#endif
DARSHAN_FORWARD_DECL(PMPI_Recv, int, (void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status));
+#ifdef HAVE_MPIIO_CONST
+DARSHAN_FORWARD_DECL(PMPI_Gather, int, (const void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm));
+#else
+DARSHAN_FORWARD_DECL(PMPI_Gather, int, (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm));
+#endif
void resolve_mpi_symbols (void)
{
@@ -182,6 +195,7 @@ void resolve_mpi_symbols (void)
MAP_OR_FAIL(PMPI_Reduce);
MAP_OR_FAIL(PMPI_Send);
MAP_OR_FAIL(PMPI_Recv);
+ MAP_OR_FAIL(PMPI_Gather);
return;
}
=====================================
darshan-runtime/lib/darshan-core.c
=====================================
--- a/darshan-runtime/lib/darshan-core.c
+++ b/darshan-runtime/lib/darshan-core.c
@@ -313,6 +313,8 @@ void darshan_core_initialize(int argc, char **argv)
void darshan_core_shutdown()
{
struct darshan_core_runtime *final_core;
+ struct darshan_header out_header;
+ struct darshan_job out_job;
int internal_timing_flag = 0;
struct tm *start_tm;
time_t start_time_tmp;
@@ -336,8 +338,6 @@ void darshan_core_shutdown()
int all_ret = 0;
int i;
uint64_t gz_fp = 0;
- struct darshan_header out_header;
- struct darshan_job out_job;
MPI_File log_fh;
MPI_Status status;
@@ -358,14 +358,14 @@ void darshan_core_shutdown()
darshan_core = NULL;
DARSHAN_CORE_UNLOCK();
- memcpy(&out_job, final_core->log_job_p, sizeof(struct darshan_job));
-
/* XXX just copy mmap files somewhere else to avoid corruption */
DARSHAN_MPI_CALL(PMPI_Barrier)(MPI_COMM_WORLD);
if(my_rank == 0)
system("cp /tmp/darshan* ~/Desktop");
DARSHAN_MPI_CALL(PMPI_Barrier)(MPI_COMM_WORLD);
+ memcpy(&out_job, final_core->log_job_p, sizeof(struct darshan_job));
+
/* indicate in the metadata field of the temporary darshan log file that
* the darshan shutdown process was invoked on the data in the log. since
* we have no way of knowing how far the shutdown process got, the data
@@ -376,6 +376,10 @@ void darshan_core_shutdown()
int meta_remain = DARSHAN_JOB_METADATA_LEN - strlen(final_core->log_job_p->metadata) - 1;
snprintf(m, meta_remain, "darshan_shutdown=yes\n");
+ /* we also need to set which modules were registered on this process and
+ * call into those modules and give them a chance to perform any necessary
+ * pre-shutdown steps.
+ */
for(i = 0; i < DARSHAN_MAX_MODS; i++)
{
if(final_core->mod_array[i])
@@ -385,6 +389,13 @@ void darshan_core_shutdown()
}
}
+ final_core->comp_buf = malloc(DARSHAN_COMP_BUF_SIZE);
+ if(!(final_core->comp_buf))
+ {
+ darshan_core_cleanup(final_core);
+ return;
+ }
+
logfile_name = malloc(PATH_MAX);
if(!logfile_name)
{
@@ -639,6 +650,9 @@ void darshan_core_shutdown()
*/
DARSHAN_MPI_CALL(PMPI_Reduce)(&(final_core->log_hdr_p->partial_flag),
&(out_header.partial_flag), 1, MPI_UINT32_T, MPI_BOR, 0, MPI_COMM_WORLD);
+ DARSHAN_MPI_CALL(PMPI_Reduce)(&(final_core->log_hdr_p->mod_ver),
+ &(out_header.mod_ver), DARSHAN_MAX_MODS, MPI_UINT32_T, MPI_MAX,
+ 0, MPI_COMM_WORLD);
if(my_rank == 0)
{
/* rank 0 is responsible for writing the log header */
@@ -1550,6 +1564,8 @@ static void darshan_core_cleanup(struct darshan_core_runtime* core)
}
}
+ if(core->comp_buf)
+ free(core->comp_buf);
free(core);
return;
@@ -1609,6 +1625,7 @@ void darshan_core_register_module(
/* register module with darshan */
darshan_core->mod_array[mod_id] = mod;
+ darshan_core->log_hdr_p->mod_ver[mod_id] = darshan_module_versions[mod_id];
DARSHAN_CORE_UNLOCK();
/* set the memory alignment and calling process's rank, if desired */
=====================================
darshan-runtime/lib/darshan-hdf5.c
=====================================
--- a/darshan-runtime/lib/darshan-hdf5.c
+++ b/darshan-runtime/lib/darshan-hdf5.c
@@ -22,7 +22,6 @@
#include "uthash.h"
#include "darshan.h"
-#include "darshan-hdf5-log-format.h"
#include "darshan-dynamic.h"
/* hope this doesn't change any time soon */
@@ -511,7 +510,9 @@ static void hdf5_get_output_data(
{
red_recv_buf = malloc(shared_rec_count * sizeof(struct darshan_hdf5_file));
if(!red_recv_buf)
+ {
return;
+ }
}
/* construct a datatype for a HDF5 file record. This is serving no purpose
=====================================
darshan-runtime/lib/darshan-mpiio.c
=====================================
--- a/darshan-runtime/lib/darshan-mpiio.c
+++ b/darshan-runtime/lib/darshan-mpiio.c
@@ -23,7 +23,6 @@
#include "uthash.h"
#include "darshan.h"
-#include "darshan-mpiio-log-format.h"
#include "darshan-dynamic.h"
/* The mpiio_file_runtime structure maintains necessary runtime metadata
@@ -1386,7 +1385,9 @@ static void mpiio_get_output_data(
{
red_recv_buf = malloc(shared_rec_count * sizeof(struct darshan_mpiio_file));
if(!red_recv_buf)
+ {
return;
+ }
}
/* construct a datatype for a MPIIO file record. This is serving no purpose
=====================================
darshan-runtime/lib/darshan-null.c
=====================================
--- a/darshan-runtime/lib/darshan-null.c
+++ b/darshan-runtime/lib/darshan-null.c
@@ -16,7 +16,6 @@
#include "uthash.h"
#include "darshan.h"
-#include "darshan-null-log-format.h"
/* The "NULL" module is an example instrumentation module implementation provided
* with Darshan, primarily to indicate how arbitrary modules may be integrated
=====================================
darshan-runtime/lib/darshan-pnetcdf.c
=====================================
--- a/darshan-runtime/lib/darshan-pnetcdf.c
+++ b/darshan-runtime/lib/darshan-pnetcdf.c
@@ -22,7 +22,6 @@
#include "uthash.h"
#include "darshan.h"
-#include "darshan-pnetcdf-log-format.h"
#include "darshan-dynamic.h"
DARSHAN_FORWARD_DECL(ncmpi_create, int, (MPI_Comm comm, const char *path, int cmode, MPI_Info info, int *ncidp));
@@ -526,7 +525,9 @@ static void pnetcdf_get_output_data(
{
red_recv_buf = malloc(shared_rec_count * sizeof(struct darshan_pnetcdf_file));
if(!red_recv_buf)
+ {
return;
+ }
}
/* construct a datatype for a PNETCDF file record. This is serving no purpose
=====================================
darshan-runtime/lib/darshan-posix.c
=====================================
--- a/darshan-runtime/lib/darshan-posix.c
+++ b/darshan-runtime/lib/darshan-posix.c
@@ -30,7 +30,6 @@
#include "utlist.h"
#include "darshan.h"
-#include "darshan-posix-log-format.h"
#include "darshan-dynamic.h"
#ifndef HAVE_OFF64_T
@@ -40,8 +39,6 @@ typedef int64_t off64_t;
#define aiocb64 aiocb
#endif
-/* TODO: more libc, fgetc, etc etc etc. */
-
DARSHAN_FORWARD_DECL(open, int, (const char *path, int flags, ...));
DARSHAN_FORWARD_DECL(open64, int, (const char *path, int flags, ...));
DARSHAN_FORWARD_DECL(creat, int, (const char* path, mode_t mode));
@@ -1892,10 +1889,7 @@ static void posix_record_reduction_op(void* infile_v, void* inoutfile_v,
tmp_file.counters[j] = infile->counters[j] + inoutfile->counters[j];
}
- if(POSIX_FILE_PARTIAL(infile))
- tmp_file.counters[POSIX_MODE] = inoutfile->counters[POSIX_MODE];
- else
- tmp_file.counters[POSIX_MODE] = infile->counters[POSIX_MODE];
+ tmp_file.counters[POSIX_MODE] = infile->counters[POSIX_MODE];
/* sum */
for(j=POSIX_BYTES_READ; j<=POSIX_BYTES_WRITTEN; j++)
@@ -1918,10 +1912,7 @@ static void posix_record_reduction_op(void* infile_v, void* inoutfile_v,
tmp_file.counters[j] = infile->counters[j] + inoutfile->counters[j];
}
- if(POSIX_FILE_PARTIAL(infile))
- tmp_file.counters[POSIX_MEM_ALIGNMENT] = inoutfile->counters[POSIX_MEM_ALIGNMENT];
- else
- tmp_file.counters[POSIX_MEM_ALIGNMENT] = infile->counters[POSIX_MEM_ALIGNMENT];
+ tmp_file.counters[POSIX_MEM_ALIGNMENT] = infile->counters[POSIX_MEM_ALIGNMENT];
/* sum */
for(j=POSIX_FILE_NOT_ALIGNED; j<=POSIX_FILE_NOT_ALIGNED; j++)
@@ -1929,10 +1920,7 @@ static void posix_record_reduction_op(void* infile_v, void* inoutfile_v,
tmp_file.counters[j] = infile->counters[j] + inoutfile->counters[j];
}
- if(POSIX_FILE_PARTIAL(infile))
- tmp_file.counters[POSIX_FILE_ALIGNMENT] = inoutfile->counters[POSIX_FILE_ALIGNMENT];
- else
- tmp_file.counters[POSIX_FILE_ALIGNMENT] = infile->counters[POSIX_FILE_ALIGNMENT];
+ tmp_file.counters[POSIX_FILE_ALIGNMENT] = infile->counters[POSIX_FILE_ALIGNMENT];
/* skip POSIX_MAX_*_TIME_SIZE; handled in floating point section */
@@ -2302,7 +2290,9 @@ static void posix_get_output_data(
{
red_recv_buf = malloc(shared_rec_count * sizeof(struct darshan_posix_file));
if(!red_recv_buf)
+ {
return;
+ }
}
/* construct a datatype for a POSIX file record. This is serving no purpose
@@ -2363,7 +2353,7 @@ static void posix_shutdown()
free(posix_runtime->file_runtime_array);
free(posix_runtime);
posix_runtime = NULL;
-
+
return;
}
=====================================
darshan-test/regression/workstation-profile-conf/env.sh
=====================================
--- a/darshan-test/regression/workstation-profile-conf/env.sh
+++ b/darshan-test/regression/workstation-profile-conf/env.sh
@@ -34,5 +34,8 @@ export MPICC_PROFILE=$DARSHAN_PATH/share/mpi-profile/darshan-cc
export MPICXX_PROFILE=$DARSHAN_PATH/share/mpi-profile/darshan-cxx
export MPIF90_PROFILE=$DARSHAN_PATH/share/mpi-profile/darshan-f
export MPIF77_PROFILE=$DARSHAN_PATH/share/mpi-profile/darshan-f
+# MPICH 3.1.1 and newer use MPIFORT rather than MPIF90 and MPIF77 in env var
+# name
+export MPIFORT_PROFILE=$DARSHAN_PATH/share/mpi-profile/darshan-f
export DARSHAN_RUNJOB="mpiexec -n $DARSHAN_DEFAULT_NPROCS"
=====================================
darshan-util/Makefile.in
=====================================
--- a/darshan-util/Makefile.in
+++ b/darshan-util/Makefile.in
@@ -15,6 +15,10 @@ pkgconfigdir = $(DESTDIR)$(libdir)/pkgconfig
DARSHAN_LOG_FORMAT = $(srcdir)/../darshan-log-format.h
DARSHAN_MOD_LOG_FORMATS = $(srcdir)/../darshan-posix-log-format.h $(srcdir)/../darshan-mpiio-log-format.h $(srcdir)/../darshan-hdf5-log-format.h $(srcdir)/../darshan-pnetcdf-log-format.h
DARSHAN_MOD_LOGUTIL_HEADERS = darshan-posix-logutils.h darshan-mpiio-logutils.h darshan-hdf5-logutils.h darshan-pnetcdf-logutils.h
+DARSHAN_STATIC_MOD_OBJS = darshan-posix-logutils.o darshan-mpiio-logutils.o darshan-hdf5-logutils.o darshan-pnetcdf-logutils.o darshan-bgq-logutils.o
+DARSHAN_DYNAMIC_MOD_OBJS = darshan-posix-logutils.po darshan-mpiio-logutils.po darshan-hdf5-logutils.po darshan-pnetcdf-logutils.po darshan-bgq-logutils.po
+
+#XXX BGQ
DARSHAN_ENABLE_SHARED=@DARSHAN_ENABLE_SHARED@
@@ -79,13 +83,12 @@ darshan-bgq-logutils.o: darshan-bgq-logutils.c darshan-logutils.h darshan-bgq-lo
darshan-bgq-logutils.po: darshan-bgq-logutils.c darshan-logutils.h darshan-bgq-logutils.h $(DARSHAN_LOG_FORMAT) $(srcdir)/../darshan-bgq-log-format.h | uthash-1.9.2
$(CC) $(CFLAGS_SHARED) -c $< -o $@
+libdarshan-util.a: darshan-logutils.o $(DARSHAN_STATIC_MOD_OBJS)
+ ar rcs libdarshan-util.a $^
-libdarshan-util.so: darshan-logutils.po darshan-posix-logutils.po darshan-mpiio-logutils.po darshan-hdf5-logutils.po darshan-pnetcdf-logutils.po darshan-bgq-logutils.po
+libdarshan-util.so: darshan-logutils.po $(DARSHAN_DYNAMIC_MOD_OBJS)
$(CC) $(CFLAGS_SHARED) $(LDFLAGS) -o $@ $^ $(LIBS)
-libdarshan-util.a: darshan-logutils.o darshan-posix-logutils.o darshan-mpiio-logutils.o darshan-hdf5-logutils.o darshan-pnetcdf-logutils.o darshan-bgq-logutils.o
- ar rcs libdarshan-util.a $^
-
jenkins-hash-gen: jenkins-hash-gen.c lookup3.o
$(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@ $(LIBS)
@@ -93,19 +96,19 @@ lookup3.o: lookup3.c
$(CC) $(CFLAGS) -c $< -o $@
darshan-analyzer: darshan-analyzer.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) $(DARSHAN_MOD_LOGUTIL_HEADERS) $(DARSHAN_MOD_LOG_FORMATS) libdarshan-util.a | uthash-1.9.2
- $(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@ $(LIBS)
+ $(CC) $(CFLAGS) $(LDFLAGS) $< libdarshan-util.a -o $@ $(LIBS)
-darshan-convert: darshan-convert.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) libdarshan-util.a lookup3.o | uthash-1.9.2
- $(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@ $(LIBS)
+darshan-convert: darshan-convert.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) $(DARSHAN_MOD_LOGUTIL_HEADERS) $(DARSHAN_MOD_LOG_FORMATS) libdarshan-util.a lookup3.o | uthash-1.9.2
+ $(CC) $(CFLAGS) $(LDFLAGS) $< lookup3.o libdarshan-util.a -o $@ $(LIBS)
darshan-diff: darshan-diff.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) $(DARSHAN_MOD_LOGUTIL_HEADERS) $(DARSHAN_MOD_LOG_FORMATS) libdarshan-util.a | uthash-1.9.2
$(CC) $(CFLAGS) $(LDFLAGS) $< libdarshan-util.a -o $@ $(LIBS)
darshan-parser: darshan-parser.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) $(DARSHAN_MOD_LOGUTIL_HEADERS) $(DARSHAN_MOD_LOG_FORMATS) libdarshan-util.a | uthash-1.9.2
- $(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@ $(LIBS)
+ $(CC) $(CFLAGS) $(LDFLAGS) $< libdarshan-util.a -o $@ $(LIBS)
-darshan-stitch-logs: darshan-stitch-logs.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) libdarshan-util.a | uthash-1.9.2
- $(CC) $(CFLAGS) $(LDFLAGS) $^ -o $@ $(LIBS)
+darshan-stitch-logs: darshan-stitch-logs.c darshan-logutils.h $(DARSHAN_LOG_FORMAT) $(DARSHAN_MOD_LOGUTIL_HEADERS) $(DARSHAN_MOD_LOG_FORMATS) libdarshan-util.a | uthash-1.9.2
+ $(CC) $(CFLAGS) $(LDFLAGS) $< libdarshan-util.a -o $@ $(LIBS)
#test/gztest: test/gztest.c mktestdir
# $(CC) $(CFLAGS) $(LDFLAGS) -lz $< -o $@
@@ -122,6 +125,7 @@ install:: all
install -m 755 darshan-convert $(bindir)
install -m 755 darshan-diff $(bindir)
install -m 755 darshan-parser $(bindir)
+ install -m 755 darshan-stitch-logs $(bindir)
install -m 755 $(srcdir)/darshan-summary-per-file.sh $(bindir)
install -m 755 libdarshan-util.a $(libdir)
ifeq ($(DARSHAN_ENABLE_SHARED),1)
@@ -133,6 +137,7 @@ endif
install -m 644 $(srcdir)/darshan-hdf5-logutils.h $(includedir)
install -m 644 $(srcdir)/darshan-pnetcdf-logutils.h $(includedir)
install -m 644 $(srcdir)/darshan-bgq-logutils.h $(includedir)
+ install -m 644 $(srcdir)/darshan-null-logutils.h $(includedir)
install -m 644 $(srcdir)/../darshan-posix-log-format.h $(includedir)
install -m 644 $(srcdir)/../darshan-mpiio-log-format.h $(includedir)
install -m 644 $(srcdir)/../darshan-hdf5-log-format.h $(includedir)
=====================================
darshan-util/configure
=====================================
--- a/darshan-util/configure
+++ b/darshan-util/configure
@@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for darshan-util 3.0.0-pre2.
+# Generated by GNU Autoconf 2.69 for darshan-util 3.0.0-pre3.
#
#
# Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
@@ -577,8 +577,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='darshan-util'
PACKAGE_TARNAME='darshan-util'
-PACKAGE_VERSION='3.0.0-pre2'
-PACKAGE_STRING='darshan-util 3.0.0-pre2'
+PACKAGE_VERSION='3.0.0-pre3'
+PACKAGE_STRING='darshan-util 3.0.0-pre3'
PACKAGE_BUGREPORT=''
PACKAGE_URL=''
@@ -1236,7 +1236,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures darshan-util 3.0.0-pre2 to adapt to many kinds of systems.
+\`configure' configures darshan-util 3.0.0-pre3 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@@ -1297,7 +1297,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
- short | recursive ) echo "Configuration of darshan-util 3.0.0-pre2:";;
+ short | recursive ) echo "Configuration of darshan-util 3.0.0-pre3:";;
esac
cat <<\_ACEOF
@@ -1393,7 +1393,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-darshan-util configure 3.0.0-pre2
+darshan-util configure 3.0.0-pre3
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@@ -1758,7 +1758,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by darshan-util $as_me 3.0.0-pre2, which was
+It was created by darshan-util $as_me 3.0.0-pre3, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@@ -4105,7 +4105,7 @@ fi
done
-DARSHAN_UTIL_VERSION="3.0.0-pre2"
+DARSHAN_UTIL_VERSION="3.0.0-pre3"
@@ -4621,7 +4621,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by darshan-util $as_me 3.0.0-pre2, which was
+This file was extended by darshan-util $as_me 3.0.0-pre3, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@@ -4683,7 +4683,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-darshan-util config.status 3.0.0-pre2
+darshan-util config.status 3.0.0-pre3
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"
=====================================
darshan-util/configure.in
=====================================
--- a/darshan-util/configure.in
+++ b/darshan-util/configure.in
@@ -5,7 +5,7 @@ dnl Process this file with autoconf to produce a configure script.
dnl You may need to use autoheader as well if changing any DEFINEs
dnl sanity checks, output header, location of scripts used here
-AC_INIT([darshan-util], [3.0.0-pre2])
+AC_INIT([darshan-util], [3.0.0-pre3])
AC_CONFIG_SRCDIR([darshan-logutils.h])
AC_CONFIG_AUX_DIR(../maint/config)
AC_CONFIG_HEADER(darshan-util-config.h)
=====================================
darshan-util/darshan-bgq-logutils.c
=====================================
--- a/darshan-util/darshan-bgq-logutils.c
+++ b/darshan-util/darshan-bgq-logutils.c
@@ -17,10 +17,10 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-bgq-logutils.h"
+#include "darshan-logutils.h"
-/* counter name strings for the POSIX module */
-#define X(a, b) #a,
+/* counter name strings for the BGQ module */
+#define X(a) #a,
char *bgq_counter_names[] = {
BGQ_COUNTERS
};
@@ -31,16 +31,22 @@ char *bgq_f_counter_names[] = {
#undef X
static int darshan_log_get_bgq_rec(darshan_fd fd, void* bgq_buf);
-static int darshan_log_put_bgq_rec(darshan_fd fd, void* bgq_buf);
+static int darshan_log_put_bgq_rec(darshan_fd fd, void* bgq_buf, int ver);
static void darshan_log_print_bgq_rec(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_bgq_description(void);
+static void darshan_log_print_bgq_rec_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2);
+static void darshan_log_agg_bgq_recs(void *rec, void *agg_rec, int init_flag);
struct darshan_mod_logutil_funcs bgq_logutils =
{
.log_get_record = &darshan_log_get_bgq_rec,
.log_put_record = &darshan_log_put_bgq_rec,
.log_print_record = &darshan_log_print_bgq_rec,
- .log_agg_records = NULL /* TODO: how would aggregation work for the BG/Q module ? */
+ .log_print_description = &darshan_log_print_bgq_description,
+ .log_print_diff = &darshan_log_print_bgq_rec_diff,
+ .log_agg_records = &darshan_log_agg_bgq_recs
};
static int darshan_log_get_bgq_rec(darshan_fd fd, void* bgq_buf)
@@ -73,13 +79,13 @@ static int darshan_log_get_bgq_rec(darshan_fd fd, void* bgq_buf)
}
}
-static int darshan_log_put_bgq_rec(darshan_fd fd, void* bgq_buf)
+static int darshan_log_put_bgq_rec(darshan_fd fd, void* bgq_buf, int ver)
{
struct darshan_bgq_record *rec = (struct darshan_bgq_record *)bgq_buf;
int ret;
ret = darshan_log_putmod(fd, DARSHAN_BGQ_MOD, rec,
- sizeof(struct darshan_bgq_record));
+ sizeof(struct darshan_bgq_record), ver);
if(ret < 0)
return(-1);
@@ -87,7 +93,7 @@ static int darshan_log_put_bgq_rec(darshan_fd fd, void* bgq_buf)
}
static void darshan_log_print_bgq_rec(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_bgq_record *bgq_file_rec =
@@ -112,6 +118,102 @@ static void darshan_log_print_bgq_rec(void *file_rec, char *file_name,
return;
}
+static void darshan_log_print_bgq_description()
+{
+ printf("\n# description of BGQ counters:\n");
+ printf("# BGQ_CSJOBID: BGQ control system job ID.\n");
+ printf("# BGQ_NNODES: number of BGQ compute nodes for this job.\n");
+ printf("# BGQ_RANKSPERNODE: number of MPI ranks per compute node.\n");
+ printf("# BGQ_DDRPERNODE: size in MB of DDR3 per compute node.\n");
+ printf("# BGQ_INODES: number of BGQ I/O nodes for this job.\n");
+ printf("# BGQ_*NODES: dimension of A, B, C, D, & E dimensions of torus.\n");
+ printf("# BGQ_TORUSENABLED: which dimensions of the torus are enabled.\n");
+ printf("# BGQ_F_TIMESTAMP: timestamp when the BGQ data was collected.\n");
+
+ DARSHAN_PRINT_HEADER();
+
+ return;
+}
+
+static void darshan_log_print_bgq_rec_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
+{
+ struct darshan_bgq_record *file1 = (struct darshan_bgq_record *)file_rec1;
+ struct darshan_bgq_record *file2 = (struct darshan_bgq_record *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+
+ for(i=0; i<BGQ_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file1->base_rec.rank, file1->base_rec.id, bgq_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file2->base_rec.rank, file2->base_rec.id, bgq_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file1->base_rec.rank, file1->base_rec.id, bgq_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file2->base_rec.rank, file2->base_rec.id, bgq_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<BGQ_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file1->base_rec.rank, file1->base_rec.id, bgq_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file2->base_rec.rank, file2->base_rec.id, bgq_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file1->base_rec.rank, file1->base_rec.id, bgq_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_BGQ_MOD],
+ file2->base_rec.rank, file2->base_rec.id, bgq_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
+
+ return;
+}
+
+
+static void darshan_log_agg_bgq_recs(void *rec, void *agg_rec, int init_flag)
+{
+ /* TODO: how would aggregation work for the BG/Q module ? */
+ return;
+}
+
/*
* Local variables:
* c-indent-level: 4
=====================================
darshan-util/darshan-bgq-logutils.h
=====================================
--- a/darshan-util/darshan-bgq-logutils.h
+++ b/darshan-util/darshan-bgq-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_BGQ_LOG_UTILS_H
#define __DARSHAN_BGQ_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-bgq-log-format.h"
-
extern char *bgq_counter_names[];
extern char *bgq_f_counter_names[];
=====================================
darshan-util/darshan-convert.c
=====================================
--- a/darshan-util/darshan-convert.c
+++ b/darshan-util/darshan-convert.c
@@ -375,7 +375,7 @@ int main(int argc, char **argv)
if(!hash || hash == base_rec->id)
{
- ret = mod_logutils[i]->log_put_record(outfile, mod_buf);
+ ret = mod_logutils[i]->log_put_record(outfile, mod_buf, infile->mod_ver[i]);
if(ret < 0)
{
darshan_log_close(infile);
=====================================
darshan-util/darshan-diff.c
=====================================
--- a/darshan-util/darshan-diff.c
+++ b/darshan-util/darshan-diff.c
@@ -193,7 +193,6 @@ int main(int argc, char *argv[])
for(i = 0; i < DARSHAN_MAX_MODS; i++)
{
-#if 0
/* TODO: skip modules that don't have the same format version, for now */
if(rec_ref1->mod_recs[i] && rec_ref2 && rec_ref2->mod_recs[i] &&
(file1->mod_ver[i] != file2->mod_ver[i]))
@@ -203,7 +202,6 @@ int main(int argc, char *argv[])
darshan_module_names[i], file1->mod_ver[i], file2->mod_ver[i]);
continue;
}
-#endif
while(1)
{
@@ -224,7 +222,6 @@ int main(int argc, char *argv[])
base_rec1 = (struct darshan_base_record *)mod_buf1;
base_rec2 = (struct darshan_base_record *)mod_buf2;
-
if(!base_rec1 && !base_rec2)
{
/* break out if there are no more records for this module */
@@ -307,8 +304,6 @@ int main(int argc, char *argv[])
*/
HASH_ITER(hlink, rec_hash2, rec_ref2, rec_tmp)
{
- printf("\n");
-
for(i = 0; i < DARSHAN_MAX_MODS; i++)
{
while(rec_ref2->mod_recs[i])
=====================================
darshan-util/darshan-hdf5-logutils.c
=====================================
--- a/darshan-util/darshan-hdf5-logutils.c
+++ b/darshan-util/darshan-hdf5-logutils.c
@@ -17,7 +17,7 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-hdf5-logutils.h"
+#include "darshan-logutils.h"
/* counter name strings for the HDF5 module */
#define X(a) #a,
@@ -31,9 +31,12 @@ char *hdf5_f_counter_names[] = {
#undef X
static int darshan_log_get_hdf5_file(darshan_fd fd, void* hdf5_buf);
-static int darshan_log_put_hdf5_file(darshan_fd fd, void* hdf5_buf);
+static int darshan_log_put_hdf5_file(darshan_fd fd, void* hdf5_buf, int ver);
static void darshan_log_print_hdf5_file(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_hdf5_description(void);
+static void darshan_log_print_hdf5_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2);
static void darshan_log_agg_hdf5_files(void *rec, void *agg_rec, int init_flag);
struct darshan_mod_logutil_funcs hdf5_logutils =
@@ -41,6 +44,8 @@ struct darshan_mod_logutil_funcs hdf5_logutils =
.log_get_record = &darshan_log_get_hdf5_file,
.log_put_record = &darshan_log_put_hdf5_file,
.log_print_record = &darshan_log_print_hdf5_file,
+ .log_print_description = &darshan_log_print_hdf5_description,
+ .log_print_diff = &darshan_log_print_hdf5_file_diff,
.log_agg_records = &darshan_log_agg_hdf5_files
};
@@ -74,13 +79,13 @@ static int darshan_log_get_hdf5_file(darshan_fd fd, void* hdf5_buf)
}
}
-static int darshan_log_put_hdf5_file(darshan_fd fd, void* hdf5_buf)
+static int darshan_log_put_hdf5_file(darshan_fd fd, void* hdf5_buf, int ver)
{
struct darshan_hdf5_file *file = (struct darshan_hdf5_file *)hdf5_buf;
int ret;
ret = darshan_log_putmod(fd, DARSHAN_HDF5_MOD, file,
- sizeof(struct darshan_hdf5_file));
+ sizeof(struct darshan_hdf5_file), ver);
if(ret < 0)
return(-1);
@@ -88,7 +93,7 @@ static int darshan_log_put_hdf5_file(darshan_fd fd, void* hdf5_buf)
}
static void darshan_log_print_hdf5_file(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_hdf5_file *hdf5_file_rec =
@@ -113,9 +118,92 @@ static void darshan_log_print_hdf5_file(void *file_rec, char *file_name,
return;
}
-static void darshan_log_agg_hdf5_files(void *rec, void *agg_rec, int init_flag)
+static void darshan_log_print_hdf5_description()
+{
+ printf("\n# description of HDF5 counters:\n");
+ printf("# HDF5_OPENS: HDF5 file open operation counts.\n");
+ printf("# HDF5_F_OPEN_TIMESTAMP: timestamp of first HDF5 file open.\n");
+ printf("# HDF5_F_CLOSE_TIMESTAMP: timestamp of last HDF5 file close.\n");
+
+ DARSHAN_PRINT_HEADER();
+
+ return;
+}
+
+static void darshan_log_print_hdf5_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
{
+ struct darshan_hdf5_file *file1 = (struct darshan_hdf5_file *)file_rec1;
+ struct darshan_hdf5_file *file2 = (struct darshan_hdf5_file *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+ for(i=0; i<HDF5_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file1->base_rec.rank, file1->base_rec.id, hdf5_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file2->base_rec.rank, file2->base_rec.id, hdf5_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file1->base_rec.rank, file1->base_rec.id, hdf5_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file2->base_rec.rank, file2->base_rec.id, hdf5_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<HDF5_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file1->base_rec.rank, file1->base_rec.id, hdf5_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file2->base_rec.rank, file2->base_rec.id, hdf5_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file1->base_rec.rank, file1->base_rec.id, hdf5_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_HDF5_MOD],
+ file2->base_rec.rank, file2->base_rec.id, hdf5_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
+
+ return;
+}
+
+static void darshan_log_agg_hdf5_files(void *rec, void *agg_rec, int init_flag)
+{
return;
}
=====================================
darshan-util/darshan-hdf5-logutils.h
=====================================
--- a/darshan-util/darshan-hdf5-logutils.h
+++ b/darshan-util/darshan-hdf5-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_HDF5_LOG_UTILS_H
#define __DARSHAN_HDF5_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-hdf5-log-format.h"
-
extern char *hdf5_counter_names[];
extern char *hdf5_f_counter_names[];
=====================================
darshan-util/darshan-logutils.c
=====================================
--- a/darshan-util/darshan-logutils.c
+++ b/darshan-util/darshan-logutils.c
@@ -93,7 +93,7 @@ static int darshan_log_noz_read(darshan_fd fd, struct darshan_log_map map,
void *buf, int len, int reset_strm_flag);
/* each module's implementation of the darshan logutil functions */
-#define X(a, b, c) c,
+#define X(a, b, c, d) d,
struct darshan_mod_logutil_funcs *mod_logutils[DARSHAN_MAX_MODS] =
{
DARSHAN_MODULE_IDS
@@ -139,6 +139,7 @@ darshan_fd darshan_log_open(const char *name)
ret = darshan_log_getheader(tmp_fd);
if(ret < 0)
{
+ fprintf(stderr, "Error: failed to read darshan log file header.\n");
close(tmp_fd->state->fildes);
free(tmp_fd->state);
free(tmp_fd);
@@ -728,7 +729,7 @@ int darshan_log_getmod(darshan_fd fd, darshan_module_id mod_id,
* returns number of bytes written on success, -1 on failure
*/
int darshan_log_putmod(darshan_fd fd, darshan_module_id mod_id,
- void *mod_buf, int mod_buf_sz)
+ void *mod_buf, int mod_buf_sz, int ver)
{
struct darshan_fd_int_state *state = fd->state;
int ret;
@@ -753,6 +754,9 @@ int darshan_log_putmod(darshan_fd fd, darshan_module_id mod_id,
return(-1);
}
+ /* set the version number for this module's data */
+ fd->mod_ver[mod_id] = ver;
+
return(0);
}
@@ -839,6 +843,30 @@ static int darshan_log_getheader(darshan_fd fd)
return(-1);
}
+ /* read the version number so we know how to process this log */
+ ret = darshan_log_read(fd, &fd->version, 8);
+ if(ret < 8)
+ {
+ fprintf(stderr, "Error: invalid log file (failed to read version).\n");
+ return(-1);
+ }
+
+ /* other log file versions can be detected and handled here */
+ if(strcmp(fd->version, "3.00"))
+ {
+ fprintf(stderr, "Error: incompatible darshan file.\n");
+ fprintf(stderr, "Error: expected version %s\n", DARSHAN_LOG_VERSION);
+ return(-1);
+ }
+
+ /* seek back so we can read the entire header */
+ ret = darshan_log_seek(fd, 0);
+ if(ret < 0)
+ {
+ fprintf(stderr, "Error: unable to seek in darshan log file.\n");
+ return(-1);
+ }
+
/* read uncompressed header from log file */
ret = darshan_log_read(fd, &header, sizeof(header));
if(ret != (int)sizeof(header))
@@ -847,9 +875,6 @@ static int darshan_log_getheader(darshan_fd fd)
return(-1);
}
- /* save the version string */
- strncpy(fd->version, header.version_string, 8);
-
if(header.magic_nr == DARSHAN_MAGIC_NR)
{
/* no byte swapping needed, this file is in host format already */
@@ -880,8 +905,10 @@ static int darshan_log_getheader(darshan_fd fd)
}
}
+ /* set some fd fields based on what's stored in the header */
fd->comp_type = header.comp_type;
fd->partial_flag = header.partial_flag;
+ memcpy(fd->mod_ver, header.mod_ver, DARSHAN_MAX_MODS * sizeof(uint32_t));
/* save the mapping of data within log file to this file descriptor */
memcpy(&fd->rec_map, &(header.rec_map), sizeof(struct darshan_log_map));
=====================================
darshan-util/darshan-logutils.h
=====================================
--- a/darshan-util/darshan-logutils.h
+++ b/darshan-util/darshan-logutils.h
@@ -35,6 +35,8 @@ struct darshan_fd_s
struct darshan_log_map job_map;
struct darshan_log_map rec_map;
struct darshan_log_map mod_map[DARSHAN_MAX_MODS];
+ /* module-specific log-format versions contained in log */
+ uint32_t mod_ver[DARSHAN_MAX_MODS];
/* KEEP OUT -- remaining state hidden in logutils source */
struct darshan_fd_int_state *state;
@@ -56,6 +58,9 @@ struct darshan_mod_logutil_funcs
/* retrieve a single module record from the log file.
* return 1 on successful read of record, 0 on no more
* module data, -1 on error
+ * - 'fd' is the file descriptor to get record from
+ * - 'buf' is the buffer to store the record in
+ * - 'rec_id' is the corresponding darshan record id
*/
int (*log_get_record)(
darshan_fd fd,
@@ -63,31 +68,44 @@ struct darshan_mod_logutil_funcs
);
/* put a single module record into the log file.
* return 0 on success, -1 on error
+ * - 'fd' is the file descriptor to put record into
+ * - 'buf' is the buffer containing the record data
+ * - 'rec_id' is the corresponding darshan record id
*/
int (*log_put_record)(
darshan_fd fd,
- void *buf
+ void *buf,
+ int ver
);
- /* print the counters for a given log file record */
+ /* print the counters for a given log record
+ * - 'file_rec' is the record's data buffer
+ * - 'file_name' is the file path string for the record
+ * - 'mnt-pt' is the file path mount point string
+ * - 'fs_type' is the file system type string
+ * - 'ver' is the version of the record
+ */
void (*log_print_record)(
void *file_rec,
char *file_name,
char *mnt_pt,
- char *fs_type
- );
- /* combine two records into a single aggregate record */
- void (*log_agg_records)(
- void *rec,
- void *agg_rec,
- int init_flag
+ char *fs_type,
+ int ver
);
- /* print a text diff of 2 module I/O records */
+ /* print module-specific description of I/O characterization data */
+ void (*log_print_description)(void);
+ /* print a text diff of 2 module I/O records */
void (*log_print_diff)(
void *rec1,
char *name1,
void *rec2,
char *name2
);
+ /* combine two records into a single aggregate record */
+ void (*log_agg_records)(
+ void *rec,
+ void *agg_rec,
+ int init_flag
+ );
};
extern struct darshan_mod_logutil_funcs *mod_logutils[];
@@ -114,7 +132,7 @@ int darshan_log_puthash(darshan_fd fd, struct darshan_record_ref *hash);
int darshan_log_getmod(darshan_fd fd, darshan_module_id mod_id,
void *mod_buf, int mod_buf_sz);
int darshan_log_putmod(darshan_fd fd, darshan_module_id mod_id,
- void *mod_buf, int mod_buf_sz);
+ void *mod_buf, int mod_buf_sz, int ver);
void darshan_log_close(darshan_fd file);
/* convenience macros for printing Darshan counters */
=====================================
darshan-util/darshan-mpiio-logutils.c
=====================================
--- a/darshan-util/darshan-mpiio-logutils.c
+++ b/darshan-util/darshan-mpiio-logutils.c
@@ -17,7 +17,7 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-mpiio-logutils.h"
+#include "darshan-logutils.h"
/* counter name strings for the MPI-IO module */
#define X(a) #a,
@@ -31,9 +31,12 @@ char *mpiio_f_counter_names[] = {
#undef X
static int darshan_log_get_mpiio_file(darshan_fd fd, void* mpiio_buf);
-static int darshan_log_put_mpiio_file(darshan_fd fd, void* mpiio_buf);
+static int darshan_log_put_mpiio_file(darshan_fd fd, void* mpiio_buf, int ver);
static void darshan_log_print_mpiio_file(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_mpiio_description(void);
+static void darshan_log_print_mpiio_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2);
static void darshan_log_agg_mpiio_files(void *rec, void *agg_rec, int init_flag);
struct darshan_mod_logutil_funcs mpiio_logutils =
@@ -41,6 +44,8 @@ struct darshan_mod_logutil_funcs mpiio_logutils =
.log_get_record = &darshan_log_get_mpiio_file,
.log_put_record = &darshan_log_put_mpiio_file,
.log_print_record = &darshan_log_print_mpiio_file,
+ .log_print_description = &darshan_log_print_mpiio_description,
+ .log_print_diff = &darshan_log_print_mpiio_file_diff,
.log_agg_records = &darshan_log_agg_mpiio_files
};
@@ -74,13 +79,13 @@ static int darshan_log_get_mpiio_file(darshan_fd fd, void* mpiio_buf)
}
}
-static int darshan_log_put_mpiio_file(darshan_fd fd, void* mpiio_buf)
+static int darshan_log_put_mpiio_file(darshan_fd fd, void* mpiio_buf, int ver)
{
struct darshan_mpiio_file *file = (struct darshan_mpiio_file *)mpiio_buf;
int ret;
ret = darshan_log_putmod(fd, DARSHAN_MPIIO_MOD, file,
- sizeof(struct darshan_mpiio_file));
+ sizeof(struct darshan_mpiio_file), ver);
if(ret < 0)
return(-1);
@@ -88,7 +93,7 @@ static int darshan_log_put_mpiio_file(darshan_fd fd, void* mpiio_buf)
}
static void darshan_log_print_mpiio_file(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_mpiio_file *mpiio_file_rec =
@@ -113,12 +118,117 @@ static void darshan_log_print_mpiio_file(void *file_rec, char *file_name,
return;
}
-static void darshan_log_agg_mpiio_files(void *rec, void *agg_rec, int init_flag)
+static void darshan_log_print_mpiio_description()
{
+ printf("\n# description of MPIIO counters:\n");
+ printf("# MPIIO_INDEP_*: MPI independent operation counts.\n");
+ printf("# MPIIO_COLL_*: MPI collective operation counts.\n");
+ printf("# MPIIO_SPLIT_*: MPI split collective operation counts.\n");
+ printf("# MPIIO_NB_*: MPI non blocking operation counts.\n");
+ printf("# READS,WRITES,and OPENS are types of operations.\n");
+ printf("# MPIIO_SYNCS: MPI file sync operation counts.\n");
+ printf("# MPIIO_HINTS: number of times MPI hints were used.\n");
+ printf("# MPIIO_VIEWS: number of times MPI file views were used.\n");
+ printf("# MPIIO_MODE: MPI-IO access mode that file was opened with.\n");
+ printf("# MPIIO_BYTES_*: total bytes read and written at MPI-IO layer.\n");
+ printf("# MPIIO_RW_SWITCHES: number of times access alternated between read and write.\n");
+ printf("# MPIIO_MAX_*_TIME_SIZE: size of the slowest read and write operations.\n");
+ printf("# MPIIO_SIZE_*_AGG_*: histogram of MPI datatype total sizes for read and write operations.\n");
+ printf("# MPIIO_ACCESS*_ACCESS: the four most common total access sizes.\n");
+ printf("# MPIIO_ACCESS*_COUNT: count of the four most common total access sizes.\n");
+ printf("# MPIIO_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).\n");
+ printf("# MPIIO_*_RANK_BYTES: total bytes transferred at MPI-IO layer by the fastest and slowest ranks (for shared files).\n");
+ printf("# MPIIO_F_OPEN_TIMESTAMP: timestamp of first open.\n");
+ printf("# MPIIO_F_*_START_TIMESTAMP: timestamp of first MPI-IO read/write.\n");
+ printf("# MPIIO_F_*_END_TIMESTAMP: timestamp of last MPI-IO read/write.\n");
+ printf("# MPIIO_F_CLOSE_TIMESTAMP: timestamp of last close.\n");
+ printf("# MPIIO_F_READ/WRITE/META_TIME: cumulative time spent in MPI-IO read, write, or metadata operations.\n");
+ printf("# MPIIO_F_MAX_*_TIME: duration of the slowest MPI-IO read and write operations.\n");
+ printf("# MPIIO_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).\n");
+ printf("# MPIIO_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).\n");
+
+ DARSHAN_PRINT_HEADER();
+
+ return;
+}
+
+static void darshan_log_print_mpiio_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
+{
+ struct darshan_mpiio_file *file1 = (struct darshan_mpiio_file *)file_rec1;
+ struct darshan_mpiio_file *file2 = (struct darshan_mpiio_file *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+
+ for(i=0; i<MPIIO_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file1->base_rec.rank, file1->base_rec.id, mpiio_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file2->base_rec.rank, file2->base_rec.id, mpiio_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file1->base_rec.rank, file1->base_rec.id, mpiio_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file2->base_rec.rank, file2->base_rec.id, mpiio_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<MPIIO_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file1->base_rec.rank, file1->base_rec.id, mpiio_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file2->base_rec.rank, file2->base_rec.id, mpiio_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file1->base_rec.rank, file1->base_rec.id, mpiio_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_MPIIO_MOD],
+ file2->base_rec.rank, file2->base_rec.id, mpiio_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
return;
}
+static void darshan_log_agg_mpiio_files(void *rec, void *agg_rec, int init_flag)
+{
+ return;
+}
+
/*
* Local variables:
* c-indent-level: 4
=====================================
darshan-util/darshan-mpiio-logutils.h
=====================================
--- a/darshan-util/darshan-mpiio-logutils.h
+++ b/darshan-util/darshan-mpiio-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_MPIIO_LOG_UTILS_H
#define __DARSHAN_MPIIO_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-mpiio-log-format.h"
-
extern char *mpiio_counter_names[];
extern char *mpiio_f_counter_names[];
=====================================
darshan-util/darshan-null-logutils.c
=====================================
--- a/darshan-util/darshan-null-logutils.c
+++ b/darshan-util/darshan-null-logutils.c
@@ -17,7 +17,7 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-null-logutils.h"
+#include "darshan-logutils.h"
/* integer counter name strings for the NULL module */
#define X(a) #a,
@@ -33,9 +33,12 @@ char *null_f_counter_names[] = {
/* prototypes for each of the NULL module's logutil functions */
static int darshan_log_get_null_record(darshan_fd fd, void* null_buf);
-static int darshan_log_put_null_record(darshan_fd fd, void* null_buf);
+static int darshan_log_put_null_record(darshan_fd fd, void* null_buf, int ver);
static void darshan_log_print_null_record(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_null_description(void);
+static void darshan_log_print_null_record_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2);
static void darshan_log_agg_null_records(void *rec, void *agg_rec, int init_flag);
/* structure storing each function needed for implementing the darshan
@@ -47,12 +50,15 @@ struct darshan_mod_logutil_funcs null_logutils =
.log_get_record = &darshan_log_get_null_record,
.log_put_record = &darshan_log_put_null_record,
.log_print_record = &darshan_log_print_null_record,
+ .log_print_description = &darshan_log_print_null_description,
+ .log_print_diff = &darshan_log_print_null_record_diff,
.log_agg_records = &darshan_log_agg_null_records
};
/* retrieve a NULL record from log file descriptor 'fd', storing the
* buffer in 'null_buf' and the corresponding Darshan record id in
- * 'rec_id'. Return 1 on successful record read, .
+ * 'rec_id'. Return 1 on successful record read, 0 on no more data,
+ * and -1 on error.
*/
static int darshan_log_get_null_record(darshan_fd fd, void* null_buf)
{
@@ -89,14 +95,14 @@ static int darshan_log_get_null_record(darshan_fd fd, void* null_buf)
/* write the NULL record stored in 'null_buf' to log file descriptor 'fd'.
* Return 0 on success, -1 on failure
*/
-static int darshan_log_put_null_record(darshan_fd fd, void* null_buf)
+static int darshan_log_put_null_record(darshan_fd fd, void* null_buf, int ver)
{
struct darshan_null_record *rec = (struct darshan_null_record *)null_buf;
int ret;
/* append NULL record to darshan log file */
ret = darshan_log_putmod(fd, DARSHAN_NULL_MOD, rec,
- sizeof(struct darshan_null_record));
+ sizeof(struct darshan_null_record), ver);
if(ret < 0)
return(-1);
@@ -105,7 +111,7 @@ static int darshan_log_put_null_record(darshan_fd fd, void* null_buf)
/* print all I/O data record statistics for the given NULL record */
static void darshan_log_print_null_record(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_null_record *null_rec =
@@ -133,12 +139,95 @@ static void darshan_log_print_null_record(void *file_rec, char *file_name,
return;
}
-static void darshan_log_agg_null_records(void *rec, void *agg_rec, int init_flag)
+/* print out a description of the NULL module record fields */
+static void darshan_log_print_null_description()
{
+ printf("\n# description of NULL counters:\n");
+ printf("# NULL_BARS: number of 'bar' function calls.\n");
+ printf("# NULL_BAR_DAT: value set by last call to function 'bar'.\n");
+ printf("# NULL_F_BAR_TIMESTAMP: timestamp of the first call to function 'bar'.\n");
+ printf("# NULL_F_BAR_DURATION: duration of the last call to function 'bar'.\n");
+
+ return;
+}
+
+static void darshan_log_print_null_record_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
+{
+ struct darshan_null_record *file1 = (struct darshan_null_record *)file_rec1;
+ struct darshan_null_record *file2 = (struct darshan_null_record *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+
+ for(i=0; i<NULL_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file1->base_rec.rank, file1->base_rec.id, null_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file2->base_rec.rank, file2->base_rec.id, null_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file1->base_rec.rank, file1->base_rec.id, null_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file2->base_rec.rank, file2->base_rec.id, null_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<NULL_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file1->base_rec.rank, file1->base_rec.id, null_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file2->base_rec.rank, file2->base_rec.id, null_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file1->base_rec.rank, file1->base_rec.id, null_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_NULL_MOD],
+ file2->base_rec.rank, file2->base_rec.id, null_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
return;
}
+static void darshan_log_agg_null_records(void *rec, void *agg_rec, int init_flag)
+{
+ return;
+}
+
/*
* Local variables:
* c-indent-level: 4
=====================================
darshan-util/darshan-null-logutils.h
=====================================
--- a/darshan-util/darshan-null-logutils.h
+++ b/darshan-util/darshan-null-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_NULL_LOG_UTILS_H
#define __DARSHAN_NULL_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-null-log-format.h"
-
/* declare NULL module counter name strings and logutil definition as
* extern variables so they can be used in other utilities
*/
=====================================
darshan-util/darshan-parser.c
=====================================
--- a/darshan-util/darshan-parser.c
+++ b/darshan-util/darshan-parser.c
@@ -313,17 +313,17 @@ int main(int argc, char **argv)
}
/* print breakdown of each log file region's contribution to file size */
- printf("\n# log file region sizes\n");
+ printf("\n# log file regions\n");
printf("# -------------------------------------------------------\n");
- printf("# header: %zu bytes\n", sizeof(struct darshan_header));
- printf("# job data: %zu bytes\n", fd->job_map.len);
- printf("# record table: %zu bytes\n", fd->rec_map.len);
+ printf("# header: %zu bytes (uncompressed)\n", sizeof(struct darshan_header));
+ printf("# job data: %zu bytes (compressed)\n", fd->job_map.len);
+ printf("# record table: %zu bytes (compressed)\n", fd->rec_map.len);
for(i=0; i<DARSHAN_MAX_MODS; i++)
{
if(fd->mod_map[i].len)
{
- printf("# %s module: %zu bytes\n", darshan_module_names[i],
- fd->mod_map[i].len);
+ printf("# %s module: %zu bytes (compressed), ver=%d\n",
+ darshan_module_names[i], fd->mod_map[i].len, fd->mod_ver[i]);
}
}
@@ -335,6 +335,21 @@ int main(int argc, char **argv)
printf("# mount entry:\t%s\t%s\n", mnt_pts[i], fs_types[i]);
}
+ if(mask & OPTION_BASE)
+ {
+ printf("\n# description of columns:\n");
+ printf("# <module>: module responsible for this I/O record.\n");
+ printf("# <rank>: MPI rank. -1 indicates that the file is shared\n");
+ printf("# across all processes and statistics are aggregated.\n");
+ printf("# <record id>: hash of the record's file path\n");
+ printf("# <counter name> and <counter value>: statistical counters.\n");
+ printf("# A value of -1 indicates that Darshan could not monitor\n");
+ printf("# that counter, and its value should be ignored.\n");
+ printf("# <file name>: full file path for the record.\n");
+ printf("# <mount pt>: mount point that the file resides on.\n");
+ printf("# <fs type>: type of file system that the file resides on.\n");
+ }
+
/* warn user if this log file is incomplete */
pdata.rank_cumul_io_time = malloc(sizeof(double)*job.nprocs);
pdata.rank_cumul_md_time = malloc(sizeof(double)*job.nprocs);
@@ -388,8 +403,9 @@ int main(int argc, char **argv)
if(mask & OPTION_BASE)
{
- /* TODO: does each module print header of what each counter means??? */
- DARSHAN_PRINT_HEADER();
+ /* print a header describing the module's I/O characterization data */
+ if(mod_logutils[i]->log_print_description)
+ mod_logutils[i]->log_print_description();
}
ret = mod_logutils[i]->log_get_record(fd, mod_buf);
@@ -432,7 +448,7 @@ int main(int argc, char **argv)
{
/* print the corresponding module data for this record */
mod_logutils[i]->log_print_record(mod_buf, ref->name,
- mnt_pt, fs_type);
+ mnt_pt, fs_type, fd->mod_ver[i]);
}
/* we calculate more detailed stats for POSIX and MPI-IO modules,
@@ -705,8 +721,7 @@ void posix_accum_file(struct darshan_posix_file *pfile,
case POSIX_MODE:
case POSIX_MEM_ALIGNMENT:
case POSIX_FILE_ALIGNMENT:
- if(POSIX_FILE_PARTIAL(tmp))
- tmp->counters[i] = pfile->counters[i];
+ tmp->counters[i] = pfile->counters[i];
break;
case POSIX_MAX_BYTE_READ:
case POSIX_MAX_BYTE_WRITTEN:
=====================================
darshan-util/darshan-pnetcdf-logutils.c
=====================================
--- a/darshan-util/darshan-pnetcdf-logutils.c
+++ b/darshan-util/darshan-pnetcdf-logutils.c
@@ -17,7 +17,7 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-pnetcdf-logutils.h"
+#include "darshan-logutils.h"
/* counter name strings for the PNETCDF module */
#define X(a) #a,
@@ -31,9 +31,12 @@ char *pnetcdf_f_counter_names[] = {
#undef X
static int darshan_log_get_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf);
-static int darshan_log_put_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf);
+static int darshan_log_put_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf, int ver);
static void darshan_log_print_pnetcdf_file(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_pnetcdf_description(void);
+static void darshan_log_print_pnetcdf_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2);
static void darshan_log_agg_pnetcdf_files(void *rec, void *agg_rec, int init_flag);
struct darshan_mod_logutil_funcs pnetcdf_logutils =
@@ -41,6 +44,8 @@ struct darshan_mod_logutil_funcs pnetcdf_logutils =
.log_get_record = &darshan_log_get_pnetcdf_file,
.log_put_record = &darshan_log_put_pnetcdf_file,
.log_print_record = &darshan_log_print_pnetcdf_file,
+ .log_print_description = &darshan_log_print_pnetcdf_description,
+ .log_print_diff = &darshan_log_print_pnetcdf_file_diff,
.log_agg_records = &darshan_log_agg_pnetcdf_files
};
@@ -74,13 +79,13 @@ static int darshan_log_get_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf)
}
}
-static int darshan_log_put_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf)
+static int darshan_log_put_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf, int ver)
{
struct darshan_pnetcdf_file *file = (struct darshan_pnetcdf_file *)pnetcdf_buf;
int ret;
ret = darshan_log_putmod(fd, DARSHAN_PNETCDF_MOD, file,
- sizeof(struct darshan_pnetcdf_file));
+ sizeof(struct darshan_pnetcdf_file), ver);
if(ret < 0)
return(-1);
@@ -88,7 +93,7 @@ static int darshan_log_put_pnetcdf_file(darshan_fd fd, void* pnetcdf_buf)
}
static void darshan_log_print_pnetcdf_file(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_pnetcdf_file *pnetcdf_file_rec =
@@ -113,9 +118,93 @@ static void darshan_log_print_pnetcdf_file(void *file_rec, char *file_name,
return;
}
-static void darshan_log_agg_pnetcdf_files(void *rec, void *agg_rec, int init_flag)
+static void darshan_log_print_pnetcdf_description()
+{
+ printf("\n# description of PNETCDF counters:\n");
+ printf("# PNETCDF_INDEP_OPENS: PNETCDF independent file open operation counts.\n");
+ printf("# PNETCDF_COLL_OPENS: PNETCDF collective file open operation counts.\n");
+ printf("# PNETCDF_F_OPEN_TIMESTAMP: timestamp of first PNETCDF file open.\n");
+ printf("# PNETCDF_F_CLOSE_TIMESTAMP: timestamp of last PNETCDF file close.\n");
+
+ DARSHAN_PRINT_HEADER();
+
+ return;
+}
+
+static void darshan_log_print_pnetcdf_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
{
+ struct darshan_pnetcdf_file *file1 = (struct darshan_pnetcdf_file *)file_rec1;
+ struct darshan_pnetcdf_file *file2 = (struct darshan_pnetcdf_file *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+ for(i=0; i<PNETCDF_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file1->base_rec.rank, file1->base_rec.id, pnetcdf_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file2->base_rec.rank, file2->base_rec.id, pnetcdf_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file1->base_rec.rank, file1->base_rec.id, pnetcdf_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file2->base_rec.rank, file2->base_rec.id, pnetcdf_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<PNETCDF_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file1->base_rec.rank, file1->base_rec.id, pnetcdf_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file2->base_rec.rank, file2->base_rec.id, pnetcdf_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file1->base_rec.rank, file1->base_rec.id, pnetcdf_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_PNETCDF_MOD],
+ file2->base_rec.rank, file2->base_rec.id, pnetcdf_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
+
+ return;
+}
+
+static void darshan_log_agg_pnetcdf_files(void *rec, void *agg_rec, int init_flag)
+{
return;
}
=====================================
darshan-util/darshan-pnetcdf-logutils.h
=====================================
--- a/darshan-util/darshan-pnetcdf-logutils.h
+++ b/darshan-util/darshan-pnetcdf-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_PNETCDF_LOG_UTILS_H
#define __DARSHAN_PNETCDF_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-pnetcdf-log-format.h"
-
extern char *pnetcdf_counter_names[];
extern char *pnetcdf_f_counter_names[];
=====================================
darshan-util/darshan-posix-logutils.c
=====================================
--- a/darshan-util/darshan-posix-logutils.c
+++ b/darshan-util/darshan-posix-logutils.c
@@ -17,7 +17,7 @@
#include <fcntl.h>
#include <errno.h>
-#include "darshan-posix-logutils.h"
+#include "darshan-logutils.h"
/* counter name strings for the POSIX module */
#define X(a) #a,
@@ -31,20 +31,22 @@ char *posix_f_counter_names[] = {
#undef X
static int darshan_log_get_posix_file(darshan_fd fd, void* posix_buf);
-static int darshan_log_put_posix_file(darshan_fd fd, void* posix_buf);
+static int darshan_log_put_posix_file(darshan_fd fd, void* posix_buf, int ver);
static void darshan_log_print_posix_file(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
-static void darshan_log_agg_posix_files(void *rec, void *agg_rec, int init_flag);
+ char *file_name, char *mnt_pt, char *fs_type, int ver);
+static void darshan_log_print_posix_description(void);
static void darshan_log_print_posix_file_diff(void *file_rec1, char *file_name1,
void *file_rec2, char *file_name2);
+static void darshan_log_agg_posix_files(void *rec, void *agg_rec, int init_flag);
struct darshan_mod_logutil_funcs posix_logutils =
{
.log_get_record = &darshan_log_get_posix_file,
.log_put_record = &darshan_log_put_posix_file,
.log_print_record = &darshan_log_print_posix_file,
+ .log_print_description = &darshan_log_print_posix_description,
+ .log_print_diff = &darshan_log_print_posix_file_diff,
.log_agg_records = &darshan_log_agg_posix_files,
- .log_print_diff = &darshan_log_print_posix_file_diff
};
static int darshan_log_get_posix_file(darshan_fd fd, void* posix_buf)
@@ -77,13 +79,13 @@ static int darshan_log_get_posix_file(darshan_fd fd, void* posix_buf)
}
}
-static int darshan_log_put_posix_file(darshan_fd fd, void* posix_buf)
+static int darshan_log_put_posix_file(darshan_fd fd, void* posix_buf, int ver)
{
struct darshan_posix_file *file = (struct darshan_posix_file *)posix_buf;
int ret;
ret = darshan_log_putmod(fd, DARSHAN_POSIX_MOD, file,
- sizeof(struct darshan_posix_file));
+ sizeof(struct darshan_posix_file), ver);
if(ret < 0)
return(-1);
@@ -91,7 +93,7 @@ static int darshan_log_put_posix_file(darshan_fd fd, void* posix_buf)
}
static void darshan_log_print_posix_file(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type)
+ char *mnt_pt, char *fs_type, int ver)
{
int i;
struct darshan_posix_file *posix_file_rec =
@@ -116,6 +118,113 @@ static void darshan_log_print_posix_file(void *file_rec, char *file_name,
return;
}
+static void darshan_log_print_posix_description()
+{
+ printf("\n# description of POSIX counters:\n");
+ printf("# POSIX_*: posix operation counts.\n");
+ printf("# READS,WRITES,OPENS,SEEKS,STATS, and MMAPS are types of operations.\n");
+ printf("# POSIX_MODE: mode that file was opened in.\n");
+ printf("# POSIX_BYTES_*: total bytes read and written.\n");
+ printf("# POSIX_MAX_BYTE_*: highest offset byte read and written.\n");
+ printf("# POSIX_CONSEC_*: number of exactly adjacent reads and writes.\n");
+ printf("# POSIX_SEQ_*: number of reads and writes from increasing offsets.\n");
+ printf("# POSIX_RW_SWITCHES: number of times access alternated between read and write.\n");
+ printf("# POSIX_*_ALIGNMENT: memory and file alignment.\n");
+ printf("# POSIX_*_NOT_ALIGNED: number of reads and writes that were not aligned.\n");
+ printf("# POSIX_MAX_*_TIME_SIZE: size of the slowest read and write operations.\n");
+ printf("# POSIX_SIZE_*_*: histogram of read and write access sizes.\n");
+ printf("# POSIX_STRIDE*_STRIDE: the four most common strides detected.\n");
+ printf("# POSIX_STRIDE*_COUNT: count of the four most common strides.\n");
+ printf("# POSIX_ACCESS*_ACCESS: the four most common access sizes.\n");
+ printf("# POSIX_ACCESS*_COUNT: count of the four most common access sizes.\n");
+ printf("# POSIX_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).\n");
+ printf("# POSIX_*_RANK_BYTES: bytes transferred by the fastest and slowest ranks (for shared files).\n");
+ printf("# POSIX_F_OPEN_TIMESTAMP: timestamp of first open.\n");
+ printf("# POSIX_F_*_START_TIMESTAMP: timestamp of first read/write.\n");
+ printf("# POSIX_F_*_END_TIMESTAMP: timestamp of last read/write.\n");
+ printf("# POSIX_F_CLOSE_TIMESTAMP: timestamp of last close.\n");
+ printf("# POSIX_F_READ/WRITE/META_TIME: cumulative time spent in read, write, or metadata operations.\n");
+ printf("# POSIX_F_MAX_*_TIME: duration of the slowest read and write operations.\n");
+ printf("# POSIX_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).\n");
+ printf("# POSIX_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).\n");
+
+ DARSHAN_PRINT_HEADER();
+
+ return;
+}
+
+static void darshan_log_print_posix_file_diff(void *file_rec1, char *file_name1,
+ void *file_rec2, char *file_name2)
+{
+ struct darshan_posix_file *file1 = (struct darshan_posix_file *)file_rec1;
+ struct darshan_posix_file *file2 = (struct darshan_posix_file *)file_rec2;
+ int i;
+
+ /* NOTE: we assume that both input records are the same module format version */
+
+ for(i=0; i<POSIX_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file1->base_rec.rank, file1->base_rec.id, posix_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file2->base_rec.rank, file2->base_rec.id, posix_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ else if(file1->counters[i] != file2->counters[i])
+ {
+ printf("- ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file1->base_rec.rank, file1->base_rec.id, posix_counter_names[i],
+ file1->counters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file2->base_rec.rank, file2->base_rec.id, posix_counter_names[i],
+ file2->counters[i], file_name2, "", "");
+ }
+ }
+
+ for(i=0; i<POSIX_F_NUM_INDICES; i++)
+ {
+ if(!file2)
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file1->base_rec.rank, file1->base_rec.id, posix_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+
+ }
+ else if(!file1)
+ {
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file2->base_rec.rank, file2->base_rec.id, posix_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ else if(file1->fcounters[i] != file2->fcounters[i])
+ {
+ printf("- ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file1->base_rec.rank, file1->base_rec.id, posix_f_counter_names[i],
+ file1->fcounters[i], file_name1, "", "");
+ printf("+ ");
+ DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
+ file2->base_rec.rank, file2->base_rec.id, posix_f_counter_names[i],
+ file2->fcounters[i], file_name2, "", "");
+ }
+ }
+
+ return;
+}
+
static void darshan_log_agg_posix_files(void *rec, void *agg_rec, int init_flag)
{
struct darshan_posix_file *psx_rec = (struct darshan_posix_file *)rec;
@@ -300,78 +409,6 @@ static void darshan_log_agg_posix_files(void *rec, void *agg_rec, int init_flag)
return;
}
-static void darshan_log_print_posix_file_diff(void *file_rec1, char *file_name1,
- void *file_rec2, char *file_name2)
-{
- struct darshan_posix_file *file1 = (struct darshan_posix_file *)file_rec1;
- struct darshan_posix_file *file2 = (struct darshan_posix_file *)file_rec2;
- int i;
-
- /* NOTE: we assume that both input records are the same module format version */
-
- for(i=0; i<POSIX_NUM_INDICES; i++)
- {
- if(!file2)
- {
- printf("- ");
- DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file1->base_rec.rank, file1->base_rec.id, posix_counter_names[i],
- file1->counters[i], file_name1, "", "");
-
- }
- else if(!file1)
- {
- printf("+ ");
- DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file2->base_rec.rank, file2->base_rec.id, posix_counter_names[i],
- file2->counters[i], file_name2, "", "");
- }
- else if(file1->counters[i] != file2->counters[i])
- {
- printf("- ");
- DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file1->base_rec.rank, file1->base_rec.id, posix_counter_names[i],
- file1->counters[i], file_name1, "", "");
- printf("+ ");
- DARSHAN_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file2->base_rec.rank, file2->base_rec.id, posix_counter_names[i],
- file2->counters[i], file_name2, "", "");
- }
- }
-
- for(i=0; i<POSIX_F_NUM_INDICES; i++)
- {
- if(!file2)
- {
- printf("- ");
- DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file1->base_rec.rank, file1->base_rec.id, posix_f_counter_names[i],
- file1->fcounters[i], file_name1, "", "");
-
- }
- else if(!file1)
- {
- printf("+ ");
- DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file2->base_rec.rank, file2->base_rec.id, posix_f_counter_names[i],
- file2->fcounters[i], file_name2, "", "");
- }
- else if(file1->fcounters[i] != file2->fcounters[i])
- {
- printf("- ");
- DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file1->base_rec.rank, file1->base_rec.id, posix_f_counter_names[i],
- file1->fcounters[i], file_name1, "", "");
- printf("+ ");
- DARSHAN_F_COUNTER_PRINT(darshan_module_names[DARSHAN_POSIX_MOD],
- file2->base_rec.rank, file2->base_rec.id, posix_f_counter_names[i],
- file2->fcounters[i], file_name2, "", "");
- }
- }
-
- return;
-}
-
/*
* Local variables:
* c-indent-level: 4
=====================================
darshan-util/darshan-posix-logutils.h
=====================================
--- a/darshan-util/darshan-posix-logutils.h
+++ b/darshan-util/darshan-posix-logutils.h
@@ -7,9 +7,6 @@
#ifndef __DARSHAN_POSIX_LOG_UTILS_H
#define __DARSHAN_POSIX_LOG_UTILS_H
-#include "darshan-logutils.h"
-#include "darshan-posix-log-format.h"
-
extern char *posix_counter_names[];
extern char *posix_f_counter_names[];
=====================================
darshan-util/darshan-stitch-logs.c
=====================================
--- a/darshan-util/darshan-stitch-logs.c
+++ b/darshan-util/darshan-stitch-logs.c
@@ -404,7 +404,7 @@ int main(int argc, char *argv[])
/* iterate over active darshan modules and gather module data to write
* to the stitched together output log
*/
- for(i = 0; i < DARSHAN_MPIIO_MOD; i++)
+ for(i = 0; i < DARSHAN_MAX_MODS; i++)
{
if(!mod_logutils[i]) continue;
@@ -424,21 +424,6 @@ int main(int argc, char *argv[])
return(-1);
}
- /* write out the shared records first */
- HASH_ITER(hlink, shared_rec_hash, sref, stmp)
- {
- ret = mod_logutils[i]->log_put_record(stitch_fd, sref->agg_rec);
- if(ret < 0)
- {
- fprintf(stderr,
- "Error: unable to write %s module record to output darshan log.\n",
- darshan_module_names[i]);
- globfree(&globbuf);
- darshan_log_close(stitch_fd);
- unlink(stitch_logname);
- return(-1);
- }
- }
}
for(j = 0; j < globbuf.gl_pathc; j++)
@@ -456,6 +441,25 @@ int main(int argc, char *argv[])
return(-1);
}
+ if(j == 0 && shared_rec_hash)
+ {
+ /* write out the shared records first */
+ HASH_ITER(hlink, shared_rec_hash, sref, stmp)
+ {
+ ret = mod_logutils[i]->log_put_record(stitch_fd, sref->agg_rec, in_fd->mod_ver[i]);
+ if(ret < 0)
+ {
+ fprintf(stderr,
+ "Error: unable to write %s module record to output darshan log.\n",
+ darshan_module_names[i]);
+ globfree(&globbuf);
+ darshan_log_close(stitch_fd);
+ unlink(stitch_logname);
+ return(-1);
+ }
+ }
+ }
+
/* loop over module records and write them to output file */
while((ret = mod_logutils[i]->log_get_record(in_fd, mod_buf)) == 1)
{
@@ -465,7 +469,7 @@ int main(int argc, char *argv[])
if(sref)
continue; /* skip shared records */
- ret = mod_logutils[i]->log_put_record(stitch_fd, mod_buf);
+ ret = mod_logutils[i]->log_put_record(stitch_fd, mod_buf, in_fd->mod_ver[i]);
if(ret < 0)
{
fprintf(stderr,
=====================================
darshan-util/doc/darshan-util.txt
=====================================
--- a/darshan-util/doc/darshan-util.txt
+++ b/darshan-util/doc/darshan-util.txt
@@ -66,12 +66,16 @@ application will likely be found in a centralized directory, with the path
and log file name in the following format:
----
-<YEAR>/<MONTH>/<DAY>/<USERNAME>_<BINARY_NAME>_<JOB_ID>_<DATE>_<UNIQUE_ID>_<TIMING>.darshan.gz
+<YEAR>/<MONTH>/<DAY>/<USERNAME>_<BINARY_NAME>_<JOB_ID>_<DATE>_<UNIQUE_ID>_<TIMING>.darshan
----
This is a binary format file that summarizes I/O activity. As of version
2.0.0 of Darshan, this file is portable and does not have to be analyzed on
-the same system that executed the job.
+the same system that executed the job. Also, note that Darshan logs generated
+with Darshan versions preceding version 3.0 will have the extension `darshan.gz`
+(or `darshan.bz2` if compressed using bzip2 format). These logs are not compatible
+with Darshan 3.0 utilities, and thus must be analyzed using an appropriate version
+(2.x) of the darshan-util package.
=== darshan-job-summary.pl
@@ -462,9 +466,9 @@ Byte and for the aggregate performance is MiB/s (1024*1024 Bytes/s).
===== Files
Use the `--file` option to get totals based on file usage.
-The first column is the count of files for that type, the second column is
-number of bytes for that type and the third column is the maximum offset
-accessed.
+Each line has 3 columns. The first column is the count of files for that
+type of file, the second column is number of bytes for that type, and the third
+column is the maximum offset accessed.
* total: All files
* read_only: Files that were only read from
@@ -473,9 +477,6 @@ accessed.
* unique: Files that were opened on only one rank
* shared: File that were opened by more than one rank
-Each line has 3 columns. The first column is the count of files for that
-type of file, the second column is number of bytes for that type, and the third
-column is the maximum offset accessed.
.Example output
----
@@ -557,8 +558,6 @@ If the `--bzip2` flag is given, then the output file will be re-compressed in
bzip2 format rather than libz format. It also has command line options for
anonymizing personal data, adding metadata annotation to the log header, and
restricting the output to a specific instrumented file.
-* darshan-diff: compares two darshan log files and shows counters that
-differ.
* darshan-analyzer: walks an entire directory tree of Darshan log files and
produces a summary of the types of access methods used in those log files.
* darshan-logutils*: this is a library rather than an executable, but it
=====================================
doc/darshan-modularization.txt
=====================================
--- a/doc/darshan-modularization.txt
+++ b/doc/darshan-modularization.txt
@@ -32,8 +32,8 @@ cd darshan
git checkout dev-modular
----
-For details on configuring and building the Darshan runtime and utility repositories,
-consult the documentation from previous versions
+For details on configuring, building, and using the Darshan runtime and utility
+repositories, consult the documentation from previous versions
(http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html[darshan-runtime] and
http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-util.html[darshan-util]) -- the
necessary steps for building these repositories should not have changed in the new version of
@@ -121,13 +121,13 @@ component so it is included in the output I/O characterization.
The static initialization approach is useful for modules that do not have function calls
that can be intercepted and instead can just grab all I/O characterization data at Darshan
startup or shutdown time. A module can be statically initialized at Darshan startup time
-by adding it to the `mod_static_init_fns` list at the top of the `lib/darshan-core.c`
-source file.
+by adding its initializatin routine to the `mod_static_init_fns` list at the top of the
+`lib/darshan-core.c` source file.
-*NOTE*: Modules which require static initialization should typically provide a corresponding
-configure option to prevent the module from being built and capturing I/O data. The ability
-to disable a module using a configure option is especially necessary for system-specific
-modules which can not be built or used on other systems.
+*NOTE*: Modules may wish to add a corresponding configure option to disable the module
+from attempting to gather I/O data. The ability to disable a module using a configure
+option is especially necessary for system-specific modules which can not be built or
+used on other systems.
Most instrumentation modules can just bootstrap themselves within wrapper functions during
normal application execution. Each of Darshan's current I/O library instrumentation modules
@@ -197,7 +197,7 @@ Within darshan-runtime, the darshan-core component manages the initialization an
Darshan environment, provides an interface for modules to register themselves and their data
records with Darshan, and manages the compressing and the writing of the resultant I/O
characterization. As illustrated in Figure 1, the darshan-core runtime environment intercepts
-`MPI_Init` and `MPI_Finalize` routines to initialize and shutdown the darshan runtime environment,
+`MPI_Init` and `MPI_Finalize` routines to initialize and shutdown the Darshan runtime environment,
respectively.
Each of the functions provided by `darshan-core` to interface with instrumentation modules are
@@ -212,8 +212,8 @@ void darshan_core_register_module(
int *sys_mem_alignment);
The `darshan_core_register_module` function registers Darshan instrumentation modules with the
-`darshan-core` runtime environment. This function needs to be called at least once for any module
-that will contribute data to Darshan's final I/O characterization.
+`darshan-core` runtime environment. This function needs to be called once for any module that
+will contribute data to Darshan's final I/O characterization.
* _mod_id_ is a unique identifier for the given module, which is defined in the Darshan log
format header file (`darshan-log-format.h`).
@@ -398,8 +398,8 @@ struct darshan_record_ref *, which should be initialized to `NULL` for reading).
is defined by the `uthash` hash table implementation and includes corresponding macros for
searching, iterating, and deleting records from the hash. For detailed documentation on using this
hash table, consult `uthash` documentation in `darshan-util/uthash-1.9.2/doc/txt/userguide.txt`.
-The `darshan-posix-parser` utility (for parsing POSIX module information out of a Darshan log)
-provides an example of how this hash table may be used. Returns `0` on success, `-1` on failure.
+The `darshan-parser` utility (for parsing module information out of a Darshan log) provides an
+example of how this hash table may be used. Returns `0` on success, `-1` on failure.
[source,c]
int darshan_log_getmod(darshan_fd fd, darshan_module_id mod_id, void *mod_buf, int mod_buf_sz);
@@ -448,9 +448,10 @@ the module's record structure:
* Add a module identifier to the `DARSHAN_MODULE_IDS` macro at the top of the `darshan-log-format.h`
header. In this macro, the first field is a corresponding enum value that can be used to
-identify the module, the second field is a string name for the module, and the third field
-is a corresponding pointer to a Darshan log utility implementation for this module (which
-can be set to `NULL` until the module has its own log utility implementation).
+identify the module, the second field is a string name for the module, the third field is the
+current version number of the given module's log format, and the fourth field is a corresponding
+pointer to a Darshan log utility implementation for this module (which can be set to `NULL`
+until the module has its own log utility implementation).
* Add a top-level header that defines an I/O data record structure for the module. Consider
the "NULL" module and POSIX module log format headers for examples (`darshan-null-log-format.h`
View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/compare/f78c456f736a007d5efc644293836f9bfc431691...96801d4a5894134dd47159b4f5cf9c31e797d4b5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-commits/attachments/20160119/46617886/attachment-0001.html>
More information about the Darshan-commits
mailing list