<html lang='en'>
<head>
<meta content='text/html; charset=utf-8' http-equiv='Content-Type'>
<title>
GitLab
</title>
</meta>
</head>
<style>
img {
max-width: 100%;
height: auto;
}
p.details {
font-style:italic;
color:#777
}
.footer p {
font-size:small;
color:#777
}
pre.commit-message {
white-space: pre-wrap;
}
.file-stats a {
text-decoration: none;
}
.file-stats .new-file {
color: #090;
}
.file-stats .deleted-file {
color: #B00;
}
</style>
<body>
<div class='content'>
<h3>Shane Snyder pushed to branch dev-modular at <a href="https://xgitlab.cels.anl.gov/darshan/darshan">darshan / darshan</a></h3>
<h4>
Commits:
</h4>
<ul>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/7272638723031ecbdaa9d898e499647293c11b4b">72726387</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-04-29T10:59:18Z</i>
</div>
<pre class='commit-message'>fix for gnuplot version number check
- contributed by Kay Thust</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/1d20246489ae357833ee195d7607ca2e1af347fc">1d202464</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-06-30T22:31:31Z</i>
</div>
<pre class='commit-message'>fix map_or_fail typo in lio_listio wrapper</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/341bf09f292b2c02feb5a7b290b0f5a54fcf6ad4">341bf09f</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-06-30T22:36:16Z</i>
</div>
<pre class='commit-message'>bug fix for aio_return wrapper</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/14ad627477e4aaa2e937f8c9d5078eb9dca5af24">14ad6274</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-07-01T09:16:47Z</i>
</div>
<pre class='commit-message'>update Changelog for last 2 bug fixes</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/6be9df5bf66d0c2433d3e828fb8fce2752b1f5c8">6be9df5b</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-08-06T18:01:41Z</i>
</div>
<pre class='commit-message'>bug fix in common access counter logic</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/fbce8030d52a56218fd8b813b180d8060ce4a99b">fbce8030</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-08-07T08:47:18Z</i>
</div>
<pre class='commit-message'>update changelog for common access bug</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/f68228a5a98d67a3d1e2e29edf94c4090a1a7392">f68228a5</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-08-19T11:55:50Z</i>
</div>
<pre class='commit-message'>integrate expanded darshan-parser documentation
- provided by Huong Luu</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/c5a276374ea8c4443f716725e2d06cc3c7ae8b90">c5a27637</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-08T12:30:55Z</i>
</div>
<pre class='commit-message'>trivial stat collection script
- for each job, count number of files that used collectives, independent
operations, or posix</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/0ffb95474e51fb29c7b73876e7997208a4f6e17c">0ffb9547</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-08T16:41:44Z</i>
</div>
<pre class='commit-message'>friendlier output</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/676030f16b4f8c1a062b6abdcbd893aa35698bcc">676030f1</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-08T16:45:43Z</i>
</div>
<pre class='commit-message'>bug fix</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/7eecf5d0dd5acdf93afdd63825574538b0bec904">7eecf5d0</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-25T13:05:00Z</i>
</div>
<pre class='commit-message'>markdown version of readme for gitlab</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/3d3c4257128b07120f8b572e38410370572a664a">3d3c4257</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-25T13:06:48Z</i>
</div>
<pre class='commit-message'>trying to figure out why markdown link is grumpy</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/83f12d5f90c214f2cc852451f1b6c06a019158d1">83f12d5f</a></strong>
<div>
<span>by Phil Carns</span>
<i>at 2015-09-25T13:16:52Z</i>
</div>
<pre class='commit-message'>whitespace change to test commit hooks</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/f6fbcec40d2573e0c34f519c58460b0b4c611d47">f6fbcec4</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-01T11:19:41Z</i>
</div>
<pre class='commit-message'>Merge remote-tracking branch 'origin/master' into dev-modular
Conflicts:
ChangeLog
darshan-runtime/configure
darshan-runtime/configure.in
darshan-runtime/darshan.h
darshan-runtime/lib/darshan-posix.c
darshan-util/configure
darshan-util/configure.in
darshan-util/doc/darshan-util.txt</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/41575785a890ced023ae36e6781ab7bd63440274">41575785</a></strong>
<div>
<span>by Shane Snyder</span>
<i>at 2015-10-01T11:46:02Z</i>
</div>
<pre class='commit-message'>small text changes in darshan-util docs</pre>
</li>
</ul>
<h4>4 changed files:</h4>
<ul>
<li class='file-stats'>
<a href='#diff-0'>
ChangeLog
</a>
</li>
<li class='file-stats'>
<a href='#diff-1'>
README
→
README.md
</a>
</li>
<li class='file-stats'>
<a href='#diff-2'>
<span class='new-file'>
+
darshan-test/darshan-gather-mpi-posix-usage.pl
</span>
</a>
</li>
<li class='file-stats'>
<a href='#diff-3'>
darshan-util/doc/darshan-util.txt
</a>
</li>
</ul>
<h4>Changes:</h4>
<li id='diff-0'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274#diff-0'>
<strong>
ChangeLog
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/ChangeLog
</span><span style="color: #000000;background-color: #ddffdd">+++ b/ChangeLog
</span><span style="color: #aaaaaa">@@ -27,6 +27,16 @@ Darshan-3.0.0-pre1
</span> darshan-util components are mostly the same and still located in
their respective directories ('darshan-runtime/doc' and 'darshan-util/doc')
<span style="color: #000000;background-color: #ddffdd">+darshan-2.3.2-pre1
+=============
+* Fix gnuplot version number check to allow darshan-job-summary.pl to work
+ with gnuplot 5.0 (Kay Thust)
+* Fix function pointer mapping typo in lio_listio64 wrapper (Shane Snyder)
+* Fix faulty logic in extracting I/O data from the aio_return
+ wrapper (Shane Snyder)
+* Fix bug in common access counter logic (Shane Snyder)
+* Expand and clarify darshan-parser documentation (Huong Luu)
+
</span> darshan-2.3.1
=============
* added documentation and example configuration files for using the -profile
</code></pre>
<br>
</li>
<li id='diff-1'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274#diff-1'>
<strong>
README
</strong>
→
<strong>
README.md
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/README
</span><span style="color: #000000;background-color: #ddffdd">+++ b/README.md
</span><span style="color: #aaaaaa">@@ -1,4 +1,14 @@
</span>-The Darshan source tree is divided into two parts:
<span style="color: #000000;background-color: #ddffdd">+Darshan is a lightweight I/O characterization tool that transparently
+captures I/O access pattern information from HPC applications.
+Darshan can be used to tune applications for increased scientific
+productivity or to gain insight into trends in large-scale computing
+systems.
+
+Please see the
+[Darshan web page](http://www.mcs.anl.gov/research/projects/darshan)
+for more in-depth news and documentation.
+
+The Darshan source tree is divided into two main parts:
</span>
- darshan-runtime: to be installed on systems where you intend to
instrument MPI applications. See darshan-runtime/doc/darshan-runtime.txt
<span style="color: #aaaaaa">@@ -8,9 +18,7 @@ The Darshan source tree is divided into two parts:
</span> log files produced by darshan-runtime. See
darshan-util/doc/darshan-util.txt for installation instructions.
-General documentation can be found on the Darshan documentation web page:
-http://www.mcs.anl.gov/darshan/documentation/
-
The darshan-test directory contains various test harnesses, benchmarks,
patches, and unsupported utilites that are mainly of interest to Darshan
developers.
<span style="color: #000000;background-color: #ddffdd">+
</span></code></pre>
<br>
</li>
<li id='diff-2'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274#diff-2'>
<strong>
darshan-test/darshan-gather-mpi-posix-usage.pl
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- /dev/null
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-test/darshan-gather-mpi-posix-usage.pl
</span><span style="color: #aaaaaa">@@ -0,0 +1,98 @@
</span><span style="color: #000000;background-color: #ddffdd">+#!/usr/bin/perl -w
+
+# This script will go through all of the darshan logs in a given
+# subdirectory and summarize a few basic statistics about data usage and
+# performance, producing a text file with text in columns
+
+#<jobid> <version> <start ascii> <end ascii> <start unix> <end unix> <nprocs> <bytes read> <bytes written> <perf estimate>
+
+use strict;
+use File::Find;
+
+sub wanted
+{
+ my $file = $_;
+ my $line;
+ my $version = 0.0;
+ my $nprocs = 0;
+ my $start = 0;
+ my $end = 0;
+ my $start_a = "";
+ my $end_a = "";
+ my $jobid = 0;
+ my $bytes_r = 0;
+ my $bytes_w = 0;
+ my $perf = 0.0;
+ my @fields;
+ my $mpi_coll_count = 0;
+ my $mpi_indep_count = 0;
+ my $posix_count = 0;
+
+ # only operate on darshan log files
+ $file =~ /\.darshan\.gz$/ or return;
+
+ # grab jobid from name, old logs don't store it in the file
+ if($file =~ /_id(\d+)_/) {
+ $jobid = $1;
+ }
+
+ if(!(open(SUMMARY, "darshan-parser --file-list-detailed $file |")))
+ {
+ print(STDERR "Failed to parse $File::Find::name\n");
+ return;
+ }
+
+ while ($line = <SUMMARY>) {
+ if($line =~ /^#/) {
+ next;
+ }
+ if($line =~ /^\s/) {
+ next;
+ }
+
+ @fields = split(/\s/, $line);
+
+ if($#fields == 34)
+ {
+ if($fields[13] > 0){
+ $mpi_coll_count ++;
+ }
+ elsif($fields[12] > 0){
+ $mpi_indep_count ++;
+ }
+ elsif($fields[14] > 0){
+ $posix_count ++;
+ }
+
+ }
+ }
+
+ print(STDOUT "$jobid\t$mpi_coll_count\t$mpi_indep_count\t$posix_count\n");
+ close(SUMMARY);
+}
+
+sub main
+{
+ my @paths;
+
+ if($#ARGV < 0) {
+ die("usage: darshan-gather-stats.pl <one or more log directories>\n");
+ }
+
+ @paths = @ARGV;
+
+ print("# <jobid>\t<#files_using_collectives>\t<#files_using_indep>\t<#files_using_posix>\n");
+ print("# NOTE: a given file will only show up in one category, with preference in the order shown above (i.e. a file that used collective I/O will not show up in the indep or posix category).\n");
+
+ find(\&wanted, @paths);
+
+}
+
+main();
+
+# Local variables:
+# c-indent-level: 4
+# c-basic-offset: 4
+# End:
+#
+# vim: ts=8 sts=4 sw=4 expandtab
</span></code></pre>
<br>
</li>
<li id='diff-3'>
<a href='https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274#diff-3'>
<strong>
darshan-util/doc/darshan-util.txt
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="color: #000000;background-color: #ffdddd">--- a/darshan-util/doc/darshan-util.txt
</span><span style="color: #000000;background-color: #ddffdd">+++ b/darshan-util/doc/darshan-util.txt
</span><span style="color: #aaaaaa">@@ -131,11 +131,10 @@ specified file.
</span>
=== darshan-parser
-In order to obtained a full, human readable dump of all information
-contained in a log file, you can use the `darshan-parser` command
-line utility. It does not require any additional command line tools.
-The following example essentially converts the contents of the log file
-into a fully expanded text file:
<span style="color: #000000;background-color: #ddffdd">+You can use the `darshan-parser` command line utility to obtain a
+complete, human-readable, text-format dump of all information contained
+in a log file. The following example converts the contents of the
+log file into a fully expanded text file:
</span>
----
darshan-parser carns_my-app_id114525_7-27-58921_19.darshan.gz > ~/job-characterization.txt
<span style="color: #aaaaaa">@@ -146,8 +145,14 @@ The format of this output is described in the following section.
</span> === Guide to darshan-parser output
The beginning of the output from darshan-parser displays a summary of
-overall information about the job. The following table defines the meaning
-of each line:
<span style="color: #000000;background-color: #ddffdd">+overall information about the job. Additional job-level summary information
+can also be produced using the `--perf`, `--file`, `--file-list`, or
+`--file-list-detailed` command line options. See the
+<<addsummary,Additional summary output>> section for more information about
+those options.
+
+The following table defines the meaning
+of each line in the default header section of the output:
</span>
[cols="25%,75%",options="header"]
|====
<span style="color: #aaaaaa">@@ -365,6 +370,7 @@ value of 1 MiB for optimal file alignment.
</span> |====
==== Additional summary output
<span style="color: #000000;background-color: #ddffdd">+[[addsummary]]
</span>
The following sections describe addtitional parser options that provide
summary I/O characterization data for the given log.
<span style="color: #aaaaaa">@@ -373,8 +379,7 @@ summary I/O characterization data for the given log.
</span>
===== Performance
-Use the '--perf' option to get performance approximations using four
-different computations.
<span style="color: #000000;background-color: #ddffdd">+Job performance information can be generated using the `--perf` command-line option.
</span>
.Example output
----
<span style="color: #aaaaaa">@@ -407,6 +412,54 @@ different computations.
</span> # agg_perf_by_slowest: 2206.983935
----
<span style="color: #000000;background-color: #ddffdd">+The `total_bytes` line shows the total number of bytes transferred
+(read/written) by the job. That is followed by three sections:
+
+.I/O timing for unique files
+
+This section reports information about any files that were *not* opened
+by every rank in the job. This includes independent files (opened by
+1 process) and partially shared files (opened by a proper subset of
+the job's processes). The I/O time for this category of file access
+is reported based on the *slowest* rank of all processes that performed this
+type of file access.
+
+* unique files: slowest_rank_io_time: total I/O time for unique files
+ (including both metadata + data transfer time)
+* unique files: slowest_rank_meta_time: metadata time for unique files
+* unique files: slowest_rank: the rank of the slowest process
+
+.I/O timing for shared files
+
+This section reports information about files that were globally shared (i.e.
+opened by every rank in the job). This section estimates performance for
+globally shared files using four different methods. The `time_by_slowest`
+is generally the most accurate, but it may not available in some older Darshan
+log files.
+
+* shared files: time_by_cumul_*: adds the cumulative time across all
+ processes and divides by the number of processes (inaccurate when there is
+ high variance among processes).
+** shared files: time_by_cumul_io_only: include metadata AND data transfer
+ time for global shared files
+** shared files: time_by_cumul_meta_only: metadata time for global shared
+ files
+* shared files: time_by_open: difference between timestamp of open and
+ close (inaccurate if file is left open without I/O activity)
+* shared files: time_by_open_lastio: difference between timestamp of open
+ and the timestamp of last I/O (similar to above but fixes case where file is
+ left open after I/O is complete)
+* shared files: time_by_slowest : measures time according to which rank was
+ the slowest to perform both metadata operations and data transfer for each
+ shared file. (most accurate but requires newer log version)
+
+.Aggregate performance
+
+Performance is calculated by dividing the total bytes by the I/O time
+(shared files and unique files combined) computed
+using each of the four methods described in the previous output section. Note the unit for total bytes is
+Byte and for the aggregate performance is MiB/s (1024*1024 Bytes/s).
+
</span> ===== Files
Use the `--file` option to get totals based on file usage.
The first column is the count of files for that type, the second column is
<span style="color: #aaaaaa">@@ -416,9 +469,14 @@ accessed.
</span> * total: All files
* read_only: Files that were only read from
* write_only: Files that were only written to
<span style="color: #000000;background-color: #ddffdd">+* read_write: Files that were both read and written
</span> * unique: Files that were opened on only one rank
* shared: File that were opened by more than one rank
<span style="color: #000000;background-color: #ddffdd">+Each line has 3 columns. The first column is the count of files for that
+type of file, the second column is number of bytes for that type, and the third
+column is the maximum offset accessed.
+
</span> .Example output
----
# files
<span style="color: #aaaaaa">@@ -433,10 +491,11 @@ accessed.
</span>
===== Totals
-Use the `--total` option to get all statistics as an aggregate total.
-Statistics that make sense to be aggregated are aggregated. Other statistics
-may be a minimum or maximum if that makes sense. Other data maybe zeroed if
-it doesn't make sense to aggregate the data.
<span style="color: #000000;background-color: #ddffdd">+Use the `--total` option to get all statistics as an aggregate total rather
+than broken down per file. Each field is either summed across files and
+process (for values such as number of opens), set to global minimums and
+maximums (for values such as open time and close time), or zeroed out (for
+statistics that are nonsensical in aggregate).
</span>
.Example output
----
<span style="color: #aaaaaa">@@ -475,11 +534,18 @@ file.
</span> 5041708885572677970 /projects/SSSPPg/snyder/ior/ior.dat 1024 16.342061 1.705930
----
<span style="color: #000000;background-color: #ddffdd">+This data could be post-processed to compute more in-depth statistics, such as
+the total number of MPI files and total number of POSIX files used in a
+job, categorizing files into independent/unique/local files (opened by
+1 process), subset/partially shared files (opened by a proper subset of
+processes) or globally shared files (opened by all processes), and ranking
+files according to how much time was spent performing I/O in each file.
+
</span> ===== Detailed file list
The `--file-list-detailed` is the same as --file-list except that it
produces many columns of output containing statistics broken down by file.
-This option is mainly useful for automated analysis.
<span style="color: #000000;background-color: #ddffdd">+This option is mainly useful for more detailed automated analysis.
</span>
=== Other darshan-util utilities
</code></pre>
<br>
</li>
</div>
<div class='footer' style='margin-top: 10px;'>
<p>
—
<br>
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274">View it on GitLab</a>
<script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","action":{"@type":"ViewAction","name":["merge_requests","issues","commit"],"url":"https://xgitlab.cels.anl.gov/darshan/darshan/compare/9274a0db254dad9177a80c1781259daab3b254cf...41575785a890ced023ae36e6781ab7bd63440274"}}</script>
</p>
</div>
</body>
</html>