<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en">
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<title>
GitLab
</title>
</head>
<body>
<style type="text/css">
img {
max-width: 100%; height: auto;
}
</style>
<div class="content">
<h3>
Glenn K. Lockwood pushed to branch lustre-mod
at <a href="https://xgitlab.cels.anl.gov/darshan/darshan">darshan / darshan</a>
</h3>
<h4>
Commits:
</h4>
<ul>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/08956bd2eee22efd8ea8cc658010644bc6669562">08956bd2</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-18T21:38:05-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">new fs-specific counters for storage server and metadata server counts</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/19147a7b7c664691ec70c77994c58f6787f1d6e0">19147a7b</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-18T21:39:00-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">checkpoint of code that populates OST and MDT counts per file system--doesn't seem to work though...</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/01bc23aa44b66b12495dd99a25cd5e89dfea238f">01bc23aa</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-18T21:39:48-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">moved the call to the lustre module up into the file open macro</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/b547fbfe2f9f163a4051bf0bdf9e89cfbe3318f6">b547fbfe</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-18T21:40:18-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">replaced the dummy counter with some real counters to describe file geometry on Lustre</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/64a4a226de5f7097f98cff039704c0752b534de9">64a4a226</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-18T21:40:53-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">work in progress on enabling new lustre counters for files</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/4b63144a1ddf7d454852dde7819797ed32b50940">4b63144a</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-19T19:38:33-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">fixed bug that was preventing OST/MDT counts from being populated correctly</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/08d83a03ecf5e596a3130b0340ab80a8dade9680">08d83a03</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-19T19:41:24-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">added filename hashing so multiple opens/closes don't create multiple lustre-mod records; added proper reducer so that shared files are flattened into a single set of counters whose values are dictated by rank 0</pre>
</li>
<li>
<strong><a href="https://xgitlab.cels.anl.gov/darshan/darshan/commit/0934519c3a554a90443917e93ba0fb367aca6047">0934519c</a></strong>
<div>
<span>by Glenn K. Lockwood</span>
<i>at 2016-05-20T07:10:58-07:00</i>
</div>
<pre class="commit-message" style="white-space: pre-wrap">tool to test ioctl/llapi calls, based on stat-perf.c</pre>
</li>
</ul>
<h4>6 changed files:</h4>
<ul>
<li class="file-stats">
<a href="#620f2ecad2bb6f74b2fcd0134963a841" style="text-decoration: none">
darshan-lustre-log-format.h
</a>
</li>
<li class="file-stats">
<a href="#ad29afc395839758d41094872298bd0d" style="text-decoration: none">
darshan-runtime/darshan.h
</a>
</li>
<li class="file-stats">
<a href="#c0b0bf6d71bc5fc7e6d50d69c8aa2413" style="text-decoration: none">
darshan-runtime/lib/darshan-core.c
</a>
</li>
<li class="file-stats">
<a href="#2ace16276b68c7d5aef163bb260bd9f7" style="text-decoration: none">
darshan-runtime/lib/darshan-lustre.c
</a>
</li>
<li class="file-stats">
<a href="#4dfe7e78b540daa005ea4b5f0458c90d" style="text-decoration: none">
darshan-runtime/lib/darshan-posix.c
</a>
</li>
<li class="file-stats">
<a href="#172f4bf093caeeba24730a1adff8f15a" style="text-decoration: none">
<span class="new-file" style="color: #090">
+
darshan-test/2.x/llapi-perf.c
</span>
</a>
</li>
</ul>
<h4>Changes:</h4>
<li id="620f2ecad2bb6f74b2fcd0134963a841">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-0">
<strong>
darshan-lustre-log-format.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- a/darshan-lustre-log-format.h
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-lustre-log-format.h
</span><span style="color: #aaaaaa">@@ -12,8 +12,16 @@
</span>
/* TODO: add integer counters here (e.g., counter for stripe width, stripe size, etc etc) */
#define LUSTRE_COUNTERS \
<span style="background: #ffdddd; color: #000000">- /* dummy counter */\
- X(LUSTRE_TEST_COUNTER) \
</span><span style="background: #ddffdd; color: #000000">+ /* number of OSTs for file system */\
+ X(LUSTRE_OSTS) \
+ /* number of MDTs for file system */\
+ X(LUSTRE_MDTS) \
+ /* bytes per stripe for file */\
+ X(LUSTRE_STRIPE_SIZE) \
+ /* number of stripes (OSTs) for file */\
+ X(LUSTRE_STRIPE_WIDTH) \
+ /* index of first OST for file */\
+ X(LUSTRE_STRIPE_OFFSET) \
</span> /* end of counters */\
X(LUSTRE_NUM_INDICES)
</code></pre>
<br>
</li>
<li id="ad29afc395839758d41094872298bd0d">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-1">
<strong>
darshan-runtime/darshan.h
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- a/darshan-runtime/darshan.h
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-runtime/darshan.h
</span><span style="color: #aaaaaa">@@ -85,6 +85,8 @@ struct darshan_fs_info
</span> {
int fs_type;
int block_size;
<span style="background: #ddffdd; color: #000000">+ int ost_count;
+ int mdt_count;
</span> };
/* paths that darshan will not trace */
</code></pre>
<br>
</li>
<li id="c0b0bf6d71bc5fc7e6d50d69c8aa2413">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-2">
<strong>
darshan-runtime/lib/darshan-core.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- a/darshan-runtime/lib/darshan-core.c
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-runtime/lib/darshan-core.c
</span><span style="color: #aaaaaa">@@ -18,6 +18,8 @@
</span> #include <time.h>
#include <limits.h>
#include <pthread.h>
<span style="background: #ddffdd; color: #000000">+#include <dirent.h>
+#include <sys/ioctl.h>
</span> #include <sys/types.h>
#include <sys/stat.h>
#include <sys/vfs.h>
<span style="color: #aaaaaa">@@ -30,6 +32,9 @@
</span> #include "darshan-core.h"
#include "darshan-dynamic.h"
<span style="background: #ddffdd; color: #000000">+/* XXX stick this into autoconf .h */
+#include <lustre/lustreapi.h>
+
</span> extern char* __progname;
extern char* __progname_full;
<span style="color: #aaaaaa">@@ -961,6 +966,34 @@ static void add_entry(char* trailing_data, int* space_left, struct mntent *entry
</span> else
mnt_data_array[mnt_data_count].fs_info.block_size = 4096;
<span style="background: #ddffdd; color: #000000">+ /* attempt to retrieve OST and MDS counts from Lustre */
+ mnt_data_array[mnt_data_count].fs_info.ost_count = -1;
+ mnt_data_array[mnt_data_count].fs_info.mdt_count = -1;
+ if ( statfsbuf.f_type == LL_SUPER_MAGIC )
+ {
+ int n_ost, n_mdt;
+ int ret_ost, ret_mdt;
+ DIR *mount_dir;
+
+ mount_dir = opendir( entry->mnt_dir );
+ if ( mount_dir )
+ {
+ /* n_ost and n_mdt are used for both input and output to ioctl */
+ n_ost = 0;
+ n_mdt = 1;
+
+ ret_ost = ioctl( dirfd(mount_dir), LL_IOC_GETOBDCOUNT, &n_ost );
+ ret_mdt = ioctl( dirfd(mount_dir), LL_IOC_GETOBDCOUNT, &n_mdt );
+
+ if ( !(ret_ost < 0 || ret_mdt < 0) )
+ {
+ mnt_data_array[mnt_data_count].fs_info.ost_count = n_ost;
+ mnt_data_array[mnt_data_count].fs_info.mdt_count = n_mdt;
+ }
+ closedir( mount_dir );
+ }
+ }
+
</span> /* store mount information for use in header of darshan log */
ret = snprintf(tmp_mnt, 256, "\n%s\t%s",
entry->mnt_type, entry->mnt_dir);
</code></pre>
<br>
</li>
<li id="2ace16276b68c7d5aef163bb260bd9f7">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-3">
<strong>
darshan-runtime/lib/darshan-lustre.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- a/darshan-runtime/lib/darshan-lustre.c
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-runtime/lib/darshan-lustre.c
</span><span style="color: #aaaaaa">@@ -16,12 +16,22 @@
</span> #include <stdlib.h>
#include <assert.h>
#include <pthread.h>
<span style="background: #ddffdd; color: #000000">+#include <sys/ioctl.h>
+
+/* XXX stick this into autoconf .h */
+#include <lustre/lustreapi.h>
</span>
#include "uthash.h"
#include "darshan.h"
#include "darshan-dynamic.h"
<span style="background: #ddffdd; color: #000000">+struct lustre_record_runtime
+{
+ struct darshan_lustre_record *record;
+ UT_hash_handle hlink;
+};
+
</span> /* we just use a simple array for storing records. the POSIX module
* only calls into the Lustre module for new records, so we will never
* have to search for an existing Lustre record (assuming the Lustre
<span style="color: #aaaaaa">@@ -30,8 +40,10 @@
</span> struct lustre_runtime
{
struct darshan_lustre_record *record_array;
<span style="background: #ddffdd; color: #000000">+ struct lustre_record_runtime *record_runtime_array;
</span> int record_array_size;
int record_array_ndx;
<span style="background: #ddffdd; color: #000000">+ struct lustre_record_runtime *record_hash;
</span> };
static struct lustre_runtime *lustre_runtime = NULL;
<span style="color: #aaaaaa">@@ -45,52 +57,93 @@ static void lustre_begin_shutdown(void);
</span> static void lustre_get_output_data(MPI_Comm mod_comm, darshan_record_id *shared_recs,
int shared_rec_count, void **lustre_buf, int *lustre_buf_sz);
static void lustre_shutdown(void);
<span style="background: #ddffdd; color: #000000">+static int lustre_record_compare(const void* a_p, const void* b_p);
+static void lustre_record_reduction_op(void* infile_v, void* inoutfile_v,
+ int *len, MPI_Datatype *datatype);
</span>
#define LUSTRE_LOCK() pthread_mutex_lock(&lustre_runtime_mutex)
#define LUSTRE_UNLOCK() pthread_mutex_unlock(&lustre_runtime_mutex)
<span style="background: #ffdddd; color: #000000">-/* TODO: is there any way we can further compact Lustre data to save space?
- * e.g., are all files in the same directory guaranteed same striping parameters?
- * if so, can we store stripe parameters on per-directory basis and the OST
- * list on a per-file basis? maybe the storage savings are small enough this isn't
- * worth it, but nice to keep in mind
- */
-
-void darshan_instrument_lustre_file(char *filepath)
</span><span style="background: #ddffdd; color: #000000">+void darshan_instrument_lustre_file(const char* filepath, int fd)
</span> {
<span style="background: #ffdddd; color: #000000">- struct darshan_lustre_record *rec;
</span><span style="background: #ddffdd; color: #000000">+ struct lustre_record_runtime *rec_rt;
+ struct darshan_fs_info fs_info;
</span> darshan_record_id rec_id;
<span style="background: #ddffdd; color: #000000">+ int limit_flag;
</span>
LUSTRE_LOCK();
/* make sure the lustre module is already initialized */
lustre_runtime_initialize();
/* if the array is full, we just back out */
<span style="background: #ffdddd; color: #000000">- if(lustre_runtime->record_array_ndx >= lustre_runtime->record_array_size)
- return;
</span><span style="background: #ddffdd; color: #000000">+ limit_flag = (lustre_runtime->record_array_ndx >= lustre_runtime->record_array_size);
</span>
/* register a Lustre file record with Darshan */
<span style="background: #ddffdd; color: #000000">+ fs_info.fs_type = -1;
</span> darshan_core_register_record(
(void *)filepath,
strlen(filepath),
DARSHAN_LUSTRE_MOD,
1,
<span style="background: #ffdddd; color: #000000">- 0,
</span><span style="background: #ddffdd; color: #000000">+ limit_flag,
</span> &rec_id,
<span style="background: #ffdddd; color: #000000">- NULL);
</span><span style="background: #ddffdd; color: #000000">+ &fs_info);
</span>
/* if record id is 0, darshan has no more memory for instrumenting */
if(rec_id == 0)
<span style="background: #ddffdd; color: #000000">+ {
+ LUSTRE_UNLOCK();
</span> return;
<span style="background: #ddffdd; color: #000000">+ }
</span>
<span style="background: #ffdddd; color: #000000">- /* allocate a new lustre record and append it to the array */
- rec = &(lustre_runtime->record_array[lustre_runtime->record_array_ndx++]);
- rec->rec_id = rec_id;
- rec->rank = my_rank;
-
- /* TODO: gather lustre data, store in record hash */
- /* counters in lustre_ref->record->counters */
- rec->counters[LUSTRE_TEST_COUNTER] = 88;
</span><span style="background: #ddffdd; color: #000000">+ /* search the hash table for this file record, and initialize if not found */
+ HASH_FIND(hlink, lustre_runtime->record_hash, &rec_id, sizeof(darshan_record_id), rec_rt );
+ if ( !rec_rt ) {
+ struct darshan_lustre_record *rec;
+ struct lov_user_md *lum;
+ size_t lumsize = sizeof(struct lov_user_md) +
+ LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data);
+
+ /* allocate a new lustre record and append it to the array */
+ rec_rt = &(lustre_runtime->record_runtime_array[lustre_runtime->record_array_ndx]);
+ rec_rt->record = &(lustre_runtime->record_array[lustre_runtime->record_array_ndx]);
+ rec = rec_rt->record;
+ rec->rec_id = rec_id;
+ rec->rank = my_rank;
+
+ /* implicit assumption here that none of these counters will change
+ * after the first time a file is opened. This may not always be
+ * true in the future */
+ if ( fs_info.fs_type != -1 )
+ {
+ rec->counters[LUSTRE_OSTS] = fs_info.ost_count;
+ rec->counters[LUSTRE_MDTS] = fs_info.mdt_count;
+ }
+ else
+ {
+ rec->counters[LUSTRE_OSTS] = -1;
+ rec->counters[LUSTRE_MDTS] = -1;
+ }
+
+ if ( (lum = calloc(1, lumsize)) != NULL ) {
+ lum->lmm_magic = LOV_USER_MAGIC;
+ /* don't care about the return code for ioctl */
+ ioctl( fd, LL_IOC_LOV_GETSTRIPE, (void *)lum );
+ rec->counters[LUSTRE_STRIPE_SIZE] = lum->lmm_stripe_size;
+ rec->counters[LUSTRE_STRIPE_WIDTH] = lum->lmm_stripe_count;
+ rec->counters[LUSTRE_STRIPE_OFFSET] = 0; /* this currently doesn't work; lum->lmm_objects[0].l_ost_idx isn't being populated */
+ /* TODO: add explicit list of OSTs */
+ free(lum);
+ }
+ else
+ {
+ rec->counters[LUSTRE_STRIPE_SIZE] = -1;
+ rec->counters[LUSTRE_STRIPE_WIDTH] = -1;
+ rec->counters[LUSTRE_STRIPE_OFFSET] = -1;
+ }
+ HASH_ADD(hlink, lustre_runtime->record_hash, record->rec_id, sizeof(darshan_record_id), rec_rt);
+ lustre_runtime->record_array_ndx++;
+ }
</span>
LUSTRE_UNLOCK();
return;
<span style="color: #aaaaaa">@@ -142,6 +195,16 @@ static void lustre_runtime_initialize()
</span> memset(lustre_runtime->record_array, 0, lustre_runtime->record_array_size *
sizeof(struct darshan_lustre_record));
<span style="background: #ddffdd; color: #000000">+ lustre_runtime->record_runtime_array = malloc(lustre_runtime->record_array_size *
+ sizeof(struct lustre_record_runtime));
+ if(!lustre_runtime->record_runtime_array)
+ {
+ lustre_runtime->record_array_size = 0;
+ return;
+ }
+ memset(lustre_runtime->record_runtime_array, 0, lustre_runtime->record_array_size *
+ sizeof(struct lustre_record_runtime));
+
</span> return;
}
<span style="color: #aaaaaa">@@ -168,13 +231,75 @@ static void lustre_get_output_data(
</span> void **lustre_buf,
int *lustre_buf_sz)
{
<span style="background: #ddffdd; color: #000000">+ struct lustre_record_runtime *file;
+ int i;
+ struct darshan_lustre_record *red_send_buf = NULL;
+ struct darshan_lustre_record *red_recv_buf = NULL;
+ MPI_Datatype red_type;
+ MPI_Op red_op;
+
</span> assert(lustre_runtime);
<span style="background: #ffdddd; color: #000000">- /* TODO: determine lustre record shared across all processes,
- * and have only rank 0 write these records out. No shared
- * reductions should be necessary as the Lustre data for a
- * given file should be the same on each process
</span><span style="background: #ddffdd; color: #000000">+ /* if there are globally shared files, do a shared file reduction */
+ /* NOTE: the shared file reduction is also skipped if the
+ * DARSHAN_DISABLE_SHARED_REDUCTION environment variable is set.
</span> */
<span style="background: #ddffdd; color: #000000">+ if(shared_rec_count && !getenv("DARSHAN_DISABLE_SHARED_REDUCTION"))
+ {
+ /* necessary initialization of shared records */
+ for(i = 0; i < shared_rec_count; i++)
+ {
+ HASH_FIND(hlink, lustre_runtime->record_hash, &shared_recs[i],
+ sizeof(darshan_record_id), file);
+ assert(file);
+
+ file->record->rank = -1;
+ }
+
+ /* sort the array of files descending by rank so that we get all of the
+ * shared files (marked by rank -1) in a contiguous portion at end
+ * of the array
+ */
+ qsort(lustre_runtime->record_array, lustre_runtime->record_array_ndx,
+ sizeof(struct darshan_lustre_record), lustre_record_compare);
+
+ /* make *send_buf point to the shared files at the end of sorted array */
+ red_send_buf =
+ &(lustre_runtime->record_array[lustre_runtime->record_array_ndx-shared_rec_count]);
+
+ /* allocate memory for the reduction output on rank 0 */
+ if(my_rank == 0)
+ {
+ red_recv_buf = malloc(shared_rec_count * sizeof(struct darshan_lustre_record));
+ if(!red_recv_buf)
+ {
+ return;
+ }
+ }
+
+ DARSHAN_MPI_CALL(PMPI_Type_contiguous)(sizeof(struct darshan_lustre_record),
+ MPI_BYTE, &red_type);
+ DARSHAN_MPI_CALL(PMPI_Type_commit)(&red_type);
+ DARSHAN_MPI_CALL(PMPI_Op_create)(lustre_record_reduction_op, 1, &red_op);
+ DARSHAN_MPI_CALL(PMPI_Reduce)(red_send_buf, red_recv_buf,
+ shared_rec_count, red_type, red_op, 0, mod_comm);
+
+ /* clean up reduction state */
+ if(my_rank == 0)
+ {
+ int tmp_ndx = lustre_runtime->record_array_ndx - shared_rec_count;
+ memcpy(&(lustre_runtime->record_array[tmp_ndx]), red_recv_buf,
+ shared_rec_count * sizeof(struct darshan_lustre_record));
+ free(red_recv_buf);
+ }
+ else
+ {
+ lustre_runtime->record_array_ndx -= shared_rec_count;
+ }
+
+ DARSHAN_MPI_CALL(PMPI_Type_free)(&red_type);
+ DARSHAN_MPI_CALL(PMPI_Op_free)(&red_op);
+ }
</span>
*lustre_buf = (void *)(lustre_runtime->record_array);
*lustre_buf_sz = lustre_runtime->record_array_ndx * sizeof(struct darshan_lustre_record);
<span style="color: #aaaaaa">@@ -186,14 +311,68 @@ static void lustre_shutdown(void)
</span> {
assert(lustre_runtime);
<span style="background: #ffdddd; color: #000000">- /* TODO: free data structures */
</span><span style="background: #ddffdd; color: #000000">+ HASH_CLEAR(hlink, lustre_runtime->record_hash);
</span> free(lustre_runtime->record_array);
<span style="background: #ddffdd; color: #000000">+ free(lustre_runtime->record_runtime_array);
</span> free(lustre_runtime);
lustre_runtime = NULL;
return;
}
<span style="background: #ddffdd; color: #000000">+/* compare function for sorting file records by descending rank */
+static int lustre_record_compare(const void* a_p, const void* b_p)
+{
+ const struct darshan_lustre_record* a = a_p;
+ const struct darshan_lustre_record* b = b_p;
+
+ if(a->rank < b->rank)
+ return 1;
+ if(a->rank > b->rank)
+ return -1;
+
+ return 0;
+}
+
+/* this is just boilerplate reduction code that isn't currently used */
+static void lustre_record_reduction_op(void* infile_v, void* inoutfile_v,
+ int *len, MPI_Datatype *datatype)
+{
+ struct darshan_lustre_record tmp_record;
+ struct darshan_lustre_record *infile = infile_v;
+ struct darshan_lustre_record *inoutfile = inoutfile_v;
+ int i, j;
+
+ assert(lustre_runtime);
+
+ for( i=0; i<*len; i++ )
+ {
+ memset(&tmp_record, 0, sizeof(struct darshan_lustre_record));
+ tmp_record.rec_id = infile->rec_id;
+ tmp_record.rank = -1;
+
+ /* preserve only rank 0's value */
+ for( j = LUSTRE_OSTS; j < LUSTRE_NUM_INDICES; j++)
+ {
+ if ( my_rank == 0 )
+ {
+ tmp_record.counters[j] = infile->counters[j];
+ }
+ else
+ {
+ tmp_record.counters[j] = inoutfile->counters[j];
+ }
+ }
+
+ /* update pointers */
+ *inoutfile = tmp_record;
+ inoutfile++;
+ infile++;
+ }
+
+ return;
+}
+
</span> /*
* Local variables:
* c-indent-level: 4
</code></pre>
<br>
</li>
<li id="4dfe7e78b540daa005ea4b5f0458c90d">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-4">
<strong>
darshan-runtime/lib/darshan-posix.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- a/darshan-runtime/lib/darshan-posix.c
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-runtime/lib/darshan-posix.c
</span><span style="color: #aaaaaa">@@ -39,6 +39,10 @@ typedef int64_t off64_t;
</span> #define aiocb64 aiocb
#endif
<span style="background: #ddffdd; color: #000000">+#ifndef LL_SUPER_MAGIC
+#define LL_SUPER_MAGIC 0x0BD00BD0
+#endif
+
</span> DARSHAN_FORWARD_DECL(open, int, (const char *path, int flags, ...));
DARSHAN_FORWARD_DECL(open64, int, (const char *path, int flags, ...));
DARSHAN_FORWARD_DECL(creat, int, (const char* path, mode_t mode));
<span style="color: #aaaaaa">@@ -128,6 +132,7 @@ struct posix_file_runtime
</span> int access_count;
void* stride_root;
int stride_count;
<span style="background: #ddffdd; color: #000000">+ int fs_type; /* same as darshan_fs_info->fs_type */
</span> struct posix_aio_tracker* aio_list;
UT_hash_handle hlink;
};
<span style="color: #aaaaaa">@@ -179,7 +184,7 @@ struct posix_runtime
</span> /* XXX modules don't expose an API for other modules, so use extern to get
* Lustre instrumentation function
*/
<span style="background: #ffdddd; color: #000000">-extern void darshan_instrument_lustre_file(char *filepath);
</span><span style="background: #ddffdd; color: #000000">+extern void darshan_instrument_lustre_file(const char *filepath, int fd);
</span>
static struct posix_runtime *posix_runtime = NULL;
static pthread_mutex_t posix_runtime_mutex = PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP;
<span style="color: #aaaaaa">@@ -234,6 +239,8 @@ static void posix_shutdown(void);
</span> if(file->file_record->fcounters[POSIX_F_OPEN_TIMESTAMP] == 0 || \
file->file_record->fcounters[POSIX_F_OPEN_TIMESTAMP] > __tm1) \
file->file_record->fcounters[POSIX_F_OPEN_TIMESTAMP] = __tm1; \
<span style="background: #ddffdd; color: #000000">+ if(file->fs_type == LL_SUPER_MAGIC) \
+ darshan_instrument_lustre_file(__path, __ret); \
</span> DARSHAN_TIMER_INC_NO_OVERLAP(file->file_record->fcounters[POSIX_F_META_TIME], __tm1, __tm2, file->last_meta_end); \
} while(0)
<span style="color: #aaaaaa">@@ -1561,18 +1568,12 @@ static struct posix_file_runtime* posix_file_by_name(const char *name)
</span> file->file_record->rank = my_rank;
file->file_record->counters[POSIX_MEM_ALIGNMENT] = darshan_mem_alignment;
file->file_record->counters[POSIX_FILE_ALIGNMENT] = fs_info.block_size;
<span style="background: #ddffdd; color: #000000">+ file->fs_type = fs_info.fs_type;
</span>
/* add new record to file hash table */
HASH_ADD(hlink, posix_runtime->file_hash, file_record->f_id, sizeof(darshan_record_id), file);
posix_runtime->file_array_ndx++;
<span style="background: #ffdddd; color: #000000">- /* XXX: check for lustre and call in */
-#ifndef LL_SUPER_MAGIC
-#define LL_SUPER_MAGIC 0x0BD00BD0
-#endif
- if(fs_info.fs_type == LL_SUPER_MAGIC)
- darshan_instrument_lustre_file(newname);
-
</span> if(newname != name)
free(newname);
return(file);
</code></pre>
<br>
</li>
<li id="172f4bf093caeeba24730a1adff8f15a">
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047#diff-5">
<strong>
darshan-test/2.x/llapi-perf.c
</strong>
</a>
<hr>
<pre class="highlight"><code><span style="background: #ffdddd; color: #000000">--- /dev/null
</span><span style="background: #ddffdd; color: #000000">+++ b/darshan-test/2.x/llapi-perf.c
</span><span style="color: #aaaaaa">@@ -0,0 +1,313 @@
</span><span style="background: #ddffdd; color: #000000">+/*
+ * (C) 2012 by Argonne National Laboratory.
+ * See COPYRIGHT in top-level directory.
+ */
+
+/* llapi-perf.c
+ * Time how long it takes to extract various file data from Lustre via
+ * ioctl and llapi calls from every process. -i uses ioctl, -a uses the
+ * Lustre API. This also retains the features of stat-perf.c, which
+ * times how long it takes to issue a stat64() call to the designated file
+ * from every process. -f causes it to use fstat64() rather than stat64().
+ * -l causes it to use lseek(SEEK_END) instead of stat64().
+ * -c causes it to create the file from scratch rather than operating on an
+ * existing file. -r issues a realpath() call on the file.
+ */
+
+#define _LARGEFILE64_SOURCE
+
+#include <assert.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <string.h>
+#include <sys/time.h>
+#include <mpi.h>
+#include <errno.h>
+#include <getopt.h>
+#include <sys/ioctl.h>
+#ifndef NO_LUSTRE
+#include <lustre/lustreapi.h>
+#endif
+
+static char* opt_file = NULL;
+static int opt_create = 0;
+static int opt_fstat = 0;
+static int opt_lseek = 0;
+static int opt_realpath = 0;
+static int opt_ioctl = 0;
+static int opt_llapi = 0;
+static int opt_fpp = 0;
+static int rank = -1;
+
+static int parse_args(int argc, char **argv);
+static void usage(void);
+
+int main(int argc, char **argv)
+{
+ int fd;
+ int ret;
+ double stime, etime, elapsed, slowest;
+ struct stat64 statbuf;
+ int nprocs;
+ off64_t offset, orig_offset;
+ char* new_path;
+
+ MPI_Init(&argc,&argv);
+ MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+ MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
+
+ /* parse the command line arguments */
+ parse_args(argc, argv);
+
+ MPI_Barrier(MPI_COMM_WORLD);
+
+ /* open specified file */
+ if(!opt_create)
+ {
+ fd = open(opt_file, O_RDWR);
+ if(fd < 0)
+ {
+ perror("open");
+ exit(1);
+ }
+ }
+ else
+ {
+ /* rank 0 create, everyone else open */
+ if(rank == 0 || opt_fpp)
+ {
+ fd = open(opt_file, O_RDWR|O_CREAT|O_EXCL, S_IRUSR|S_IWUSR);
+ if(fd < 0)
+ {
+ perror("open");
+ exit(1);
+ }
+ MPI_Barrier(MPI_COMM_WORLD);
+ }
+ else
+ {
+ MPI_Barrier(MPI_COMM_WORLD);
+ fd = open(opt_file, O_RDWR);
+ if(fd < 0)
+ {
+ perror("open");
+ exit(1);
+ }
+ }
+ }
+
+ MPI_Barrier(MPI_COMM_WORLD);
+ stime = MPI_Wtime();
+
+ ret = 0;
+ if(opt_fstat)
+ ret = fstat64(fd, &statbuf);
+ else if(opt_lseek)
+ {
+ /* find current position */
+ orig_offset = lseek64(fd, 0, SEEK_CUR);
+ if(orig_offset < 0)
+ ret = -1;
+ else
+ {
+ /* find end of file; this is the size */
+ offset = lseek64(fd, 0, SEEK_END);
+ if(offset < 0)
+ ret = -1;
+ else
+ {
+ /* go back to original position */
+ offset = lseek64(fd, orig_offset, SEEK_SET);
+ if(offset < 0)
+ ret = -1;
+ }
+ }
+ }
+ else if(opt_realpath)
+ {
+ new_path = realpath(opt_file, NULL);
+ if(!new_path)
+ ret = -1;
+ else
+ free(new_path);
+ }
+ else if ( opt_llapi || opt_ioctl )
+ {
+#ifdef NO_LUSTRE
+ fprintf(stderr, "Not compiled with Lustre support\n");
+ ret = -1;
+#else
+ struct lov_user_md *lum;
+ size_t lumsize = sizeof(struct lov_user_md) +
+ LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data);
+
+ lum = calloc(1, lumsize);
+ if (lum == NULL) {
+ ret = ENOMEM;
+ fprintf(stderr, "No memory\n");
+ }
+ else {
+ if ( opt_llapi )
+ {
+ ret = llapi_file_get_stripe(opt_file, lum);
+ }
+ else if ( opt_ioctl )
+ {
+ lum->lmm_magic = LOV_USER_MAGIC;
+ ret = ioctl( fd, LL_IOC_LOV_GETSTRIPE, (void *)lum );
+ }
+#ifdef DEBUG
+ /* different API/ioctl calls populate only parts of lum */
+ printf( "stripe_width=%d stripe_size=%d starting_ost=%d\n",
+ lum->lmm_stripe_count,
+ lum->lmm_stripe_size,
+ lum->lmm_stripe_count );
+#endif
+ }
+#endif
+ }
+ else
+ ret = stat64(opt_file, &statbuf);
+
+ if(ret != 0 && !opt_ioctl && !opt_llapi)
+ {
+ perror("stat64 or fstat64");
+ exit(1);
+ }
+#ifndef NO_LUSTRE
+ else if ( ret < 0 && opt_ioctl )
+ {
+ perror("ioctl");
+ exit(1);
+ }
+#endif
+
+ etime = MPI_Wtime();
+
+ elapsed = etime-stime;
+ ret = MPI_Reduce(&elapsed, &slowest, 1, MPI_DOUBLE, MPI_MAX, 0, MPI_COMM_WORLD);
+ if(ret != 0)
+ {
+ fprintf(stderr, "Error: MPI_Reduce() failure.\n");
+ exit(1);
+ }
+
+
+ slowest *= 1000.0;
+
+ if(rank == 0)
+ {
+ printf("opt_file: %s, opt_create: %d, opt_fstat: %d, opt_lseek: %d, opt_realpath: %d, opt_llapi: %d, opt_ioctl: %d, opt_fpp: %d, nprocs: %d, time: %f ms\n",
+ opt_file,
+ opt_create,
+ opt_fstat,
+ opt_lseek,
+ opt_realpath,
+ opt_llapi,
+ opt_ioctl,
+ opt_fpp,
+ nprocs,
+ slowest);
+ }
+
+ MPI_Finalize();
+ return(0);
+}
+
+static int parse_args(int argc, char **argv)
+{
+ int c;
+
+ while ((c = getopt(argc, argv, "fclripa")) != EOF) {
+ switch (c) {
+ case 'c': /* create file */
+ opt_create = 1;
+ break;
+ case 'f': /* fstat instead of stat */
+ opt_fstat = 1;
+ break;
+ case 'l': /* lseek instead of stat */
+ opt_lseek = 1;
+ break;
+ case 'r': /* realpath instead of stat */
+ opt_realpath = 1;
+ break;
+ case 'i': /* use ioctl test */
+ opt_ioctl = 1;
+ break;
+ case 'a': /* use llapi test*/
+ opt_llapi = 1;
+ break;
+ case 'p': /* file per process instead of shared file */
+ opt_fpp = 1;
+ break;
+ case 'h':
+ if (rank == 0)
+ usage();
+ exit(0);
+ case '?': /* unknown */
+ if (rank == 0)
+ usage();
+ exit(1);
+ default:
+ break;
+ }
+ }
+
+ if(opt_lseek + opt_fstat + opt_realpath + opt_ioctl + opt_llapi > 1)
+ {
+ fprintf(stderr, "Error: Only specify one of -l, -f, -i, -a, or -r.\n");
+ usage();
+ exit(1);
+ }
+
+ if(argc-optind != 1)
+ {
+ if(rank == 0)
+ usage();
+ exit(1);
+ }
+
+ if ( opt_fpp )
+ {
+ opt_file = malloc( sizeof(char) * (strlen( argv[optind] ) + 10) );
+ sprintf( opt_file, "%s.%d", argv[optind], rank );
+ }
+ else
+ {
+ opt_file = strdup(argv[optind]);
+ }
+ assert(opt_file);
+
+ return(0);
+}
+
+static void usage(void)
+{
+ printf("Usage: stat-perf [<OPTIONS>...] <FILE NAME>\n");
+ printf("\n<OPTIONS> is one or more of\n");
+ printf(" -c create new file to stat\n");
+ printf(" -p do file-per-process instead of shared file\n");
+ printf(" -f use fstat instead of stat\n");
+ printf(" -l use lseek instead of stat\n");
+ printf(" -r use realpath instead of stat\n");
+ printf(" -a use Lustre API test\n");
+ printf(" -i use ioctl Lustre test\n");
+ printf(" -h print this help\n");
+}
+
+/*
+ * Local variables:
+ * c-indent-level: 3
+ * c-basic-offset: 3
+ * tab-width: 3
+ *
+ * vim: ts=3
+ * End:
+ */
+
+
</span></code></pre>
<br>
</li>
</div>
<div class="footer" style="margin-top: 10px">
<p style="color: #777; font-size: small">
—
<br>
<a href="https://xgitlab.cels.anl.gov/darshan/darshan/compare/3266825fa44dbc8e1cefbe99d189ae491d1ed42c...0934519c3a554a90443917e93ba0fb367aca6047">View it on GitLab</a>.
<br>
You're receiving this email because of your account on xgitlab.cels.anl.gov.
If you'd like to receive fewer emails, you can
adjust your notification settings.
</p>
</div>
</body>
</html>