[Darshan-commits] [Git][darshan/darshan][dev-stdio] 98 commits: first cut at mmap logic -- posix mod only for now
Philip Carns
xgitlab at cels.anl.gov
Wed Jun 22 13:54:37 CDT 2016
Philip Carns pushed to branch dev-stdio at darshan / darshan
Commits:
0d84b5e5 by Shane Snyder at 2015-10-12T14:42:13-05:00
first cut at mmap logic -- posix mod only for now
- - - - -
c3c5e597 by Shane Snyder at 2015-10-13T16:44:32-05:00
more mmap logic and instrumentation for testing
- - - - -
e810279e by Shane Snyder at 2015-11-11T15:43:27-06:00
Merge branch 'dev-modular' into mmap-dev
Conflicts:
darshan-runtime/darshan-core.h
darshan-runtime/lib/darshan-core.c
- - - - -
799d9890 by Shane Snyder at 2015-11-24T14:45:31-06:00
update logutils to allow logs with only job info
- - - - -
bdb2a5f0 by Shane Snyder at 2015-12-01T12:40:28-06:00
runtime now mmaps header and job data to tmp log
- - - - -
3b0c668b by Shane Snyder at 2015-12-02T14:36:19-06:00
update logutils to read uncompressed log files
- - - - -
4f1b27ad by Shane Snyder at 2015-12-02T15:18:33-06:00
bug fix in mmap region pointer assignment
- - - - -
b1791172 by Shane Snyder at 2015-12-03T16:20:34-06:00
updated logutils hash read deserialization method
- - - - -
523b19e4 by Shane Snyder at 2015-12-03T16:22:25-06:00
update core runtime record hash implementation
Now, as the app executes, darshan-core persists the id->name
mapping data to the tmp log file. This is done by serializing the
id->name mapping to the mmap log file region when new records are
registered.
- - - - -
376eefa8 by Shane Snyder at 2015-12-04T10:20:47-06:00
modify register_record routine used by modules
Now, register_record() accepts an argument for the size of
the record being registered, and this is used to track how much
data each module has stored in their allocated memory region.
- - - - -
9193b34a by Shane Snyder at 2015-12-04T11:10:14-06:00
add new utility to stitch together darshan logs
- - - - -
c9618c8a by Shane Snyder at 2015-12-04T13:52:19-06:00
runtime now stores mnt & exe data at all ranks
- - - - -
4ac75905 by Shane Snyder at 2015-12-04T15:18:06-06:00
modify hash write routine to use new serialization
- - - - -
66a83b03 by Shane Snyder at 2015-12-07T13:25:52-06:00
first cut at the stitch logs utility
Currently, this tool can take a set of temporary, uncompressed log
files on a given node, and combine into a single darshan log.
Next step would be to modify this code to perform shared file
reduction logic to combine per-node log files into per-job log
files.
- - - - -
cb1c7ace by Shane Snyder at 2015-12-08T15:04:41-06:00
change runtime darshan record_ref struct
- - - - -
9d7f638b by Shane Snyder at 2015-12-08T15:16:19-06:00
change util record ref structure def
- - - - -
05acb6d4 by Shane Snyder at 2015-12-08T16:07:43-06:00
introduce base record concept in posix mod
- - - - -
ed103a2a by Shane Snyder at 2015-12-08T16:22:23-06:00
add base record concept to utilities
- - - - -
be3912a8 by Shane Snyder at 2015-12-09T10:28:40-06:00
update utilities and log format to use base rec
- - - - -
fe340438 by Shane Snyder at 2015-12-14T12:48:15-06:00
add an aggregation routine to module logutils
- - - - -
a84e8657 by Shane Snyder at 2015-12-17T16:54:46-06:00
integrate normal darshan shutdown back into code
- - - - -
80c07c27 by Shane Snyder at 2015-12-17T16:55:11-06:00
more darshan-util updates for handling tmp logs
- - - - -
cc498bfa by Shane Snyder at 2016-01-14T16:16:55-06:00
minor reorganization of some runtime code
- - - - -
b989ffe4 by Shane Snyder at 2016-01-14T16:20:15-06:00
integrate new darshan-diff util in mmap branch
- - - - -
c52fed2a by Shane Snyder at 2016-01-14T16:22:32-06:00
typos in darshan-diff
- - - - -
032f99b4 by Shane Snyder at 2016-01-14T16:42:37-06:00
forgot to install diff utility in makefile
- - - - -
d3fae782 by Shane Snyder at 2016-01-18T21:42:11-06:00
update how module & record registration
- modules now receive a memory buffer after successfully
registering with darshan-core. modules can request a specific
memory amount, but may not be honored depending on the configured
darshan memory limit
- split the previous register_record function into 2 parts:
* lookup() -> lookup darshan_record_id corresp. to path name
* register() -> register record_id & name with darshan-core
- register_record now returns true if the record added; false
otherwise (if module was out of memory or darshan is out of memory
for storing id->name mappings)
- - - - -
f78c456f by Shane Snyder at 2016-01-19T12:57:25-06:00
apply mmap code changes to other i/o modules
- - - - -
265bdaba by Shane Snyder at 2016-01-19T16:18:22-06:00
Merge branch 'dev-modular' into mmap-dev
Conflicts:
darshan-runtime/Makefile.in
darshan-runtime/darshan-core.h
darshan-runtime/lib/darshan-core.c
darshan-util/Makefile.in
darshan-util/darshan-bgq-logutils.c
darshan-util/darshan-diff.c
darshan-util/darshan-hdf5-logutils.c
darshan-util/darshan-logutils.c
darshan-util/darshan-mpiio-logutils.c
darshan-util/darshan-null-logutils.c
darshan-util/darshan-parser.c
darshan-util/darshan-pnetcdf-logutils.c
darshan-util/darshan-posix-logutils.c
- - - - -
96801d4a by Shane Snyder at 2016-01-19T17:44:11-06:00
fix post-merge build errors
dev-modular should be completely merged into mmap-dev branch
- - - - -
f57f753c by Shane Snyder at 2016-01-20T16:03:28-06:00
bug fixes from merge of mmap-dev to dev-modular
- - - - -
10c3b107 by Shane Snyder at 2016-01-20T22:21:28-06:00
bug fix in log aggregation tool
- - - - -
d3ebeb50 by Shane Snyder at 2016-01-20T22:21:43-06:00
add aggregation capability to i/o modules
- - - - -
ac6c6442 by Shane Snyder at 2016-01-22T10:08:27-06:00
bug fix in darshan put_header routine
- - - - -
d5abad0f by Shane Snyder at 2016-01-22T10:53:52-06:00
remove profiling macros from posix module
- - - - -
bb25083b by Shane Snyder at 2016-01-25T13:19:59-06:00
rename merge log utility
- - - - -
e81a9cae by Shane Snyder at 2016-01-25T15:56:06-06:00
autconf/runtime changes for configuring mmap code
- - - - -
8070c831 by Shane Snyder at 2016-01-26T12:59:32-06:00
update command line args for log merging utility
- - - - -
b4be0abd by Shane Snyder at 2016-01-26T13:00:11-06:00
allreduce job start time in darshan-core init.
- - - - -
5f4e06f6 by Shane Snyder at 2016-01-26T13:02:54-06:00
autogenerate an example epilog script
- - - - -
c276e4c2 by Shane Snyder at 2016-01-29T13:00:26-06:00
first cut at unifying mmap feature with darshan
- - - - -
a22233d7 by Shane Snyder at 2016-01-29T15:40:17-06:00
improve slurm epilog script
epilog will now skip on-node compression if only a single compute
node is used and will generate the final log file instead
- - - - -
bb625cbd by Shane Snyder at 2016-02-05T13:53:21-06:00
allow common counters to be updated on the go
- - - - -
37b1bb02 by Shane Snyder at 2016-02-05T17:27:54-06:00
add common counter aggregation to logutils
- - - - -
f26baefc by Shane Snyder at 2016-02-26T12:34:02-06:00
allow job end time to be set by darshan-merge
- - - - -
d4a3d112 by Shane Snyder at 2016-02-29T17:22:57-06:00
add variance counter reduction logic to logutils
- - - - -
04063da7 by Shane Snyder at 2016-03-01T17:43:38-06:00
common access counters are now sorted in logs
common value counters are sorted first by decreasing counts,
and then by decreasing size.
- - - - -
94430f41 by Shane Snyder at 2016-03-01T18:30:08-06:00
bug fix in aggregating common accesses
- - - - -
a3b2cfb9 by Shane Snyder at 2016-05-04T13:22:27-05:00
fix perl scripts to get perl bin from env
- - - - -
4f1dbfc7 by Shane Snyder at 2016-05-12T09:49:38-05:00
fix typos in fortran env vars in runtime docs
- - - - -
f75b4349 by Shane Snyder at 2016-05-12T17:13:29-05:00
fortran compiler generator autodetects mpich libs
this functionality is helpful especially on mira/cetus where
the mpich libraries are named depending on the fortran version
and the type of compiler being used (gcc, xl, etc.).
- - - - -
b41c9d63 by Shane Snyder at 2016-05-13T10:38:56-05:00
cxx compiler generators autodetect mpich lib name
- - - - -
5e1a0963 by Shane Snyder at 2016-05-16T14:20:38-05:00
bug fix in agg_perf_by_slowest perf calculation
this calculation was inadvertently accounting for metadata time
twice. also, clarified timing descriptions for slowest unique
file perf calculations.
- - - - -
9b2600e6 by Shane Snyder at 2016-05-19T17:41:46+00:00
bug fix in fortran compiler generator
- - - - -
7610c2fb by Shane Snyder at 2016-05-19T12:45:25-05:00
add (mpiio or posix) perf. est. to summary graphs
- - - - -
534a0e1d by Shane Snyder at 2016-05-20T10:08:03-05:00
add perf estimate to job summary graphs
- - - - -
0979cfd1 by Shane Snyder at 2016-05-20T10:43:02-05:00
update changelog for 3.0.1 release
- - - - -
76ae5f6b by Shane Snyder at 2016-05-20T10:51:01-05:00
update darshan version in runtime/util configure
- - - - -
08855b48 by Shane Snyder at 2016-05-20T12:40:46-05:00
example output logs for x86 and ppc
- - - - -
d6dad7f3 by Shane Snyder at 2016-05-26T12:29:34-05:00
add variable-length definition of name records
also some code clean up to change references of record hash
to name hash to be more clear about what it is actually storing.
- - - - -
999ee9c8 by Shane Snyder at 2016-05-26T12:53:05-05:00
darshan-util edits for var. length name records
- - - - -
8f463e51 by Shane Snyder at 2016-05-27T15:10:12-05:00
remove shared name records on nonzero ranks
- - - - -
7d35c45a by Shane Snyder at 2016-05-27T16:36:32-05:00
register/lookup_record now use strings explicitly
- - - - -
bfde9eb7 by Shane Snyder at 2016-05-31T15:47:51-05:00
numerous darshan-core api changes
- memory management is now entirely handled by darshan-core for
module records
* when registering new records, darshan-core returns the
corresponding buffer to write it to, rather than allowing
module to determine this
- modules now decide whether darshan-core should track record
names by passing either a valid name or NULL to the register
function
* this allows modules to register records that aren't really
tracked by darshan-core
- only the POSIX module has been ported to new changes for this
commit
- - - - -
9bfc7b65 by Shane Snyder at 2016-06-06T17:16:31-05:00
add record_ref interface to darshan-common
This interface provides some convenience functions for mapping
opaque record identifiers (could be a filename, file descriptor,
or any other type of file handle) to a corresponding runtime
structure referencing the actual file record. This runtime
structure should at least contain a pointer to the actual file
record, and can also persist arbitrary state for this record.
- - - - -
a6e86829 by Shane Snyder at 2016-06-07T11:09:52-05:00
bug fixes in summary per file tool
- - - - -
d8f3575a by Shane Snyder at 2016-06-08T16:58:23-05:00
darshan-core code and documentation cleanup
Cleanup the structure and documenation of the code to be in sync
with the new mmap version of Darshan. In particular, I made sure
all extra mmap logic was properly ifdef wrapped so it's more clear
which portions of the code are responsible for implementing that
feature.
- - - - -
54baa7dd by Shane Snyder at 2016-06-08T17:34:05-05:00
fix comment and shuffle some more code
- - - - -
fed37946 by Shane Snyder at 2016-06-09T12:06:25-05:00
convert to a single module shutdown function
rather than using 'begin_shutdown', 'get_output_data', and
'shutdown', we combine all three into one shutdown routine.
- - - - -
660130d9 by Shane Snyder at 2016-06-09T12:07:28-05:00
posix mod cleanup + shutdown func update
- - - - -
abf1d5ea by Shane Snyder at 2016-06-10T09:57:50-05:00
some more POSIX and common code cleanup
- - - - -
31e76e2c by Shane Snyder at 2016-06-10T09:58:06-05:00
update MPI-IO module to reflect recent changes
- - - - -
533d7fb2 by Shane Snyder at 2016-06-10T12:30:42-05:00
port hdf5 and pnetcdf modules to new API changes
- - - - -
a974efee by Shane Snyder at 2016-06-10T16:05:28-05:00
Merge branch 'master' into mmap-dev
Conflicts:
darshan-runtime/Makefile.in
darshan-runtime/configure
darshan-runtime/configure.in
darshan-runtime/darshan.h
darshan-runtime/lib/darshan-core.c
darshan-runtime/lib/darshan-hdf5.c
darshan-runtime/lib/darshan-mpiio.c
darshan-runtime/lib/darshan-pnetcdf.c
darshan-runtime/lib/darshan-posix.c
darshan-util/Makefile.in
darshan-util/darshan-convert.c
darshan-util/darshan-logutils.c
darshan-util/darshan-logutils.h
darshan-util/darshan-parser.c
- - - - -
69c954df by Shane Snyder at 2016-06-10T16:15:49-05:00
fix out-of-tree makefile bug in darshan-util
- - - - -
d599c583 by Shane Snyder at 2016-06-13T12:39:49-05:00
modifications for sorting module records
A common record comparision/sort routine is now defined in
darshan-common which can be used by any modules with fixed-length
records and whose records first field is the darshan_base_record.
The comparison was also updated to sort secondarily by darshan
record identifier to resolve a potential bug in the way we have
traditionally sorted the records.
- - - - -
30c06099 by Shane Snyder at 2016-06-13T21:44:26+00:00
fix compile error when mmap is disabled
- - - - -
c407379a by Shane Snyder at 2016-06-14T16:08:02+00:00
bug fix in initializing static modules
- - - - -
fba89f42 by Shane Snyder at 2016-06-14T16:08:36+00:00
port bgq module over to new interface changes
- - - - -
246c491e by Shane Snyder at 2016-06-14T16:56:34+00:00
parser shouldn't abort with no-name records
- - - - -
f5ff61de by Shane Snyder at 2016-06-14T14:46:15-05:00
simplify common counter logic
- - - - -
f6769e56 by Shane Snyder at 2016-06-16T19:21:22-05:00
logutil backwards compatibility for name records
- - - - -
936d1e10 by Shane Snyder at 2016-06-17T13:27:31-05:00
cleanup comments in darshan-common
- - - - -
538812e4 by Shane Snyder at 2016-06-17T14:29:13-05:00
cleanup makefile & autoconf mmap detection
- - - - -
080c680d by Shane Snyder at 2016-06-17T14:32:07-05:00
fix comment in darshan-config
- - - - -
927b8773 by Shane Snyder at 2016-06-17T16:39:58-05:00
ifdef wrap mmap #define
- - - - -
bb9d68c0 by Shane Snyder at 2016-06-17T16:40:27-05:00
honor DARSHAN_MODMEM env variable
- - - - -
ba0f880b by Shane Snyder at 2016-06-17T16:51:11-05:00
fix comments in module log format headers
- - - - -
b70be200 by Shane Snyder at 2016-06-17T18:29:14-05:00
port NULL module over to new API changes
- - - - -
0d2cb76b by Shane Snyder at 2016-06-17T20:13:21-05:00
text edits in darshan-runtime docs
- - - - -
ddd83318 by Shane Snyder at 2016-06-20T15:47:44-05:00
bug fix to byte swap module version array
- - - - -
f3afdb75 by Shane Snyder at 2016-06-20T16:03:11-05:00
move back-compat code out of dedicated directory
- - - - -
6ccaea46 by Shane Snyder at 2016-06-20T20:50:29-05:00
bgq backwards compatibility code
- - - - -
bb75028f by Shane Snyder at 2016-06-20T20:50:29-05:00
remove old modularization design docs
- - - - -
099470b3 by Shane Snyder at 2016-06-20T21:02:22-05:00
Merge remote-tracking branch 'origin/mmap-dev'
Merging mmap feature development branch back into master. Darshan
can now mmap its data structures to temporary log files at runtime to
protect against applications that do not call MPI_Finalize. Feature
can be enabled at configure time using '--enable-mmap-logs'.
- - - - -
c68b00a9 by Shane Snyder at 2016-06-21T08:58:35-05:00
eliminate possible shutdown timing skew
- - - - -
eca097a2 by Shane Snyder at 2016-06-21T12:33:12-05:00
small makefile fixes
- - - - -
1a67e237 by Phil Carns at 2016-06-22T14:53:53-04:00
Merge remote-tracking branch 'origin/master' into dev-stdio
- note: darshan-stdio module updates are still needed; this merge just
updates the rest of the code and resolves conflicts
- - - - -
23 changed files:
- ChangeLog
- darshan-bgq-log-format.h
- darshan-hdf5-log-format.h
- darshan-log-format.h
- darshan-mpiio-log-format.h
- darshan-null-log-format.h
- darshan-pnetcdf-log-format.h
- darshan-posix-log-format.h
- darshan-runtime/Makefile.in
- darshan-runtime/configure
- darshan-runtime/configure.in
- darshan-runtime/darshan-common.h
- darshan-runtime/darshan-config.in
- darshan-runtime/darshan-core.h
- darshan-runtime/darshan-gen-cxx.pl.in
- darshan-runtime/darshan-gen-fortran.pl.in
- darshan-runtime/darshan-runtime-config.h.in
- darshan-runtime/darshan.h
- darshan-runtime/doc/darshan-runtime.txt
- darshan-runtime/lib/darshan-bgq.c
- darshan-runtime/lib/darshan-common.c
- darshan-runtime/lib/darshan-core.c
- darshan-runtime/lib/darshan-hdf5.c
The diff was not included because it is too large.
View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/compare/9cd5e5c01a2e346b10a4479340e6a2c73efa44cd...1a67e237f400fbc4c09c77251e2798ed7fc6f93d
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-commits/attachments/20160622/fdb0bad9/attachment-0001.html>
More information about the Darshan-commits
mailing list