[Darshan-commits] [Git][darshan/darshan][master] 83 commits: first cut at mmap logic -- posix mod only for now

Shane Snyder xgitlab at cels.anl.gov
Mon Jun 20 21:25:49 CDT 2016 

Shane Snyder pushed to branch master at darshan / darshan


Commits:
0d84b5e5 by Shane Snyder at 2015-10-12T14:42:13-05:00
first cut at mmap logic -- posix mod only for now

- - - - -
c3c5e597 by Shane Snyder at 2015-10-13T16:44:32-05:00
more mmap logic and instrumentation for testing

- - - - -
e810279e by Shane Snyder at 2015-11-11T15:43:27-06:00
Merge branch 'dev-modular' into mmap-dev

Conflicts:
	darshan-runtime/darshan-core.h
	darshan-runtime/lib/darshan-core.c

- - - - -
799d9890 by Shane Snyder at 2015-11-24T14:45:31-06:00
update logutils to allow logs with only job info

- - - - -
bdb2a5f0 by Shane Snyder at 2015-12-01T12:40:28-06:00
runtime now mmaps header and job data to tmp log

- - - - -
3b0c668b by Shane Snyder at 2015-12-02T14:36:19-06:00
update logutils to read uncompressed log files

- - - - -
4f1b27ad by Shane Snyder at 2015-12-02T15:18:33-06:00
bug fix in mmap region pointer assignment

- - - - -
b1791172 by Shane Snyder at 2015-12-03T16:20:34-06:00
updated logutils hash read deserialization method

- - - - -
523b19e4 by Shane Snyder at 2015-12-03T16:22:25-06:00
update core runtime record hash implementation

Now, as the app executes, darshan-core persists the id->name
mapping data to the tmp log file. This is done by serializing the
id->name mapping to the mmap log file region when new records are
registered.

- - - - -
376eefa8 by Shane Snyder at 2015-12-04T10:20:47-06:00
modify register_record routine used by modules

Now, register_record() accepts an argument for the size of
the record being registered, and this is used to track how much
data each module has stored in their allocated memory region.

- - - - -
9193b34a by Shane Snyder at 2015-12-04T11:10:14-06:00
add new utility to stitch together darshan logs

- - - - -
c9618c8a by Shane Snyder at 2015-12-04T13:52:19-06:00
runtime now stores mnt & exe data at all ranks

- - - - -
4ac75905 by Shane Snyder at 2015-12-04T15:18:06-06:00
modify hash write routine to use new serialization

- - - - -
66a83b03 by Shane Snyder at 2015-12-07T13:25:52-06:00
first cut at the stitch logs utility

Currently, this tool can take a set of temporary, uncompressed log
files on a given node, and combine into a single darshan log.
Next step would be to modify this code to perform shared file
reduction logic to combine per-node log files into per-job log
files.

- - - - -
cb1c7ace by Shane Snyder at 2015-12-08T15:04:41-06:00
change runtime darshan record_ref struct

- - - - -
9d7f638b by Shane Snyder at 2015-12-08T15:16:19-06:00
change util record ref structure def

- - - - -
05acb6d4 by Shane Snyder at 2015-12-08T16:07:43-06:00
introduce base record concept in posix mod

- - - - -
ed103a2a by Shane Snyder at 2015-12-08T16:22:23-06:00
add base record concept to utilities

- - - - -
be3912a8 by Shane Snyder at 2015-12-09T10:28:40-06:00
update utilities and log format to use base rec

- - - - -
fe340438 by Shane Snyder at 2015-12-14T12:48:15-06:00
add an aggregation routine to module logutils

- - - - -
a84e8657 by Shane Snyder at 2015-12-17T16:54:46-06:00
integrate normal darshan shutdown back into code

- - - - -
80c07c27 by Shane Snyder at 2015-12-17T16:55:11-06:00
more darshan-util updates for handling tmp logs

- - - - -
cc498bfa by Shane Snyder at 2016-01-14T16:16:55-06:00
minor reorganization of some runtime code

- - - - -
b989ffe4 by Shane Snyder at 2016-01-14T16:20:15-06:00
integrate new darshan-diff util in mmap branch

- - - - -
c52fed2a by Shane Snyder at 2016-01-14T16:22:32-06:00
typos in darshan-diff

- - - - -
032f99b4 by Shane Snyder at 2016-01-14T16:42:37-06:00
forgot to install diff utility in makefile

- - - - -
d3fae782 by Shane Snyder at 2016-01-18T21:42:11-06:00
update how module & record registration

- modules now receive a memory buffer after successfully
registering with darshan-core. modules can request a specific
memory amount, but may not be honored depending on the configured
darshan memory limit
- split the previous register_record function into 2 parts:
    * lookup() -> lookup darshan_record_id corresp. to path name
    * register() -> register record_id & name with darshan-core
- register_record now returns true if the record added; false
otherwise (if module was out of memory or darshan is out of memory
for storing id->name mappings)

- - - - -
f78c456f by Shane Snyder at 2016-01-19T12:57:25-06:00
apply mmap code changes to other i/o modules

- - - - -
265bdaba by Shane Snyder at 2016-01-19T16:18:22-06:00
Merge branch 'dev-modular' into mmap-dev

Conflicts:
	darshan-runtime/Makefile.in
	darshan-runtime/darshan-core.h
	darshan-runtime/lib/darshan-core.c
	darshan-util/Makefile.in
	darshan-util/darshan-bgq-logutils.c
	darshan-util/darshan-diff.c
	darshan-util/darshan-hdf5-logutils.c
	darshan-util/darshan-logutils.c
	darshan-util/darshan-mpiio-logutils.c
	darshan-util/darshan-null-logutils.c
	darshan-util/darshan-parser.c
	darshan-util/darshan-pnetcdf-logutils.c
	darshan-util/darshan-posix-logutils.c

- - - - -
96801d4a by Shane Snyder at 2016-01-19T17:44:11-06:00
fix post-merge build errors

dev-modular should be completely merged into mmap-dev branch

- - - - -
f57f753c by Shane Snyder at 2016-01-20T16:03:28-06:00
bug fixes from merge of mmap-dev to dev-modular

- - - - -
10c3b107 by Shane Snyder at 2016-01-20T22:21:28-06:00
bug fix in log aggregation tool

- - - - -
d3ebeb50 by Shane Snyder at 2016-01-20T22:21:43-06:00
add aggregation capability to i/o modules

- - - - -
ac6c6442 by Shane Snyder at 2016-01-22T10:08:27-06:00
bug fix in darshan put_header routine

- - - - -
d5abad0f by Shane Snyder at 2016-01-22T10:53:52-06:00
remove profiling macros from posix module

- - - - -
bb25083b by Shane Snyder at 2016-01-25T13:19:59-06:00
rename merge log utility

- - - - -
e81a9cae by Shane Snyder at 2016-01-25T15:56:06-06:00
autconf/runtime changes for configuring mmap code

- - - - -
8070c831 by Shane Snyder at 2016-01-26T12:59:32-06:00
update command line args for log merging utility

- - - - -
b4be0abd by Shane Snyder at 2016-01-26T13:00:11-06:00
allreduce job start time in darshan-core init.

- - - - -
5f4e06f6 by Shane Snyder at 2016-01-26T13:02:54-06:00
autogenerate an example epilog script

- - - - -
c276e4c2 by Shane Snyder at 2016-01-29T13:00:26-06:00
first cut at unifying mmap feature with darshan

- - - - -
a22233d7 by Shane Snyder at 2016-01-29T15:40:17-06:00
improve slurm epilog script

epilog will now skip on-node compression if only a single compute
node is used and will generate the final log file instead

- - - - -
bb625cbd by Shane Snyder at 2016-02-05T13:53:21-06:00
allow common counters to be updated on the go

- - - - -
37b1bb02 by Shane Snyder at 2016-02-05T17:27:54-06:00
add common counter aggregation to logutils

- - - - -
f26baefc by Shane Snyder at 2016-02-26T12:34:02-06:00
allow job end time to be set by darshan-merge

- - - - -
d4a3d112 by Shane Snyder at 2016-02-29T17:22:57-06:00
add variance counter reduction logic to logutils

- - - - -
04063da7 by Shane Snyder at 2016-03-01T17:43:38-06:00
common access counters are now sorted in logs

common value counters are sorted first by decreasing counts,
and then by decreasing size.

- - - - -
94430f41 by Shane Snyder at 2016-03-01T18:30:08-06:00
bug fix in aggregating common accesses

- - - - -
d6dad7f3 by Shane Snyder at 2016-05-26T12:29:34-05:00
add variable-length definition of name records

also some code clean up to change references of record hash
to name hash to be more clear about what it is actually storing.

- - - - -
999ee9c8 by Shane Snyder at 2016-05-26T12:53:05-05:00
darshan-util edits for var. length name records

- - - - -
8f463e51 by Shane Snyder at 2016-05-27T15:10:12-05:00
remove shared name records on nonzero ranks

- - - - -
7d35c45a by Shane Snyder at 2016-05-27T16:36:32-05:00
register/lookup_record now use strings explicitly

- - - - -
bfde9eb7 by Shane Snyder at 2016-05-31T15:47:51-05:00
numerous darshan-core api changes

- memory management is now entirely handled by darshan-core for
  module records
    * when registering new records, darshan-core returns the
      corresponding buffer to write it to, rather than allowing
      module to determine this
- modules now decide whether darshan-core should track record
  names by passing either a valid name or NULL to the register
  function
    * this allows modules to register records that aren't really
      tracked by darshan-core
- only the POSIX module has been ported to new changes for this
  commit

- - - - -
9bfc7b65 by Shane Snyder at 2016-06-06T17:16:31-05:00
add record_ref interface to darshan-common

This interface provides some convenience functions for mapping
opaque record identifiers (could be a filename, file descriptor,
or any other type of file handle) to a corresponding runtime
structure referencing the actual file record. This runtime
structure should at least contain a pointer to the actual file
record, and can also persist arbitrary state for this record.

- - - - -
d8f3575a by Shane Snyder at 2016-06-08T16:58:23-05:00
darshan-core code and documentation cleanup

Cleanup the structure and documenation of the code to be in sync
with the new mmap version of Darshan. In particular, I made sure
all extra mmap logic was properly ifdef wrapped so it's more clear
which portions of the code are responsible for implementing that
feature.

- - - - -
54baa7dd by Shane Snyder at 2016-06-08T17:34:05-05:00
fix comment and shuffle some more code

- - - - -
fed37946 by Shane Snyder at 2016-06-09T12:06:25-05:00
convert to a single module shutdown function

rather than using 'begin_shutdown', 'get_output_data', and
'shutdown', we combine all three into one shutdown routine.

- - - - -
660130d9 by Shane Snyder at 2016-06-09T12:07:28-05:00
posix mod cleanup + shutdown func update

- - - - -
abf1d5ea by Shane Snyder at 2016-06-10T09:57:50-05:00
some more POSIX and common code cleanup

- - - - -
31e76e2c by Shane Snyder at 2016-06-10T09:58:06-05:00
update MPI-IO module to reflect recent changes

- - - - -
533d7fb2 by Shane Snyder at 2016-06-10T12:30:42-05:00
port hdf5 and pnetcdf modules to new API changes

- - - - -
a974efee by Shane Snyder at 2016-06-10T16:05:28-05:00
Merge branch 'master' into mmap-dev

Conflicts:
	darshan-runtime/Makefile.in
	darshan-runtime/configure
	darshan-runtime/configure.in
	darshan-runtime/darshan.h
	darshan-runtime/lib/darshan-core.c
	darshan-runtime/lib/darshan-hdf5.c
	darshan-runtime/lib/darshan-mpiio.c
	darshan-runtime/lib/darshan-pnetcdf.c
	darshan-runtime/lib/darshan-posix.c
	darshan-util/Makefile.in
	darshan-util/darshan-convert.c
	darshan-util/darshan-logutils.c
	darshan-util/darshan-logutils.h
	darshan-util/darshan-parser.c

- - - - -
69c954df by Shane Snyder at 2016-06-10T16:15:49-05:00
fix out-of-tree makefile bug in darshan-util

- - - - -
d599c583 by Shane Snyder at 2016-06-13T12:39:49-05:00
modifications for sorting module records

A common record comparision/sort routine is now defined in
darshan-common which can be used by any modules with fixed-length
records and whose records first field is the darshan_base_record.
The comparison was also updated to sort secondarily by darshan
record identifier to resolve a potential bug in the way we have
traditionally sorted the records.

- - - - -
30c06099 by Shane Snyder at 2016-06-13T21:44:26+00:00
fix compile error when mmap is disabled

- - - - -
c407379a by Shane Snyder at 2016-06-14T16:08:02+00:00
bug fix in initializing static modules

- - - - -
fba89f42 by Shane Snyder at 2016-06-14T16:08:36+00:00
port bgq module over to new interface changes

- - - - -
246c491e by Shane Snyder at 2016-06-14T16:56:34+00:00
parser shouldn't abort with no-name records

- - - - -
f5ff61de by Shane Snyder at 2016-06-14T14:46:15-05:00
simplify common counter logic

- - - - -
f6769e56 by Shane Snyder at 2016-06-16T19:21:22-05:00
logutil backwards compatibility for name records

- - - - -
936d1e10 by Shane Snyder at 2016-06-17T13:27:31-05:00
cleanup comments in darshan-common

- - - - -
538812e4 by Shane Snyder at 2016-06-17T14:29:13-05:00
cleanup makefile & autoconf mmap detection

- - - - -
080c680d by Shane Snyder at 2016-06-17T14:32:07-05:00
fix comment in darshan-config

- - - - -
927b8773 by Shane Snyder at 2016-06-17T16:39:58-05:00
ifdef wrap mmap #define

- - - - -
bb9d68c0 by Shane Snyder at 2016-06-17T16:40:27-05:00
honor DARSHAN_MODMEM env variable

- - - - -
ba0f880b by Shane Snyder at 2016-06-17T16:51:11-05:00
fix comments in module log format headers

- - - - -
b70be200 by Shane Snyder at 2016-06-17T18:29:14-05:00
port NULL module over to new API changes

- - - - -
0d2cb76b by Shane Snyder at 2016-06-17T20:13:21-05:00
text edits in darshan-runtime docs

- - - - -
ddd83318 by Shane Snyder at 2016-06-20T15:47:44-05:00
bug fix to byte swap module version array

- - - - -
f3afdb75 by Shane Snyder at 2016-06-20T16:03:11-05:00
move back-compat code out of dedicated directory

- - - - -
6ccaea46 by Shane Snyder at 2016-06-20T20:50:29-05:00
bgq backwards compatibility code

- - - - -
bb75028f by Shane Snyder at 2016-06-20T20:50:29-05:00
remove old modularization design docs

- - - - -
099470b3 by Shane Snyder at 2016-06-20T21:02:22-05:00
Merge remote-tracking branch 'origin/mmap-dev'

Merging mmap feature development branch back into master. Darshan
can now mmap its data structures to temporary log files at runtime to
protect against applications that do not call MPI_Finalize. Feature
can be enabled at configure time using '--enable-mmap-logs'.

- - - - -


20 changed files:

- darshan-bgq-log-format.h
- darshan-hdf5-log-format.h
- darshan-log-format.h
- darshan-mpiio-log-format.h
- darshan-null-log-format.h
- darshan-pnetcdf-log-format.h
- darshan-posix-log-format.h
- darshan-runtime/Makefile.in
- darshan-runtime/configure
- darshan-runtime/configure.in
- darshan-runtime/darshan-common.h
- darshan-runtime/darshan-config.in
- darshan-runtime/darshan-core.h
- darshan-runtime/darshan-runtime-config.h.in
- darshan-runtime/darshan.h
- darshan-runtime/doc/darshan-runtime.txt
- darshan-runtime/lib/darshan-bgq.c
- darshan-runtime/lib/darshan-common.c
- darshan-runtime/lib/darshan-core.c
- darshan-runtime/lib/darshan-hdf5.c


The diff was not included because it is too large.


View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/compare/a6e8682980f49039d4f15194840139857291f216...099470b3c45fcc3790b03f7992bb953a838239c2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-commits/attachments/20160620/a50ed965/attachment-0001.html>

More information about the Darshan-commits mailing list