[Darshan-commits] [Git][darshan/darshan][autoperf-mod-update] 113 commits: Initial setup for a python package including documenting stubs for

Kevin Harms xgitlab at cels.anl.gov
Mon Feb 1 15:25:24 CST 2021 


Kevin Harms pushed to branch autoperf-mod-update at darshan / darshan


Commits:
ff8e0a65 by Jakob Luettgau at 2019-08-01T10:32:56-05:00
Initial setup for a python package including documenting stubs for
various tests (pytest, coverage, lint/flake8) and make targets for easy
releases.

- - - - -
615f7363 by Jakob Luettgau at 2019-08-01T13:58:24-05:00
Replace with actual tests.

- - - - -
17db029e by Jakob Luettgau at 2019-08-13T09:15:59-05:00
Isolate stdio-segfault/could not inflate issue. Update examples and notebook.

- - - - -
bbdb90e7 by Jakob Luettgau at 2019-09-09T17:19:01-05:00
Merge branch 'master' into python-package

- - - - -
7454041c by Jakob Luettgau at 2019-09-11T18:19:30-05:00
Add object wrapper to cffi based parser for more pythonic interaction.

- - - - -
9e2140f4 by Jakob Luettgau at 2019-09-11T18:19:45-05:00
Merge branch 'python-package' of xgitlab.cels.anl.gov:jluettgau/darshan into python-package

- - - - -
a630ddc9 by Jakob Luettgau at 2019-09-11T18:38:15-05:00
Include example, and move example plots into renderer specific module.

- - - - -
d6b07f76 by Jakob Luettgau at 2019-09-17T21:27:56-05:00
Port summary generation to pydarshan object wrapper.

- - - - -
2e9dddc2 by Jakob Luettgau at 2019-12-28T14:31:01+01:00
Add basic dxt support (without segments). Updates to documentation, and example notebooks.

- - - - -
1c3d6d4d by Jakob Luettgau at 2019-12-30T22:37:51+01:00
Add log_get_dxt_record to python cffi backend and report object wrapper.

- - - - -
8b2b4734 by Jakob Luettgau at 2020-01-04T14:57:57+01:00
Have cffi backend cache module index and name_records along with log handle as they are often used internally.

- - - - -
6ba187a4 by Jakob Luettgau at 2020-01-05T23:39:38+01:00
Add timeline report generator.

- - - - -
ac6e8ca0 by Jakob Luettgau at 2020-02-24T16:57:22+01:00
Add additional documentation, simplefy documentation.

- - - - -
e587da8c by Jakob Luettgau at 2020-02-24T17:21:32+01:00
Merge branch 'master' into python-package

- - - - -
e0773049 by Jakob Luettgau at 2020-02-26T18:37:07+01:00
Refactoring to move aggregations to experimental for the time being. Convieniently reenable by calling darshan.enable_experimental() after import darshan.

- - - - -
ed91ae3c by Jakob Luettgau at 2020-02-26T19:18:56+01:00
Have notebooks with some of the expected outputs preserved.

- - - - -
18bfe771 by Jakob Luettgau at 2020-02-26T19:56:18+01:00
Note about experimental switch.

- - - - -
42a6f6de by Jakob Luettgau at 2020-04-11T00:05:22+02:00
Prepare seperation of raw data and summary namespaces, add provenance and consistency structures for report algebra.

- - - - -
5de67837 by Jakob Luettgau at 2020-04-11T00:42:28+02:00
Update jupyter notebooks to reflect refactorings (moving plots to experimental, seperating namespaces)

- - - - -
02f01e1c by Jakob Luettgau at 2020-04-11T20:47:30+02:00
Updata tests: stdio - cherry pick some value to verify, report - reflect switch from r.report to r.data.

- - - - -
41212c28 by Jakob Luettgau at 2020-04-12T03:37:45+02:00
Add additional notebooks for different features. Enable report algebra
for simple counters. Switch default plots to use report objects.

- - - - -
aaacffa0 by Jakob Luettgau at 2020-04-13T16:15:06+02:00
Add experimental filter and reduce methods.

- - - - -
88bd7814 by Jakob Luettgau at 2020-05-03T18:15:44+02:00
Homogenize aggregators, use summary namespace to store results.

- - - - -
2da3a81e by Jakob Luettgau at 2020-05-06T14:05:07+02:00
Enable function chaining for filter and reduce. Remove obsolete.

- - - - -
a9ce0631 by Jakob Luettgau at 2020-05-06T23:06:50+02:00
PEP8.

- - - - -
f2e7cd1c by Jakob Luettgau at 2020-05-12T19:01:45+02:00
PEP8.

- - - - -
02055591 by Jakob Luettgau at 2020-05-14T13:07:18+02:00
Merge remote-tracking branch 'upstream/master' into python-package

- - - - -
4184cf60 by Jakob Luettgau at 2020-05-14T20:39:42+02:00
Add spackage stub.

- - - - -
90e8d4c9 by Jakob Luettgau at 2020-06-19T21:40:25+02:00
Add two examples for pydarshan utility wrappers with hardcoded prefix on install.

- - - - -
05e30606 by Jakob Luettgau at 2020-06-19T21:42:52+02:00
Update whitelist for distribution when installing with python setup.py install.

- - - - -
4d763398 by Jakob Luettgau at 2020-06-20T20:30:11+02:00
Add experimental filtered namerecord table reconstruction to libdarshan-utils.

- - - - -
10a7898f by Jakob Luettgau at 2020-06-20T20:32:40+02:00
Enable experimental get_filtered_name_records for python bindings, add test.

- - - - -
e0b4befb by Jakob Luettgau at 2020-06-22T11:30:25+02:00
Reenable eager utils discovery but ensure LD_LIBRARY_PATH takes precedence.

- - - - -
ea927721 by Jakob Luettgau at 2020-06-22T11:33:03+02:00
Add verbosity when opening file failed (e.g., filepath provided does not exist).

- - - - -
cd295f81 by Jakob Luettgau at 2020-06-22T11:42:53+02:00
Add test for preliminary filtered name record helper.

- - - - -
572b2ff6 by Jakob Luettgau at 2020-06-22T15:40:17+02:00
Update notebook examples.

- - - - -
744ae290 by Jakob Luettgau at 2020-06-22T15:40:55+02:00
Remove outdated demo.

- - - - -
a988f682 by Jakob Luettgau at 2020-06-22T15:44:20+02:00
Merge tag 'darshan-3.2.1' into python-package

Darshan 3.2.1

- - - - -
23e9b467 by Shane Snyder at 2020-06-25T11:18:25-05:00
updated py README to indicate correct lib ver

- - - - -
8b2ee4c2 by Shane Snyder at 2020-06-25T11:19:07-05:00
pandas should be dev requirement, too

- - - - -
aab101a1 by Shane Snyder at 2020-06-25T11:19:42-05:00
fix bad file name

- - - - -
4b5d5aef by Shane Snyder at 2020-06-25T11:23:11-05:00
fix bug in tox.ini to reference right py dirs

- - - - -
9d20e69b by Shane Snyder at 2020-06-29T11:36:16-05:00
autotools support for pydarshan

- - - - -
c573b8e6 by Shane Snyder at 2020-06-29T11:36:29-05:00
fix configure-generated example pydarshan tools

- - - - -
f382e96a by Jakob Luettgau at 2020-07-02T22:11:52+02:00
Remove debug outputs.

- - - - -
9a68fed8 by Jakob Luettgau at 2020-07-03T00:45:02+02:00
Add cli submodule, with a few subcommands as blueprints.

- - - - -
2a04e59d by Jakob Luettgau at 2020-07-03T00:45:12+02:00
Merge branch 'python-package' of xgitlab.cels.anl.gov:jluettgau/darshan into python-package

- - - - -
612a313e by Jakob Luettgau at 2020-07-07T17:39:36+02:00
Remove travis CI.

- - - - -
5bc35e44 by Jakob Luettgau at 2020-07-07T17:51:17+02:00
Cleanup report and cffi backend.

- - - - -
e894d79f by Jakob Luettgau at 2020-07-07T17:55:33+02:00
Remove debug.py in favor of using logging.

- - - - -
6f1ee4f7 by Jakob Luettgau at 2020-07-07T18:31:37+02:00
Update and cleanup example notebooks (purge long outputs).

- - - - -
983828ad by Shane Snyder at 2020-07-07T11:33:20-05:00
add util fn for getting library verion

- - - - -
5f55032d by Shane Snyder at 2020-07-07T11:33:20-05:00
make --enable-shared required with pydarshan

- - - - -
0b5bba13 by Shane Snyder at 2020-07-09T12:15:29-05:00
sync api_def_c defs with current master defs

- - - - -
7d650c39 by Shane Snyder at 2020-07-09T12:16:19-05:00
bug fix in dxt read segments

- - - - -
98b2e23c by Shane Snyder at 2020-07-09T12:18:43-05:00
bug fixes for get_generic_record

- counters are int64, not uint64
- accidentally assigning counters values to fcounter array

- - - - -
93d742a0 by Shane Snyder at 2020-07-09T12:20:35-05:00
more functions for new modules (h5f/h5d, lustre)

- - - - -
82e57531 by Shane Snyder at 2020-07-09T12:20:58-05:00
sync report defs with currently supported modules

- - - - -
a1da405c by Shane Snyder at 2020-07-09T12:24:40-05:00
add example log file with lots of different mods

includes POSIX, STDIO, MPI-IO, LUSTRE, H5F, H5D, and DXT

- - - - -
196eae16 by Shane Snyder at 2020-07-09T12:47:33-05:00
Merge branch 'master' into 'python-package'

# Conflicts:
#   darshan-util/darshan-logutils.c
- - - - -
447fb73f by Jakob Luettgau at 2020-07-23T17:14:20+02:00
Add version check (enforce first three version positions to match, thus allow pydarshan to advance x.x.x.*).

- - - - -
94d3a016 by Jakob Luettgau at 2020-07-23T22:36:13+02:00
Add low-level interaction notebook example.

- - - - -
9c4ed094 by Jakob Luettgau at 2020-07-23T22:45:36+02:00
Update notebooks and part of the documention.

- - - - -
b451c8c8 by Jakob Luettgau at 2020-07-24T13:25:50+02:00
Update tox.ini to include python 3.8.

- - - - -
94573e9b by Jakob Luettgau at 2020-07-30T15:51:21+02:00
Add graph compilation example with holoviews.

- - - - -
c82301c4 by Jakob Luettgau at 2020-07-31T01:40:44+02:00
Cleanup and include example logs.

- - - - -
4912d57e by Jakob Luettgau at 2020-08-06T12:23:44+02:00
Refactor the mode switch to be named dtype, update documentation.

- - - - -
0f44520b by Jakob Luettgau at 2020-08-06T12:37:32+02:00
Change format mode flag from 'mode=..' to 'dtype=..'.

- - - - -
72682867 by Jakob Luettgau at 2020-08-06T13:44:41+02:00
Auto-combine pandas records into single dataframe.

- - - - -
c7b0910d by Jakob Luettgau at 2020-08-06T14:02:19+02:00
Update dxt record methods to conform to dtype format choice as well.

- - - - -
cd732a8f by Jakob Luettgau at 2020-08-06T14:37:05+02:00
Add isolated transformation example from dxt-timeline. (dxt2png).

- - - - -
9f12dd54 by Jakob Luettgau at 2020-08-06T14:38:40+02:00
Add comments, note about extra dependencies.

- - - - -
4e20ef37 by Jakob Luettgau at 2020-09-02T22:27:37+02:00
Update docs.

- - - - -
b6fb669b by Jakob Luettgau at 2020-09-02T22:31:52+02:00
Update documentation.

- - - - -
8db50f35 by Jakob Luettgau at 2020-09-03T14:33:22+02:00
Catch up with and merge upstream.

- - - - -
932d6999 by Jakob Luettgau at 2020-09-03T19:46:51+02:00
Collapse low-level log_get_*_record into log_get_record which takes modstring. Low-level lustre record now includes ost_ids.

- - - - -
d30ee690 by Jakob Luettgau at 2020-09-17T11:25:09+02:00
Minor updates to documentation (warning, and note markup). Remove some dead code.

- - - - -
8190931a by Jakob Luettgau at 2020-09-18T15:01:16+02:00
Have pydarshan use pythons logging facilities.

- - - - -
1e67c1db by Jakob Luettgau at 2020-10-22T14:35:30+02:00
Update report.to_dict to tolerate missing counters/fcounters.

- - - - -
f849814f by Jakob Luettgau at 2020-10-22T15:17:10+02:00
Update documentation.

- - - - -
c16aa8a2 by Jakob Luettgau at 2020-10-22T17:00:17+02:00
Add release checklist for PyDarshan.

- - - - -
91bdad64 by Jakob Luettgau at 2020-11-11T23:07:39+01:00
Make sure dtype parameter is used consistently.

- - - - -
e789f838 by Jakob Luettgau at 2020-11-21T12:32:35+01:00
Add scripts to build release wheels with darshan-utils included.

- - - - -
623dd590 by Jakob Luettgau at 2020-11-21T22:01:18+01:00
Add additional targets to wheel building automation, fix long_description to only inlcude statements compatibile to pypi's rst parser.

- - - - -
a279d721 by Jakob Luettgau at 2020-11-23T23:56:56+01:00
Add finds to gather wheels into dist before release.

- - - - -
e69f1f9a by Jakob Luettgau at 2020-11-24T16:00:16+01:00
Add extension.c to ease including additional shared libraries. Various small changes and bugfixes: libdarshan-util discovery/precedence, timeline aggregation, setup.py.

- - - - -
bf59c8e6 by Jakob Luettgau at 2020-11-24T16:16:21+01:00
Update PyDarshan release checklist.

- - - - -
52070a58 by Shane Snyder at 2020-12-08T14:20:14-06:00
remove autoconf-generated Makefile

- - - - -
c05399ed by Shane Snyder at 2020-12-09T16:51:08-06:00
delete sphinx-generated api docs

- - - - -
6d938263 by Shane Snyder at 2020-12-10T08:25:30-06:00
Merge remote-tracking branch 'upstream/master' into python-package

- - - - -
e33f2502 by Shane Snyder at 2020-12-10T14:05:02-06:00
update pytest tests for new API changes

- - - - -
62bf607b by Shane Snyder at 2020-12-10T14:09:42-06:00
add CHANGELOG for pydarshan

- - - - -
e218f068 by Shane Snyder at 2020-12-10T14:10:05-06:00
small tweaks to release checklist

- - - - -
24029dcc by Shane Snyder at 2020-12-10T14:10:57-06:00
updated pydarshan versions for latest tests

- - - - -
fd5801f5 by Jakob Luettgau at 2020-12-17T12:02:12+01:00
Clean up setup.py, add target for devenv, extend ignore patterns for git/pytest

- - - - -
23fceb41 by Jakob Luettgau at 2020-12-17T12:15:29+01:00
Remove experiments for logs with extremely a large number of name records from test directory.

- - - - -
013a0ddd by Jakob Luettgau at 2020-12-17T13:25:06+01:00
Cleanup tests a little bit (disable flake for tox), update instructions to simplify test procedure (different python versions should be moved to CI). Add a convienience target to quickly create a dev-environment with python-deps + libdarshanutil.

- - - - -
154952fa by Jakob Luettgau at 2020-12-17T15:54:00+01:00
Add a brief tutorial directory to the examples.

- - - - -
b252fc4a by Shane Snyder at 2021-01-13T21:05:20-06:00
workaround to dlopen darshan-util libraries

- - - - -
8429a8e5 by Shane Snyder at 2021-01-13T21:07:43-06:00
bumped pydarshan versions for new pypi release

- - - - -
18486628 by Jakob Luettgau at 2021-01-14T19:19:53+01:00
Allow easy debugging via logging for darshan-utils discovery.

- - - - -
54ad42ba by Jakob Luettgau at 2021-01-14T19:24:11+01:00
Resolve merge conflict.

- - - - -
ebddd484 by Jakob Luettgau at 2021-01-25T18:22:05+01:00
Fix cleanup for devenv make target.

- - - - -
f6eb53fb by Jakob Luettgau at 2021-01-25T18:55:34+01:00
Examples: Report-Algebra: Ensure all logs are opend in dtype=numpy mode so current merge/add/etc. operations work.

- - - - -
54d425d4 by Jakob Luettgau at 2021-01-26T18:34:01+01:00
Fix bumpversion using pydarshan instead of darshan subdir.

- - - - -
b498f867 by Jakob Luettgau at 2021-01-26T18:36:30+01:00
Have pydarshan module report it's own version correctly via darshan.__version__

- - - - -
c4bd287d by Jakob Luettgau at 2021-01-26T18:38:02+01:00
Update bumpversion config (do not automatically create a new tag for pydarshan releases in git.)

- - - - -
b8c412c8 by Jakob Luettgau at 2021-01-26T18:39:25+01:00
Update pydarshan setup.cfg (Don't create a bumpversion commit automatically.)

- - - - -
67f10148 by Jakob Luettgau at 2021-01-26T19:54:58+01:00
Bump pydarshan to version 0.0.6

- - - - -
1af1f5c2 by Jakob Luettgau at 2021-01-27T13:50:02+01:00
Add quick switch for verbose_discovery, refine cli argument parser to allow early exit and more debug control.

- - - - -
a0524fd9 by Jakob Luettgau at 2021-01-27T14:00:05+01:00
Add a test to quickly verify pypi env from pypi yields working pydarshan.

- - - - -
df801f02 by Shane Snyder at 2021-01-28T08:14:50-06:00
Merge branch 'python-package' into 'master'

Python Package: Rename mode switch to dtype to select datatype. Simplify interface log_get_record. Have lustre records include OSTs. Add library version check.

See merge request darshan/darshan!62
- - - - -
08951349 by Kevin Harms at 2021-02-01T21:25:20+00:00
Merge python updates from master

- - - - -


30 changed files:

- darshan-util/pydarshan/.gitignore
- + darshan-util/pydarshan/CHANGELOG.rst
- darshan-util/pydarshan/Makefile
- darshan-util/pydarshan/README.rst
- + darshan-util/pydarshan/RELEASE-CHECKLIST-PyDarshan.txt
- darshan-util/pydarshan/darshan/__init__.py
- darshan-util/pydarshan/darshan/__main__.py
- darshan-util/pydarshan/darshan/backend/cffi_backend.py
- darshan-util/pydarshan/darshan/cli/__init__.py
- darshan-util/pydarshan/darshan/discover_darshan.py
- darshan-util/pydarshan/darshan/error.py
- darshan-util/pydarshan/darshan/experimental/aggregators/create_timeline.py
- darshan-util/pydarshan/darshan/experimental/aggregators/merge.py
- + darshan-util/pydarshan/darshan/experimental/transforms/dxt2png.py
- + darshan-util/pydarshan/darshan/extension.c
- darshan-util/pydarshan/darshan/report.py
- + darshan-util/pydarshan/devel/README
- + darshan-util/pydarshan/devel/build-all.sh
- + darshan-util/pydarshan/devel/build-wheels.sh
- + darshan-util/pydarshan/devel/travis.yml
- darshan-util/pydarshan/docs/Makefile
- darshan-util/pydarshan/docs/conf.py
- − darshan-util/pydarshan/docs/darshan.backend.rst
- − darshan-util/pydarshan/docs/darshan.rst
- darshan-util/pydarshan/docs/index.rst
- darshan-util/pydarshan/docs/installation.rst → darshan-util/pydarshan/docs/install.rst
- − darshan-util/pydarshan/docs/modules.rst
- darshan-util/pydarshan/docs/usage.rst
- darshan-util/pydarshan/examples/01_darshan-introduction.ipynb
- darshan-util/pydarshan/examples/02_darshan-plotting-matplotlib.ipynb


The diff was not included because it is too large.


View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/-/compare/4f566bcbfa5cac6011f77de91f6dc6441a13b4d0...089513491e922b421cc7d3a9a5c154f50ee75518

-- 
View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/-/compare/4f566bcbfa5cac6011f77de91f6dc6441a13b4d0...089513491e922b421cc7d3a9a5c154f50ee75518
You're receiving this email because of your account on xgitlab.cels.anl.gov.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-commits/attachments/20210201/ff9ec740/attachment-0001.html>

More information about the Darshan-commits mailing list