[Darshan-commits] [Git][darshan/darshan][python-cleanup] 5 commits: Remove unused files
Kevin Harms
xgitlab at cels.anl.gov
Thu Feb 25 09:29:00 CST 2021
Kevin Harms pushed to branch python-cleanup at darshan / darshan
Commits:
55b55716 by Kevin Harms at 2021-02-24T17:54:14+00:00
Remove unused files
- - - - -
6b814c44 by Kevin Harms at 2021-02-24T20:54:29+00:00
Remove current api backend generation code as not currently uesd
- - - - -
01c3b973 by Kevin Harms at 2021-02-24T22:22:30+00:00
Update makefile to supoprt only currently valid targets
- - - - -
0867bbd6 by Kevin Harms at 2021-02-24T22:24:05+00:00
Remove statement that is flagged as warning by pytest
- - - - -
bbd90463 by Kevin Harms at 2021-02-25T15:28:38+00:00
pydarshan documentation updates
- - - - -
27 changed files:
- − darshan-util/pydarshan/CHANGELOG.rst
- darshan-util/pydarshan/Makefile
- darshan-util/pydarshan/README.rst
- darshan-util/pydarshan/darshan/__init__.py
- darshan-util/pydarshan/darshan/api_def_c.py
- darshan-util/pydarshan/darshan/backend/__init__.py
- darshan-util/pydarshan/darshan/backend/cffi_backend.py
- darshan-util/pydarshan/darshan/cli/__init__.py
- darshan-util/pydarshan/darshan/cli/info.py
- darshan-util/pydarshan/darshan/cli/name_records.py
- darshan-util/pydarshan/darshan/cli/to_json.py
- − darshan-util/pydarshan/darshan/common.py
- − darshan-util/pydarshan/darshan/data/generate_headers/Makefile
- − darshan-util/pydarshan/darshan/data/generate_headers/apply-patch.sh
- − darshan-util/pydarshan/darshan/data/generate_headers/commented-out.h
- − darshan-util/pydarshan/darshan/data/generate_headers/create-patch-for-generated.sh
- − darshan-util/pydarshan/darshan/data/generate_headers/create-patch-for-include.sh
- − darshan-util/pydarshan/darshan/data/generate_headers/darshan-api.h
- − darshan-util/pydarshan/darshan/data/generate_headers/generate-header.sh
- − darshan-util/pydarshan/darshan/data/generate_headers/patch-cffi-incompatible.patch
- − darshan-util/pydarshan/darshan/data/generate_headers/patch-darshan-log-format.patch
- − darshan-util/pydarshan/darshan/data/generate_headers/patch-darshan-logutils.patch
- darshan-util/pydarshan/darshan/experimental/__init__.py
- darshan-util/pydarshan/darshan/experimental/aggregators/__init__.py
- darshan-util/pydarshan/docs/conf.py
- darshan-util/pydarshan/setup.cfg
- − darshan-util/pydarshan/spack/py-darshan/package.py
Changes:
=====================================
darshan-util/pydarshan/CHANGELOG.rst deleted
=====================================
=====================================
darshan-util/pydarshan/Makefile
=====================================
@@ -32,17 +32,6 @@ clean-test: # remove test and coverage artifacts
rm -rf .pytest_cache
rm -rf pkgtest
-clean-devenv:
- rm -rf devenv/venv
- rm -rf devenv/libdarshanutil
- rm -rf devenv
-
-
-devenv:
- ./devel/build-libdarshanutil.sh
- python3 -m venv devenv/venv
- source devenv/venv/bin/activate && pip install -r requirements_dev.txt
-
lint: # check style with flake8
flake8 darshan tests
@@ -56,7 +45,6 @@ coverage: # check code coverage quickly with the default Python
coverage run --source darshan -m pytest
coverage report -m
coverage html
- xdg-open htmlcov/index.html
docs: clean-docs # generate Sphinx HTML documentation, including API docs
@@ -67,14 +55,6 @@ docs: clean-docs # generate Sphinx HTML documentation, including API docs
docs-show:
xdg-open docs/build/html/index.html
-servedocs: docs # compile the docs watching for changes
- watchmedo shell-command -p '*.rst' -c '$(MAKE) -C docs html' -R -D .
-
-
-bump-minior:
-
-
-
wheels:
./devel/build-all.sh
=====================================
darshan-util/pydarshan/README.rst
=====================================
@@ -1,9 +1,9 @@
-=======================
-PyDarshan Documentation
-=======================
+========
+Overview
+========
Python utilities to interact with Darshan log records of HPC applications.
-pydarshan requires darshan-utils (3.2.2+) to be installed.
+pydarshan requires darshan-utils version 3.3 or higher to be installed.
Features
--------
@@ -46,7 +46,7 @@ Installation
To install in most cases the following will work::
- pip install darshan
+ pip install --user darshan
For alternative installation instructions and installation from source refer to <docs/install.rst>
@@ -72,7 +72,6 @@ Conformance to PEPs can be tested using flake8 via::
make lint
-
Documentation
-------------
@@ -83,3 +82,29 @@ documentation can be build using make as follows::
pip install -r requirements_dev.txt
make docs
+
+File List
+---------
+
+* darshan::
+ core darshan python module code
+* devel::
+ scripts for building python wheel
+* docs::
+ markdown documentation used by sphinx to auto-generate HTML RTD style doc
+* examples::
+ Jupyter notebooks showing pydarshan usage with log files
+* tests::
+ pydarshan specific test cases
+* requirements.txt::
+ pip requirement file for minimum set of depednencies
+* requirements_dev.txt::
+ pip requirement file for depednencies needed to run development tools
+* setup.py::
+ python file for building/generating pydarshan package
+* setup.cfg::
+ input for setup.py
+* MANIFEST.in::
+ input files for setup.py package
+* tox.ini::
+ input for tox which runs the automated testing
=====================================
darshan-util/pydarshan/darshan/__init__.py
=====================================
@@ -1,4 +1,9 @@
-__version__ = '0.0.6'
+"""
+PyDarshan provides direct log access for reading binary Darshan logs.
+PyDarshan also provides a suite of analysis utilities.
+"""
+
+__version__ = '3.3.0'
__darshanutil_version__ = '3.2.1'
import logging
=====================================
darshan-util/pydarshan/darshan/api_def_c.py
=====================================
@@ -2,6 +2,8 @@
"""
The api_def_c carries a copy of CFFI compatible headers for libdarshan-util.so.
+These definitions must match the structure definitions for the associated
+darshan release.
"""
=====================================
darshan-util/pydarshan/darshan/backend/__init__.py
=====================================
@@ -0,0 +1,6 @@
+"""The backend package provides implementions for reading
+the binary representation of the darshan log file.
+The only current implementation uses CFFI which
+uses the functions defined in libdarshan-util.so
+to read the log.
+"""
=====================================
darshan-util/pydarshan/darshan/backend/cffi_backend.py
=====================================
@@ -1,4 +1,8 @@
# -*- coding: utf-8 -*-
+"""The cfii_backend package will read a darshan log
+using the functions defined in libdarshan-util.so
+and is interfaced via the python CFFI module.
+"""
import cffi
import ctypes
=====================================
darshan-util/pydarshan/darshan/cli/__init__.py
=====================================
@@ -1,3 +1,7 @@
+"""The cli package provides a basis for building future python
+based command line utilities. Currently, the existing commands
+provide basic examples with limited functionality.
+"""
import argparse
import logging
from argparse import ArgumentDefaultsHelpFormatter
=====================================
darshan-util/pydarshan/darshan/cli/info.py
=====================================
@@ -1,3 +1,6 @@
+"""The `info` subcommand provides a basic dump of information about
+the darshan log equivalent to DarshanReport.info().
+"""
import sys
import argparse
import darshan
=====================================
darshan-util/pydarshan/darshan/cli/name_records.py
=====================================
@@ -1,3 +1,6 @@
+"""The `name_records` subcommand provides a list of all the file names
+list in the darshan log hash table.
+"""
import sys
import argparse
import darshan
=====================================
darshan-util/pydarshan/darshan/cli/to_json.py
=====================================
@@ -1,3 +1,5 @@
+"""The `to_json` subcommand dumps the darshan log to json format.
+"""
import sys
import argparse
import darshan
=====================================
darshan-util/pydarshan/darshan/common.py deleted
=====================================
@@ -1,5 +0,0 @@
-# -*- coding: utf-8 -*-
-
-"""
-The darshan.common module provides common functionality required by mulitple modules.
-"""
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/Makefile deleted
=====================================
@@ -1,20 +0,0 @@
-
-
-
-all:
- @echo "Consult Makefile for available targets."
-
-
-header:
- ./generate-header.sh
- #./apply-patch.sh
-
-
-
-clean:
- rm -rf include
- rm -f generated.h
- rm -f generated.h.org
-
- rm -f generated.h.orig
- rm -f generated.h.rej
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/apply-patch.sh deleted
=====================================
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-patch -b --verbose generated.h patch-cffi-incompatible.patch
-
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/commented-out.h deleted
=====================================
@@ -1,470 +0,0 @@
-# 1 "include/darshan-logutils.h"
-# 1 "<built-in>"
-# 1 "<command-line>"
-# 31 "<command-line>"
-# 1 "/usr/include/stdc-predef.h" 1 3 4
-# 32 "<command-line>" 2
-# 1 "include/darshan-logutils.h"
-# 15 "include/darshan-logutils.h"
-struct UT_hash_handle;
-
-# 1 "include/darshan-log-format.h" 1
-# 39 "include/darshan-log-format.h"
-enum darshan_comp_type
-{
- DARSHAN_ZLIB_COMP,
- DARSHAN_BZIP2_COMP,
- DARSHAN_NO_COMP,
-};
-
-typedef uint64_t darshan_record_id;
-
-
-
-
-
-
-struct darshan_log_map
-{
- uint64_t off;
- uint64_t len;
-};
-
-
-
-
-struct darshan_header
-{
- char version_string[8];
- int64_t magic_nr;
- unsigned char comp_type;
- uint32_t partial_flag;
- struct darshan_log_map name_map;
- struct darshan_log_map mod_map[16];
- uint32_t mod_ver[16];
-};
-
-
-
-struct darshan_job
-{
- int64_t uid;
- int64_t start_time;
- int64_t end_time;
- int64_t nprocs;
- int64_t jobid;
- char metadata[1024];
-};
-
-
-struct darshan_name_record
-{
- darshan_record_id id;
- char name[1];
-};
-
-
-struct darshan_base_record
-{
- darshan_record_id id;
- int64_t rank;
-};
-
-
-
-
-
-
-# 1 "include/darshan-null-log-format.h" 1
-# 31 "include/darshan-null-log-format.h"
-enum darshan_null_indices
-{
- NULL_FOOS, NULL_FOO_MAX_DAT, NULL_NUM_INDICES,
-};
-
-
-enum darshan_null_f_indices
-{
- NULL_F_FOO_TIMESTAMP, NULL_F_FOO_MAX_DURATION, NULL_F_NUM_INDICES,
-};
-# 51 "include/darshan-null-log-format.h"
-struct darshan_null_record
-{
- struct darshan_base_record base_rec;
- int64_t counters[NULL_NUM_INDICES];
- double fcounters[NULL_F_NUM_INDICES];
-};
-# 105 "include/darshan-log-format.h" 2
-# 1 "include/darshan-posix-log-format.h" 1
-# 148 "include/darshan-posix-log-format.h"
-enum darshan_posix_indices
-{
- POSIX_OPENS, POSIX_READS, POSIX_WRITES, POSIX_SEEKS, POSIX_STATS, POSIX_MMAPS, POSIX_FSYNCS, POSIX_FDSYNCS, POSIX_MODE, POSIX_BYTES_READ, POSIX_BYTES_WRITTEN, POSIX_MAX_BYTE_READ, POSIX_MAX_BYTE_WRITTEN, POSIX_CONSEC_READS, POSIX_CONSEC_WRITES, POSIX_SEQ_READS, POSIX_SEQ_WRITES, POSIX_RW_SWITCHES, POSIX_MEM_NOT_ALIGNED, POSIX_MEM_ALIGNMENT, POSIX_FILE_NOT_ALIGNED, POSIX_FILE_ALIGNMENT, POSIX_MAX_READ_TIME_SIZE, POSIX_MAX_WRITE_TIME_SIZE, POSIX_SIZE_READ_0_100, POSIX_SIZE_READ_100_1K, POSIX_SIZE_READ_1K_10K, POSIX_SIZE_READ_10K_100K, POSIX_SIZE_READ_100K_1M, POSIX_SIZE_READ_1M_4M, POSIX_SIZE_READ_4M_10M, POSIX_SIZE_READ_10M_100M, POSIX_SIZE_READ_100M_1G, POSIX_SIZE_READ_1G_PLUS, POSIX_SIZE_WRITE_0_100, POSIX_SIZE_WRITE_100_1K, POSIX_SIZE_WRITE_1K_10K, POSIX_SIZE_WRITE_10K_100K, POSIX_SIZE_WRITE_100K_1M, POSIX_SIZE_WRITE_1M_4M, POSIX_SIZE_WRITE_4M_10M, POSIX_SIZE_WRITE_10M_100M, POSIX_SIZE_WRITE_100M_1G, POSIX_SIZE_WRITE_1G_PLUS, POSIX_STRIDE1_STRIDE, POSIX_STRIDE2_STRIDE, POSIX_STRIDE3_STRIDE, POSIX_STRIDE4_STRIDE, POSIX_STRIDE1_COUNT, POSIX_STRIDE2_COUNT, POSIX_STRIDE3_COUNT, POSIX_STRIDE4_COUNT, POSIX_ACCESS1_ACCESS, POSIX_ACCESS2_ACCESS, POSIX_ACCESS3_ACCESS, POSIX_ACCESS4_ACCESS, POSIX_ACCESS1_COUNT, POSIX_ACCESS2_COUNT, POSIX_ACCESS3_COUNT, POSIX_ACCESS4_COUNT, POSIX_FASTEST_RANK, POSIX_FASTEST_RANK_BYTES, POSIX_SLOWEST_RANK, POSIX_SLOWEST_RANK_BYTES, POSIX_NUM_INDICES,
-};
-
-
-enum darshan_posix_f_indices
-{
- POSIX_F_OPEN_START_TIMESTAMP, POSIX_F_READ_START_TIMESTAMP, POSIX_F_WRITE_START_TIMESTAMP, POSIX_F_CLOSE_START_TIMESTAMP, POSIX_F_OPEN_END_TIMESTAMP, POSIX_F_READ_END_TIMESTAMP, POSIX_F_WRITE_END_TIMESTAMP, POSIX_F_CLOSE_END_TIMESTAMP, POSIX_F_READ_TIME, POSIX_F_WRITE_TIME, POSIX_F_META_TIME, POSIX_F_MAX_READ_TIME, POSIX_F_MAX_WRITE_TIME, POSIX_F_FASTEST_RANK_TIME, POSIX_F_SLOWEST_RANK_TIME, POSIX_F_VARIANCE_RANK_TIME, POSIX_F_VARIANCE_RANK_BYTES, POSIX_F_NUM_INDICES,
-};
-# 167 "include/darshan-posix-log-format.h"
-struct darshan_posix_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[POSIX_NUM_INDICES];
- double fcounters[POSIX_F_NUM_INDICES];
-};
-# 106 "include/darshan-log-format.h" 2
-# 1 "include/darshan-mpiio-log-format.h" 1
-# 128 "include/darshan-mpiio-log-format.h"
-enum darshan_mpiio_indices
-{
- MPIIO_INDEP_OPENS, MPIIO_COLL_OPENS, MPIIO_INDEP_READS, MPIIO_INDEP_WRITES, MPIIO_COLL_READS, MPIIO_COLL_WRITES, MPIIO_SPLIT_READS, MPIIO_SPLIT_WRITES, MPIIO_NB_READS, MPIIO_NB_WRITES, MPIIO_SYNCS, MPIIO_HINTS, MPIIO_VIEWS, MPIIO_MODE, MPIIO_BYTES_READ, MPIIO_BYTES_WRITTEN, MPIIO_RW_SWITCHES, MPIIO_MAX_READ_TIME_SIZE, MPIIO_MAX_WRITE_TIME_SIZE, MPIIO_SIZE_READ_AGG_0_100, MPIIO_SIZE_READ_AGG_100_1K, MPIIO_SIZE_READ_AGG_1K_10K, MPIIO_SIZE_READ_AGG_10K_100K, MPIIO_SIZE_READ_AGG_100K_1M, MPIIO_SIZE_READ_AGG_1M_4M, MPIIO_SIZE_READ_AGG_4M_10M, MPIIO_SIZE_READ_AGG_10M_100M, MPIIO_SIZE_READ_AGG_100M_1G, MPIIO_SIZE_READ_AGG_1G_PLUS, MPIIO_SIZE_WRITE_AGG_0_100, MPIIO_SIZE_WRITE_AGG_100_1K, MPIIO_SIZE_WRITE_AGG_1K_10K, MPIIO_SIZE_WRITE_AGG_10K_100K, MPIIO_SIZE_WRITE_AGG_100K_1M, MPIIO_SIZE_WRITE_AGG_1M_4M, MPIIO_SIZE_WRITE_AGG_4M_10M, MPIIO_SIZE_WRITE_AGG_10M_100M, MPIIO_SIZE_WRITE_AGG_100M_1G, MPIIO_SIZE_WRITE_AGG_1G_PLUS, MPIIO_ACCESS1_ACCESS, MPIIO_ACCESS2_ACCESS, MPIIO_ACCESS3_ACCESS, MPIIO_ACCESS4_ACCESS, MPIIO_ACCESS1_COUNT, MPIIO_ACCESS2_COUNT, MPIIO_ACCESS3_COUNT, MPIIO_ACCESS4_COUNT, MPIIO_FASTEST_RANK, MPIIO_FASTEST_RANK_BYTES, MPIIO_SLOWEST_RANK, MPIIO_SLOWEST_RANK_BYTES, MPIIO_NUM_INDICES,
-};
-
-
-enum darshan_mpiio_f_indices
-{
- MPIIO_F_OPEN_TIMESTAMP, MPIIO_F_READ_START_TIMESTAMP, MPIIO_F_WRITE_START_TIMESTAMP, MPIIO_F_READ_END_TIMESTAMP, MPIIO_F_WRITE_END_TIMESTAMP, MPIIO_F_CLOSE_TIMESTAMP, MPIIO_F_READ_TIME, MPIIO_F_WRITE_TIME, MPIIO_F_META_TIME, MPIIO_F_MAX_READ_TIME, MPIIO_F_MAX_WRITE_TIME, MPIIO_F_FASTEST_RANK_TIME, MPIIO_F_SLOWEST_RANK_TIME, MPIIO_F_VARIANCE_RANK_TIME, MPIIO_F_VARIANCE_RANK_BYTES, MPIIO_F_NUM_INDICES,
-};
-# 147 "include/darshan-mpiio-log-format.h"
-struct darshan_mpiio_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[MPIIO_NUM_INDICES];
- double fcounters[MPIIO_F_NUM_INDICES];
-};
-# 107 "include/darshan-log-format.h" 2
-# 1 "include/darshan-hdf5-log-format.h" 1
-# 29 "include/darshan-hdf5-log-format.h"
-enum darshan_hdf5_indices
-{
- HDF5_OPENS, HDF5_NUM_INDICES,
-};
-
-
-enum darshan_hdf5_f_indices
-{
- HDF5_F_OPEN_TIMESTAMP, HDF5_F_CLOSE_TIMESTAMP, HDF5_F_NUM_INDICES,
-};
-# 48 "include/darshan-hdf5-log-format.h"
-struct darshan_hdf5_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[HDF5_NUM_INDICES];
- double fcounters[HDF5_F_NUM_INDICES];
-};
-# 108 "include/darshan-log-format.h" 2
-# 1 "include/darshan-pnetcdf-log-format.h" 1
-# 31 "include/darshan-pnetcdf-log-format.h"
-enum darshan_pnetcdf_indices
-{
- PNETCDF_INDEP_OPENS, PNETCDF_COLL_OPENS, PNETCDF_NUM_INDICES,
-};
-
-
-enum darshan_pnetcdf_f_indices
-{
- PNETCDF_F_OPEN_TIMESTAMP, PNETCDF_F_CLOSE_TIMESTAMP, PNETCDF_F_NUM_INDICES,
-};
-# 50 "include/darshan-pnetcdf-log-format.h"
-struct darshan_pnetcdf_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[PNETCDF_NUM_INDICES];
- double fcounters[PNETCDF_F_NUM_INDICES];
-};
-# 109 "include/darshan-log-format.h" 2
-# 1 "include/darshan-bgq-log-format.h" 1
-# 47 "include/darshan-bgq-log-format.h"
-enum darshan_bgq_indices
-{
- BGQ_CSJOBID, BGQ_NNODES, BGQ_RANKSPERNODE, BGQ_DDRPERNODE, BGQ_INODES, BGQ_ANODES, BGQ_BNODES, BGQ_CNODES, BGQ_DNODES, BGQ_ENODES, BGQ_TORUSENABLED, BGQ_NUM_INDICES,
-};
-
-
-enum darshan_bgq_f_indices
-{
- BGQ_F_TIMESTAMP, BGQ_F_NUM_INDICES,
-};
-# 67 "include/darshan-bgq-log-format.h"
-struct darshan_bgq_record
-{
- struct darshan_base_record base_rec;
- int64_t counters[BGQ_NUM_INDICES];
- double fcounters[BGQ_F_NUM_INDICES];
-};
-# 110 "include/darshan-log-format.h" 2
-# 1 "include/darshan-lustre-log-format.h" 1
-# 13 "include/darshan-lustre-log-format.h"
-typedef int64_t OST_ID;
-# 34 "include/darshan-lustre-log-format.h"
-enum darshan_lustre_indices
-{
- LUSTRE_OSTS, LUSTRE_MDTS, LUSTRE_STRIPE_OFFSET, LUSTRE_STRIPE_SIZE, LUSTRE_STRIPE_WIDTH, LUSTRE_NUM_INDICES,
-};
-# 46 "include/darshan-lustre-log-format.h"
-struct darshan_lustre_record
-{
- struct darshan_base_record base_rec;
- int64_t counters[LUSTRE_NUM_INDICES];
- OST_ID ost_ids[1];
-};
-# 111 "include/darshan-log-format.h" 2
-# 1 "include/darshan-stdio-log-format.h" 1
-# 75 "include/darshan-stdio-log-format.h"
-enum darshan_stdio_indices
-{
- STDIO_OPENS, STDIO_READS, STDIO_WRITES, STDIO_SEEKS, STDIO_FLUSHES, STDIO_BYTES_WRITTEN, STDIO_BYTES_READ, STDIO_MAX_BYTE_READ, STDIO_MAX_BYTE_WRITTEN, STDIO_FASTEST_RANK, STDIO_FASTEST_RANK_BYTES, STDIO_SLOWEST_RANK, STDIO_SLOWEST_RANK_BYTES, STDIO_NUM_INDICES,
-};
-
-
-enum darshan_stdio_f_indices
-{
- STDIO_F_META_TIME, STDIO_F_WRITE_TIME, STDIO_F_READ_TIME, STDIO_F_OPEN_START_TIMESTAMP, STDIO_F_CLOSE_START_TIMESTAMP, STDIO_F_WRITE_START_TIMESTAMP, STDIO_F_READ_START_TIMESTAMP, STDIO_F_OPEN_END_TIMESTAMP, STDIO_F_CLOSE_END_TIMESTAMP, STDIO_F_WRITE_END_TIMESTAMP, STDIO_F_READ_END_TIMESTAMP, STDIO_F_FASTEST_RANK_TIME, STDIO_F_SLOWEST_RANK_TIME, STDIO_F_VARIANCE_RANK_TIME, STDIO_F_VARIANCE_RANK_BYTES, STDIO_F_NUM_INDICES,
-};
-# 95 "include/darshan-stdio-log-format.h"
-struct darshan_stdio_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[STDIO_NUM_INDICES];
- double fcounters[STDIO_F_NUM_INDICES];
-};
-# 112 "include/darshan-log-format.h" 2
-
-# 1 "include/darshan-dxt-log-format.h" 1
-# 19 "include/darshan-dxt-log-format.h"
-typedef struct segment_info {
- int64_t offset;
- int64_t length;
- double start_time;
- double end_time;
-} segment_info;
-# 36 "include/darshan-dxt-log-format.h"
-struct dxt_file_record {
- struct darshan_base_record base_rec;
- int64_t shared_record;
- char hostname[64];
-
- int64_t write_count;
- int64_t read_count;
-};
-# 114 "include/darshan-log-format.h" 2
-# 143 "include/darshan-log-format.h"
-typedef enum
-{
- DARSHAN_NULL_MOD, DARSHAN_POSIX_MOD, DARSHAN_MPIIO_MOD, DARSHAN_HDF5_MOD, DARSHAN_PNETCDF_MOD, DARSHAN_BGQ_MOD, DARSHAN_LUSTRE_MOD, DARSHAN_STDIO_MOD, DXT_POSIX_MOD, DXT_MPIIO_MOD,
-} darshan_module_id;
-
-
-
-
-static char * const darshan_module_names[] =
-{
- "NULL", "POSIX", "MPI-IO", "HDF5", "PNETCDF", "BG/Q", "LUSTRE", "STDIO", "DXT_POSIX", "DXT_MPIIO",
-};
-
-
-
-
-static const int darshan_module_versions[] =
-{
- 1, 3, 2, 1, 1, 2, 1, 1, 1, 1,
-};
-# 18 "include/darshan-logutils.h" 2
-
-
-
-
-
-
-struct darshan_fd_int_state;
-
-
-struct darshan_fd_s
-{
-
- char version[8];
-
-
-
- int swap_flag;
-
- int partial_flag;
-
- enum darshan_comp_type comp_type;
-
- struct darshan_log_map job_map;
- struct darshan_log_map name_map;
- struct darshan_log_map mod_map[16];
-
- uint32_t mod_ver[16];
-
-
- struct darshan_fd_int_state *state;
-};
-
-
-typedef struct darshan_fd_s* darshan_fd;
-
-/*
-struct darshan_name_record_ref
-{
- struct darshan_name_record *name_record;
- UT_hash_handle hlink;
-};
-
-struct lustre_record_ref
-{
- struct darshan_lustre_record *rec;
- UT_hash_handle hlink;
-};
-*/
-
-/*
-struct darshan_mnt_info
-{
- char mnt_type[(4096 - sizeof(struct darshan_job) - 1)];
- char mnt_path[(4096 - sizeof(struct darshan_job) - 1)];
-};
-*/
-
-
-
-struct darshan_mod_logutil_funcs
-{
-
-
-
-
-
-
-
- int (*log_get_record)(
- darshan_fd fd,
- void** buf
- );
-
-
-
-
-
- int (*log_put_record)(
- darshan_fd fd,
- void *buf
- );
-
-
-
-
-
-
- void (*log_print_record)(
- void *rec,
- char *file_name,
- char *mnt_pt,
- char *fs_type
- );
-
-
-
- void (*log_print_description)(
- int ver);
-
- void (*log_print_diff)(
- void *rec1,
- char *name1,
- void *rec2,
- char *name2
- );
-
- void (*log_agg_records)(
- void *rec,
- void *agg_rec,
- int init_flag
- );
-};
-
-extern struct darshan_mod_logutil_funcs *mod_logutils[];
-
-# 1 "include/darshan-posix-logutils.h" 1
-# 10 "include/darshan-posix-logutils.h"
-extern char *posix_counter_names[];
-extern char *posix_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs posix_logutils;
-# 130 "include/darshan-logutils.h" 2
-# 1 "include/darshan-mpiio-logutils.h" 1
-# 10 "include/darshan-mpiio-logutils.h"
-extern char *mpiio_counter_names[];
-extern char *mpiio_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs mpiio_logutils;
-# 131 "include/darshan-logutils.h" 2
-# 1 "include/darshan-hdf5-logutils.h" 1
-# 10 "include/darshan-hdf5-logutils.h"
-extern char *hdf5_counter_names[];
-extern char *hdf5_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs hdf5_logutils;
-# 132 "include/darshan-logutils.h" 2
-# 1 "include/darshan-pnetcdf-logutils.h" 1
-# 10 "include/darshan-pnetcdf-logutils.h"
-extern char *pnetcdf_counter_names[];
-extern char *pnetcdf_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs pnetcdf_logutils;
-# 133 "include/darshan-logutils.h" 2
-# 1 "include/darshan-bgq-logutils.h" 1
-# 10 "include/darshan-bgq-logutils.h"
-extern char *bgq_counter_names[];
-extern char *bgq_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs bgq_logutils;
-# 134 "include/darshan-logutils.h" 2
-# 1 "include/darshan-lustre-logutils.h" 1
-# 10 "include/darshan-lustre-logutils.h"
-extern char *lustre_counter_names[];
-
-extern struct darshan_mod_logutil_funcs lustre_logutils;
-# 135 "include/darshan-logutils.h" 2
-# 1 "include/darshan-stdio-logutils.h" 1
-# 13 "include/darshan-stdio-logutils.h"
-extern char *stdio_counter_names[];
-extern char *stdio_f_counter_names[];
-
-extern struct darshan_mod_logutil_funcs stdio_logutils;
-# 136 "include/darshan-logutils.h" 2
-
-
-# 1 "include/darshan-dxt-logutils.h" 1
-# 11 "include/darshan-dxt-logutils.h"
-extern struct darshan_mod_logutil_funcs dxt_posix_logutils;
-extern struct darshan_mod_logutil_funcs dxt_mpiio_logutils;
-
-void dxt_log_print_posix_file(void *file_rec, char *file_name,
- char *mnt_pt, char *fs_type, struct lustre_record_ref *rec_ref);
-void dxt_log_print_mpiio_file(void *file_rec,
- char *file_name, char *mnt_pt, char *fs_type);
-# 139 "include/darshan-logutils.h" 2
-
-darshan_fd darshan_log_open(const char *name);
-darshan_fd darshan_log_create(const char *name, enum darshan_comp_type comp_type,
- int partial_flag);
-int darshan_log_get_job(darshan_fd fd, struct darshan_job *job);
-int darshan_log_put_job(darshan_fd fd, struct darshan_job *job);
-int darshan_log_get_exe(darshan_fd fd, char *buf);
-int darshan_log_put_exe(darshan_fd fd, char *buf);
-int darshan_log_get_mounts(darshan_fd fd, struct darshan_mnt_info **mnt_data_array,
- int* count);
-int darshan_log_put_mounts(darshan_fd fd, struct darshan_mnt_info *mnt_data_array,
- int count);
-int darshan_log_get_namehash(darshan_fd fd, struct darshan_name_record_ref **hash);
-int darshan_log_put_namehash(darshan_fd fd, struct darshan_name_record_ref *hash);
-int darshan_log_get_mod(darshan_fd fd, darshan_module_id mod_id,
- void *mod_buf, int mod_buf_sz);
-int darshan_log_put_mod(darshan_fd fd, darshan_module_id mod_id,
- void *mod_buf, int mod_buf_sz, int ver);
-void darshan_log_close(darshan_fd file);
-void darshan_log_print_version_warnings(const char *version_string);
-
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/create-patch-for-generated.sh deleted
=====================================
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-diff -cB generated.h.org generated.h > patch-cffi-incompatible.patch
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/create-patch-for-include.sh deleted
=====================================
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-diff -cB include.org/darshan-logutils.h include/darshan-logutils.h > patch-darshan-logutils.patch
-diff -cB include.org/darshan-log-format.h include/darshan-log-format.h > patch-darshan-log-format.patch
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/darshan-api.h deleted
=====================================
@@ -1,113 +0,0 @@
-/* from darshan-logutils.h */
-struct darshan_mnt_info
-{
- char mnt_type[3031];
- char mnt_path[3031];
-};
-
-struct darshan_mod_info
-{
- char *name;
- int len;
- int ver;
- int idx;
-};
-
-/* from darshan-log-format.h */
-struct darshan_job
-{
- int64_t uid;
- int64_t start_time;
- int64_t end_time;
- int64_t nprocs;
- int64_t jobid;
- char metadata[1024];
-};
-struct darshan_base_record
-{
- uint64_t id;
- int64_t rank;
-};
-
-struct darshan_posix_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[64];
- double fcounters[17];
-};
-
-struct darshan_stdio_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[13];
- double fcounters[15];
-};
-
-struct darshan_mpiio_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[51];
- double fcounters[15];
-};
-
-struct darshan_hdf5_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[1];
- double fcounters[2];
-};
-
-struct darshan_pnetcdf_file
-{
- struct darshan_base_record base_rec;
- int64_t counters[2];
- double fcounters[2];
-};
-
-struct darshan_bgq_record
-{
- struct darshan_base_record base_rec;
- int64_t counters[11];
- double fcounters[1];
-};
-
-/* from darshan-apxc-log-format.h */
-struct darshan_apxc_header_record
-{
- struct darshan_base_record base_rec;
- int64_t magic;
- int nblades;
- int nchassis;
- int ngroups;
- int memory_mode;
- int cluster_mode;
-};
-struct darshan_apxc_perf_record
-{
- struct darshan_base_record base_rec;
- int64_t counters[396];
-};
-
-/* counter names */
-char *apxc_counter_names[];
-char *bgq_counter_names[];
-char *bgq_f_counter_names[];
-char *hdf5_counter_names[];
-char *hdf5_f_counter_names[];
-char *mpiio_counter_names[];
-char *mpiio_f_counter_names[];
-char *pnetcdf_counter_names[];
-char *pnetcdf_f_counter_names[];
-char *posix_counter_names[];
-char *posix_f_counter_names[];
-char *stdio_counter_names[];
-char *stdio_f_counter_names[];
-
-/* Supported Functions */
-void* darshan_log_open(char *);
-int darshan_log_get_job(void *, struct darshan_job *);
-void darshan_log_close(void*);
-int darshan_log_get_exe(void*, char *);
-int darshan_log_get_mounts(void*, struct darshan_mnt_info **, int*);
-void darshan_log_get_modules(void*, struct darshan_mod_info **, int*);
-int darshan_log_get_record(void*, int, void **);
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/generate-header.sh deleted
=====================================
@@ -1,23 +0,0 @@
-#!/bin/bash
-
-# Generates a CFFI compatible header from an existing darshan-utils installations.
-# Note: The python bindings utilize a number of newer functions in darshan-util.
-
-
-# discover darshan
-darshan_install=$(cd $(dirname $(which darshan-parser)); cd ..; pwd -L)
-
-
-cp -R $darshan_install/include include
-# TODO: removing all non darshan header inlcudes works farily well except for structures depending on UT_hash
-
-
-# Remove unecessary includes to keep resulting header small
-patch -b --verbose include/darshan-logutils.h patch-darshan-logutils.patch
-patch -b --verbose include/darshan-log-format.h patch-darshan-log-format.patch
-
-# Execute preprocessor to expand macros etc.
-gcc -E include/darshan-logutils.h > generated.h
-
-# Make header CFFI compatible by patching runtime-dependent structures (e.g., sizeof)
-patch -b --verbose generated.h patch-cffi-incompatible.patch
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/patch-cffi-incompatible.patch deleted
=====================================
@@ -1,38 +0,0 @@
-*** generated.h.org 2018-06-30 18:16:31.104359748 -0400
---- generated.h 2018-06-30 18:18:46.285180237 -0400
-***************
-*** 305,310 ****
---- 305,311 ----
- };
- typedef struct darshan_fd_s* darshan_fd;
-
-+ /*
- struct darshan_name_record_ref
- {
- struct darshan_name_record *name_record;
-***************
-*** 317,327 ****
- struct darshan_lustre_record *rec;
- UT_hash_handle hlink;
- };
-
- struct darshan_mnt_info
- {
-! char mnt_type[(4096 - sizeof(struct darshan_job) - 1)];
-! char mnt_path[(4096 - sizeof(struct darshan_job) - 1)];
- };
-
-
---- 318,329 ----
- struct darshan_lustre_record *rec;
- UT_hash_handle hlink;
- };
-+ */
-
- struct darshan_mnt_info
- {
-! char mnt_type[3031];
-! char mnt_path[3031];
- };
-
-
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/patch-darshan-log-format.patch deleted
=====================================
@@ -1,22 +0,0 @@
-*** include.org/darshan-log-format.h 2018-06-30 18:43:37.815199355 -0400
---- include/darshan-log-format.h 2018-06-30 18:51:50.942544439 -0400
-***************
-*** 7,23 ****
- #ifndef __DARSHAN_LOG_FORMAT_H
- #define __DARSHAN_LOG_FORMAT_H
-
-- #include <unistd.h>
-- #include <sys/types.h>
-- #include <stdint.h>
-- #include <inttypes.h>
--
-- #if !defined PRId64
-- #error failed to detect PRId64
-- #endif
-- #if !defined PRIu64
-- #error failed to detect PRIu64
-- #endif
-
- /* update this on file format changes */
- /* NOTE: this should be updated when general changes are made to the
---- 7,12 ----
=====================================
darshan-util/pydarshan/darshan/data/generate_headers/patch-darshan-logutils.patch deleted
=====================================
@@ -1,18 +0,0 @@
-*** include.org/darshan-logutils.h 2018-06-30 18:43:37.811199360 -0400
---- include/darshan-logutils.h 2018-06-30 18:53:02.536449338 -0400
-***************
-*** 7,19 ****
- #ifndef __DARSHAN_LOG_UTILS_H
- #define __DARSHAN_LOG_UTILS_H
-
-- #include <limits.h>
-- #include <zlib.h>
-- #ifdef HAVE_LIBBZ2
-- #include <bzlib.h>
-- #endif
--
-- #include "uthash-1.9.2/src/uthash.h"
-
- #include "darshan-log-format.h"
-
---- 7,12 ----
=====================================
darshan-util/pydarshan/darshan/experimental/__init__.py
=====================================
@@ -0,0 +1,4 @@
+"""The experimental package provides some analysis functions for
+aggregating darshan log data. The functions are not well
+tested.
+"""
=====================================
darshan-util/pydarshan/darshan/experimental/aggregators/__init__.py
=====================================
@@ -0,0 +1,12 @@
+"""Aggregator functions are monkey patched into the DarshanReport object during
+initialization of DarshanReport class. These functions are only enabled if the
+`darshan.enable_experimental(True)` function is called.
+
+Example::
+ import darshan
+ import darshan.report
+ dasrhan.enable_experimental(True)
+ report = darshan.report.DarshanReport()
+ ...
+ result = report.agg_ioops()
+"""
=====================================
darshan-util/pydarshan/docs/conf.py
=====================================
@@ -63,8 +63,8 @@ source_suffix = '.rst'
master_doc = 'index'
# General information about the project.
-project = u'Darshan'
-copyright = u"2019, ANL"
+project = u'PyDarshan'
+copyright = u"2021, Argonne National Laboratory"
author = u""
# The version info for the project you're documenting, acts as replacement
=====================================
darshan-util/pydarshan/setup.cfg
=====================================
@@ -22,6 +22,5 @@ ignore = E501 E231 E265 E303 E271 E272
test = pytest
[tool:pytest]
-collect_ignore = ['setup.py']
testpaths = tests
=====================================
darshan-util/pydarshan/spack/py-darshan/package.py deleted
=====================================
@@ -1,20 +0,0 @@
-# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details..
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-from spack import *
-
-
-class PyDarshan(PythonPackage):
- """Python wrapper -- and more -- for Aaron Quinlan's BEDTools"""
-
- homepage = "http://www.mcs.anl.gov/research/projects/darshan/"
- url = "https://pypi.io/packages/source/p/darshan/darshan-0.0.1.tar.gz"
-
- version('0.0.1', sha256='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
-
- depends_on('py-cffi', type=('build', 'run'))
- depends_on('py-numpy', type=('build', 'run'))
- depends_on('py-pandas', type=('build', 'run'))
- depends_on('py-setuptools', type='build')
View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/-/compare/649e3d35eeac81ae6273add1951aa338cf20334f...bbd9046301950275305b7e1b99629b66f8495295
--
View it on GitLab: https://xgitlab.cels.anl.gov/darshan/darshan/-/compare/649e3d35eeac81ae6273add1951aa338cf20334f...bbd9046301950275305b7e1b99629b66f8495295
You're receiving this email because of your account on xgitlab.cels.anl.gov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-commits/attachments/20210225/e511ee0a/attachment-0001.html>
More information about the Darshan-commits
mailing list