Note
Reproducible metric tooling. The exact container image behind the published results is on the GitHub Container Registry (see Setup).
Empirical mining study measuring static code-quality metrics before vs after the first detectable adoption of an AI coding assistant in open-source repositories, using a quasi-experimental Interrupted Time Series + Difference-in-Differences design. Final cohort: 272 treated / 299 control (Python + TypeScript).
Research questions: RQ1 average cyclomatic complexity, RQ2 bug-fix commit ratio, RQ3 variation by language / activity. Robust effects on other metrics are reported as exploratory.
Methods reference generated per run:
data/out/METHODS.md(+PAPER_NOTES.mdfor the paper writer).
flowchart TD
SRC["① Discover<br/>S1 markers · S2 trailers · seed list · snowball (dep/fork/contrib)"]
SRC --> POOL["repos.parquet<br/>candidate pool, tagged discovery_origin"]
POOL --> FILT["② Filter (API-only)<br/>stars · created before 2024 · Python/TypeScript · not archived"]
FILT --> DET{"③ Detect-all<br/>blobless clone + full S1/S2/S3 signal<br/>(no S1-only pre-screen)"}
DET -->|"valid S1/S2"| TRE["Treated<br/>measured at the true treatment date"]
DET -->|"valid S3 only"| S3SENS["S3 sensitivity<br/>(measured, not in headline)"]
DET -->|"no in-window signal"| CON["Control<br/>synthetic cutoff = median treated date"]
DET -.->|"400 MB to 10 GB"| BIG["classified only<br/>(too big to measure, excluded from N)"]
TRE --> MEAS["④ Measure (full clone, ≤ 400 MB)<br/>4 snapshots T-6/T-1/T+1/T+6 + shifted -30/-60/-90 d<br/>containerized cloc · ruff · eslint · lizard · jscpd<br/>(read-only mount, --network=none, fixed configs)"]
CON --> MEAS
MEAS --> STAT["⑤ Stats<br/>Wilcoxon · Cliff's δ + bootstrap CI · Benjamini-Hochberg<br/>2×2 and covariate-adjusted DiD · Mann-Whitney strata (RQ3)<br/>sensitivities: S1-only · S3 · shifted"]
STAT --> REP["⑥ Report → data/out/<br/>LaTeX tables · figures · RESULTS / METHODS / PAPER_NOTES<br/>validation.tex · config.used.yaml · results.pdf"]
Requirements: Python ≥ 3.11, uv, and a
container runtime (Podman or Docker) for the pinned metric-tool image. Redis
is optional; the permanent on-disk store (data/cache/) is the source of truth.
uv sync
cp .env.example .env # add GITHUB_TOKEN
# Metric-tool image: pull the exact image used for the study,
# or build it from docker/. Either way the local tag must be mining-tools:3.
podman pull ghcr.io/lukaswestholt/mining-tools@sha256:d92700d6511bf9c7060ec3b2c787f889e8fd4c324cc19fe7930a1e863eb87464
podman tag ghcr.io/lukaswestholt/mining-tools@sha256:d92700d6511bf9c7060ec3b2c787f889e8fd4c324cc19fe7930a1e863eb87464 mining-tools:3
# or build from source: podman build -t mining-tools:3 docker/ (cloc/ruff/lizard/eslint/jscpd).env:
| Variable | Description |
|---|---|
GITHUB_TOKEN |
GitHub PAT with repo scope |
GITHUB_CACHE_ENABLED |
false to skip Redis and use the disk cache only |
All study tunables (observation window, inclusion gates, size caps, image
tag, snowball quota, shifted-treatment days, stats params, seed) live in
config.yaml, the single source of truth, snapshotted to
data/out/config.used.yaml on every run.
# full production pipeline: detect-all → route → measure → stats → report
GITHUB_CACHE_ENABLED=false uv run python run_production.pyIt operates on the frozen data/tables/repos.parquet and is resumable:
signal detection streams to data/tables/detect.ndjson, per-repo measurement to
treated.ndjson / control.ndjson, and the permanent measurement cache (keyed
by (repo_id, commit_hash, tool, image_tag)) skips any commit already measured,
so re-runs do almost no cloning or container work.
- Discover. Marker/path/commit search + manual seed list + pip/fork/
contributor snowball →
repos.parquet(tagged withdiscovery_origin). - Filter. Cheap API-only inclusion (stars, created-before, language,
archived); excluded rows kept with
exclusion_reason. - Detect-all. A blobless clone + full S1/S2/S3 signal detection on every candidate decides treated-vs-control by the repo's real signal (no cheap S1-only pre-screen). Two size caps: measured only if ≤ 400 MB; detection runs up to 10 GB, so larger repos are still classified but excluded from N.
- Measure. Treated at their true treatment date, controls at the
per-language synthetic cutoff; four snapshots (T-6/T-1/T+1/T+6) + shifted
anchors (-30/-60/-90 d), full clone + git-worktree per snapshot, containerized
tools (read-only mount,
--network=none, fixed configs). - Stats. Paired Wilcoxon, Cliff's δ + bootstrap CI, Benjamini-Hochberg, 2×2 + covariate-adjusted DiD, Mann-Whitney strata (RQ3); S1-only / S3 / shifted sensitivities.
- Report. The
data/out/bundle.
- Tables (
\input-able):results_main.tex(per-metric n),results_strata.tex,flow.tex,descriptives.tex,counts.tex,sensitivity_s1/s3/shifted.tex,validation.tex. - Figures (PDF + sibling
.json):fig_trajectories/forest/did/repo_age/creation_scatter. - Docs:
RESULTS.md,METHODS.md(config-derived),PAPER_NOTES.md,SENSITIVITY_*.md,SHIFTED_TREATMENT.md, compiledresults.pdf,config.used.yaml. - Dataset:
data/tables/{repos,snapshots,stats,stats_s1,stats_s3,stats_shift*}.{parquet,csv}.
uv run ai-impact validate-bugfix --sample 200 # write blind two-rater CSVs
# hand-label validation_rater_A.csv + _B.csv (see data/out/LABELING_GUIDE.md)
uv run ai-impact validate-bugfix --adjudicate # dump disagreements to resolve
uv run ai-impact validate-bugfix --score # κ + precision/recall/F1 → validation.texuv run ai-impact <cmd>: discover, filter, gate-report (size/gate
boundary report). Legacy local-analysis helpers (metrics, analyze-commits,
find-regression) operate on a single local checkout and are not on the study
path.
uv run pytest # 164 tests
uv run pre-commit run --all-files # ruff lint (select=ALL @ 80) + ruff-formatParts of the analysis pipeline were written with the assistance of Claude Code and reviewed by the author.