Skip to content
This repository was archived by the owner on Jul 17, 2026. It is now read-only.

Commit cc74402

Browse files
nick-furrclaude
andcommitted
docs: complete the maintenance-mode snapshot (build log, ADR index, resume map)
Final documentation pass so the repo is the source of truth before parking the project: log today session in notes/session-log.md; note in the README that active.md is v0.10 defense-in-depth while v0.9 stays the pinned baseline; link the parse/vision-hybrid and tiling design specs + plans from the ADR index so docs/decisions is a complete review entry point; add the substrate-text false-positive class and on-drawing highlighting (coords already captured) to STATE.md so the resume map is complete. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 3a2745b commit cc74402

4 files changed

Lines changed: 43 additions & 3 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,7 @@ The earlier 5/24 figure of 0.811 was measured against a non-deterministic stack
213213

214214
**Tiling (now the production raster path).** For sheets where source resolution exceeds Anthropic's ~1568 px server-side resize cap for Sonnet 4.6, splitting the PDF into ≤1568 px-long-edge tiles and merging per-tile extractions cracks the model-resolution ceiling. **case_006** (real hand-drawn bathroom elevation, stuck at recall ≤0.538 for two weeks across every prompt iteration) lifted to recall=0.538 + precision=0.636 (vs the pre-tiling baseline 0.231 / 0.273) — biggest single-case recall gain in the project to date. Tiling now ships as the production Vision path for raster/scanned sources, with `markup-postprocess.js` filtering the worst false positives the merge introduces. The remaining refinement is **conditional tiling** — tile only when a sheet's resolution actually demands it, rather than every raster page, since indiscriminate tiling still inflates extracted count on clean sheets.
215215

216-
Prompt versions are tracked in [`prompts/CHANGELOG.md`](prompts/CHANGELOG.md). The active prompt is [`prompts/active.md`](prompts/active.md), loaded by `extraction-service.js` at startup. Judgment uses Claude Haiku at `temperature: 0` (single deterministic call per pair; the prior 3x majority-vote pattern was a band-aid for the temperature default and is now removed), matching by conceptual equivalence rather than literal text. Case naming and the substrate × markup realism taxonomy are documented in [`evals/CONVENTIONS.md`](evals/CONVENTIONS.md).
216+
Prompt versions are tracked in [`prompts/CHANGELOG.md`](prompts/CHANGELOG.md). The active prompt is [`prompts/active.md`](prompts/active.md), loaded by `extraction-service.js` at startup. (`active.md` is **v0.10** — a defense-in-depth "never fabricate" clause layered on v0.9. It is *not* the confabulation fix: tiling addressed the root cause, illegibility; the prose guard alone failed its objective, see the CHANGELOG v0.10 entry. **v0.9 remains the pinned aggregate baseline** because v0.10's prompt-only delta is unattributable at the aggregate.) Judgment uses Claude Haiku at `temperature: 0` (single deterministic call per pair; the prior 3x majority-vote pattern was a band-aid for the temperature default and is now removed), matching by conceptual equivalence rather than literal text. Case naming and the substrate × markup realism taxonomy are documented in [`evals/CONVENTIONS.md`](evals/CONVENTIONS.md).
217217

218218
Determinism characterization: aggregate σ ≈ 0.003 across runs. Per-case has soft variance (σ ≈ 0.05) on borderline judgments due to Anthropic-side residual non-determinism at temp=0. Rule of thumb: aggregate moves >0.02 are signal; per-case moves >0.1 on borderline cases are signal. See `prompts/CHANGELOG.md` "v0.9 (rebaselined)" entry for full numbers and ADR 0003 for the architectural decision.
219219

STATE.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@ matches on conceptual equivalence. Prompt versions are tracked in
3535
Indiscriminate tiling triples extracted count and hurts precision on clean sheets;
3636
`markup-postprocess.js` filters the worst of it. The open work is gating tiling on actual
3737
sheet resolution rather than tiling all raster.
38+
- **Substrate-text false positives are the open precision class.** On the tiled vision
39+
path, the model transcribes the drawing's own printed black text as if it were redlines.
40+
Post-processing removed the location-only-cloud and cross-tile-dup FPs, but ~8/13 of
41+
case_012's FPs are this substrate-text class — can't be deduped away. The fix is a prompt
42+
rule ("extract only the colored annotation layer, never the substrate") and, for digital
43+
PDFs, the parse path (which sidesteps it entirely by reading the annotation objects).
3844
- **Real hand-drawn / photographed sheets are the hard ceiling.** Clean digital markups
3945
score well; photographed reviewer markup is where recall falls off.
4046
- **Clarification loop is one-directional** — ambiguous items auto-flag, but there's no
@@ -55,8 +61,11 @@ matches on conceptual equivalence. Prompt versions are tracked in
5561
`src/utils/pdf-annotation-probe.js`, `src/services/parse-extraction-service.js`).
5662
Phase 2 is the `digital_flattened` regime — digital PDFs whose markups were flattened
5763
into the page and so carry no annotation layer to parse.
58-
4. **Clarification reply workflow** — close the loop on auto-flagged ambiguous markups.
59-
5. **Formatted PDF report export** — for clean drafter handoff.
64+
4. **On-drawing highlighting** — the parse path already captures `{ page, rect, subtype }`
65+
coordinates per markup (PDF points, bottom-left origin); the UI to overlay them on the
66+
rendered drawing is not built. Lowest-risk visual win, data is already there.
67+
5. **Clarification reply workflow** — close the loop on auto-flagged ambiguous markups.
68+
6. **Formatted PDF report export** — for clean drafter handoff.
6069

6170
## Pointers
6271

docs/decisions/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,14 @@ Decisions are recorded chronologically. Each entry follows [Michael Nygard's ADR
77
| [0001](0001-async-sse-extraction.md) | Async SSE for extraction | Accepted | Background job + SSE progress stream so multi-page extractions survive load-balancer timeouts. |
88
| [0002](0002-sqlite-persistence.md) | SQLite via better-sqlite3 | Accepted | Embedded SQL database with row-level updates; replaces flat-file JSON that lost concurrent writes. |
99
| [0003](0003-pin-model-versions.md) | Make eval deterministic — pin temperatures and follow per-generation model-pinning convention | Accepted | Pin `temperature: 0` on both extraction and judge API calls; drop the now-redundant 3x judge vote; for 4.6+ generation use dateless form (it IS the pin per Anthropic docs), for 4.5 and earlier use dated form. |
10+
11+
## Design specs & plans
12+
13+
The two largest architectural changes were captured as full design specs + implementation plans rather than short ADRs. They're the primary reference for *why* the extraction pipeline is shaped the way it is:
14+
15+
| Decision | Spec | Plan |
16+
|---|---|---|
17+
| **Parse/vision hybrid routing** — probe the annotation layer; parse digital PDFs losslessly, reserve tiled vision for raster/scanned | [`specs/2026-06-02-parse-vision-hybrid-extraction-design.md`](../superpowers/specs/2026-06-02-parse-vision-hybrid-extraction-design.md) | [`plans/2026-06-02-parse-vision-hybrid-phase1.md`](../superpowers/plans/2026-06-02-parse-vision-hybrid-phase1.md) |
18+
| **Tiling + precision post-processing** — split large sheets into ≤1568px tiles to beat the model resize ceiling; deterministic post-pass to claw back the precision cost | [`specs/2026-05-29-tiling-precision-postprocessing-design.md`](../superpowers/specs/2026-05-29-tiling-precision-postprocessing-design.md) | [`plans/2026-05-29-tiling-precision-postprocessing.md`](../superpowers/plans/2026-05-29-tiling-precision-postprocessing.md) |
19+
20+
Prompt-level decisions (two-pass extraction, the v0.10 anti-confabulation guard, the temperature-pinning rebaseline) live in [`prompts/CHANGELOG.md`](../../prompts/CHANGELOG.md). The full build narrative with numbers and dead ends is in [`notes/session-log.md`](../../notes/session-log.md).

notes/session-log.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,26 @@ This file is **facts only** — no post framing, hooks, or voice notes. Build-in
66

77
---
88

9+
## 2026-06-05 — froze to maintenance mode + doc-accuracy pass
10+
11+
**Summary:** Parked RedlineIQ as a public portfolio piece. No feature work — got the repo to a clean, presentable, resumable snapshot: maintenance-mode README banner, a `STATE.md` resume map, dead-code removal, a security patch, and a documentation-accuracy pass that brought the README/STATE in line with the shipped hybrid pipeline. Tagged `v0.9-portfolio`.
12+
13+
**What happened:**
14+
1. **Freeze + resume map:** added a "portfolio / maintenance mode" banner to the README and wrote `STATE.md` (current standing, known limitations, top follow-ups to resume on). Annotated tag `v0.9-portfolio` marks the snapshot.
15+
2. **Dead-code cleanup:** removed 7 orphaned root-level `.js` files (`api.js`, `extraction-service.js`, `markup.js`, `project-service.js`, `pdf-converter.js`, `index.js`, `extract-cli.js`) — pre-`src/`-reorg duplicates that imported `../config`-style paths and were broken where they sat. Live app is entirely `src/`; nothing imported them. No behavior change.
16+
3. **Confidentiality:** gitignored the two real third-party plan PDFs (case_011 Bohler grading, case_012 San Marcos framing) — publicly downloadable drawings with my own markups, not redistributed via the repo. Consistent with the existing `case_C*` / `dogfood/sources/` convention.
17+
4. **Security:** `npm audit` flagged react-router (2 high + 1 moderate, all the same GHSA cluster) in `client/`. `npm audit fix` bumped it past the advisories; lockfile-only, client build verified, audit clean.
18+
5. **Doc-accuracy pass (the substantive one):** the README still documented the original single-pass Claude Vision flow and predated PR #2. Brought README + STATE in line with what `job-service.js` actually runs — source-type probe (`pdfjs-dist`) → parse path for digital PDFs, tiled vision for raster; documented the previously-missing LangFuse + Sentry observability; added the new modules to the key-files map and the new env vars to the config table; corrected the eval/Next-steps sections that still called tiling "eval-only" (it ships as the production raster path). Also linked the design specs/plans from the ADR index and noted that `active.md` is v0.10 (defense-in-depth) while v0.9 stays the pinned aggregate baseline.
19+
20+
**Commits:**
21+
- `9cb4ef2` — chore: remove dead pre-`src/` root duplicate modules
22+
- `b7d1450` — docs: freeze RedlineIQ as a portfolio snapshot (README banner, STATE.md, ignore real plans)
23+
- `32a7dce` — chore(deps): patch react-router to clear 2 high-severity advisories
24+
- `3a2745b` — docs: update README + STATE to match the shipped hybrid pipeline
25+
- tag `v0.9-portfolio` — portfolio / maintenance-mode snapshot marker
26+
27+
---
28+
929
## 2026-06-02 — parse/vision hybrid Phase 1 shipped (annotation-layer parse path)
1030

1131
**Summary:** Built the parse path the 5/29 "wrong-tool" realization pointed to: digital, un-flattened PDFs now route through a lossless `pdfjs-dist` annotation-layer parser instead of vision. Markup text + exact coordinates are read straight from the PDF; one cheap **text-only** Claude call assigns the semantic labels (type / related_to / confidence / ambiguous) that aren't stored in the file. The tiled-vision path is untouched — PDFs with no annotation layer route to vision exactly as before. **Scored end-to-end in the eval harness, the parse path holds recall and lifts precision sharply on the same sheet: case_012 recall 0.846 / precision 0.917 vs the vision path's 0.846 / 0.625 — equal recall, +0.292 precision.** Merged to main via PR #2 (merge commit `854fdbb`). TDD throughout; 39 offline assertions across 5 suites.

0 commit comments

Comments
 (0)