Skip to content

Commit b03c0f0

Browse files
bradleygauthierbradleygauthier
committed
feat(vault): eval + RRF retrieval, evals harness and scripts; postgres/config updates
1 parent ea4f372 commit b03c0f0

24 files changed

Lines changed: 917 additions & 13 deletions

‎evals/baseline.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"model": "nomic-embed-text",
3+
"corpus_hash": "9f513c5c8cd63fe0d0eb0180171c68fe289f645fa9133541c1db6c28b7ab7187",
4+
"legacy": {
5+
"recall@5": 1.0,
6+
"recall@10": 1.0,
7+
"mrr@10": 1.0
8+
},
9+
"rrf": {
10+
"recall@5": 1.0,
11+
"recall@10": 1.0,
12+
"mrr@10": 1.0
13+
}
14+
}

‎evals/build_corpus.py‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
#!/usr/bin/env python3
2+
# Copyright 2026 Quantum Pipes Technologies, LLC
3+
# SPDX-License-Identifier: Apache-2.0
4+
"""Author the retrieval-eval corpus + golden set (askqp-100 I2).
5+
6+
Writes ``corpus/*.md`` and ``golden.jsonl`` deterministically. The content is
7+
synthetic operational documentation (no private data). Run from repos/vault:
8+
9+
python evals/build_corpus.py
10+
"""
11+
12+
from __future__ import annotations
13+
14+
import json
15+
from pathlib import Path
16+
17+
_HERE = Path(__file__).resolve().parent
18+
19+
# doc id -> markdown body. Distinctive vocabulary per topic so each golden
20+
# question has an unambiguous target.
21+
CORPUS: dict[str, str] = {
22+
"incident-response": (
23+
"# Incident Response\n\n"
24+
"When an incident is detected, the on-call engineer must acknowledge it "
25+
"within 15 minutes. Severity is classified as P0 (critical), P1 (high), "
26+
"P2 (medium), or P3 (low). A P0 incident requires immediate escalation to "
27+
"the incident commander and a status page update.\n"
28+
),
29+
"onboarding": (
30+
"# Employee Onboarding\n\n"
31+
"Current onboarding takes three weeks. The goal is to reduce it to two "
32+
"weeks by parallelizing equipment setup and access provisioning. IT and HR "
33+
"coordinate to guarantee day-one readiness for every new hire.\n"
34+
),
35+
"auth-migration": (
36+
"# Authentication Service Migration\n\n"
37+
"The migration to the new authentication service is under way. Legacy API "
38+
"clients must obtain updated tokens before the old endpoint is retired. A "
39+
"deprecation notice goes out to integrators one month ahead of cutover.\n"
40+
),
41+
"backup-policy": (
42+
"# Backup Policy\n\n"
43+
"Full database backups run nightly at 02:00 UTC and are retained for 30 "
44+
"days. A monthly snapshot is archived to cold storage for one year. Restore "
45+
"drills are performed quarterly to prove the backups are usable.\n"
46+
),
47+
"data-retention": (
48+
"# Data Retention\n\n"
49+
"Customer records are retained for seven years to meet financial reporting "
50+
"obligations. Application logs are kept for 90 days. Personally "
51+
"identifiable information is purged within 30 days of account deletion.\n"
52+
),
53+
"api-rate-limits": (
54+
"# API Rate Limits\n\n"
55+
"Each API key is allowed 600 requests per minute, with a short burst "
56+
"allowance of 100. Exceeding the ceiling returns HTTP 429 with a "
57+
"Retry-After header indicating when to try again.\n"
58+
),
59+
"encryption-standards": (
60+
"# Encryption Standards\n\n"
61+
"Data at rest is encrypted with AES-256-GCM. Data in transit uses TLS 1.3. "
62+
"Keys are rotated every quarter. Deprecated algorithms such as RSA-1024 and "
63+
"MD5 are prohibited across all services.\n"
64+
),
65+
"kubernetes-deploy": (
66+
"# Kubernetes Deployment\n\n"
67+
"Services deploy with a rolling update strategy and readiness probes. Every "
68+
"workload runs a minimum of three replicas. Major releases use a blue-green "
69+
"cutover so traffic shifts only after the new version is healthy.\n"
70+
),
71+
"postgres-tuning": (
72+
"# PostgreSQL Tuning\n\n"
73+
"Connection pooling is handled by pgbouncer in transaction mode. "
74+
"shared_buffers is set to roughly 25 percent of system memory. Autovacuum "
75+
"thresholds are lowered on high-write tables to avoid bloat.\n"
76+
),
77+
"oncall-rotation": (
78+
"# On-Call Rotation\n\n"
79+
"The on-call rotation is weekly, with a primary and a secondary engineer. "
80+
"Handoff happens every Monday at 10:00. If the primary does not respond, the "
81+
"page escalates to the secondary automatically.\n"
82+
),
83+
"code-review": (
84+
"# Code Review Policy\n\n"
85+
"Every pull request requires two approvals before merge. Self-merge is not "
86+
"allowed. Continuous integration must be green. Changes that touch "
87+
"authentication require an additional security review.\n"
88+
),
89+
"release-process": (
90+
"# Release Process\n\n"
91+
"A production release requires two approvals and a green CI pipeline. During "
92+
"a code freeze before a major release, no new features may land: only "
93+
"critical bug fixes are permitted. Release notes are mandatory.\n"
94+
),
95+
}
96+
97+
# Golden retrieval set. relevant_resource_ids are doc ids that SHOULD be retrieved.
98+
GOLDEN: list[dict] = [
99+
{"id": "q001", "question": "How long does the on-call engineer have to acknowledge an incident?",
100+
"relevant_resource_ids": ["incident-response"], "category": "lookup"},
101+
{"id": "q002", "question": "What is the goal for reducing new-hire onboarding time?",
102+
"relevant_resource_ids": ["onboarding"], "category": "lookup"},
103+
{"id": "q003", "question": "What does a P0 incident severity require?",
104+
"relevant_resource_ids": ["incident-response"], "category": "acronym"},
105+
{"id": "q004", "question": "Which two teams coordinate to make a new hire ready on day one?",
106+
"relevant_resource_ids": ["onboarding"], "category": "synthesis"},
107+
{"id": "q005", "question": "When are full database backups taken and how long are they kept?",
108+
"relevant_resource_ids": ["backup-policy"], "category": "lookup"},
109+
{"id": "q006", "question": "How many years are customer records kept?",
110+
"relevant_resource_ids": ["data-retention"], "category": "paraphrase"},
111+
{"id": "q007", "question": "What is the request ceiling per minute for an API key?",
112+
"relevant_resource_ids": ["api-rate-limits"], "category": "lookup"},
113+
{"id": "q008", "question": "Which algorithm encrypts data at rest?",
114+
"relevant_resource_ids": ["encryption-standards"], "category": "acronym"},
115+
{"id": "q009", "question": "How many approvals does a production release need?",
116+
"relevant_resource_ids": ["release-process", "code-review"], "category": "synthesis"},
117+
{"id": "q010", "question": "What is not permitted during a code freeze?",
118+
"relevant_resource_ids": ["release-process"], "category": "negation"},
119+
]
120+
121+
122+
def main() -> None:
123+
corpus_dir = _HERE / "corpus"
124+
corpus_dir.mkdir(exist_ok=True)
125+
for doc_id, body in sorted(CORPUS.items()):
126+
(corpus_dir / f"{doc_id}.md").write_text(body, encoding="utf-8")
127+
with (_HERE / "golden.jsonl").open("w", encoding="utf-8") as f:
128+
for row in GOLDEN:
129+
f.write(json.dumps({**row, "author": "bootstrap", "added": "2026-07-10"}) + "\n")
130+
print(f"wrote {len(CORPUS)} corpus docs + {len(GOLDEN)} golden questions")
131+
132+
133+
if __name__ == "__main__":
134+
main()

‎evals/corpus/api-rate-limits.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# API Rate Limits
2+
3+
Each API key is allowed 600 requests per minute, with a short burst allowance of 100. Exceeding the ceiling returns HTTP 429 with a Retry-After header indicating when to try again.

‎evals/corpus/auth-migration.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Authentication Service Migration
2+
3+
The migration to the new authentication service is under way. Legacy API clients must obtain updated tokens before the old endpoint is retired. A deprecation notice goes out to integrators one month ahead of cutover.

‎evals/corpus/backup-policy.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Backup Policy
2+
3+
Full database backups run nightly at 02:00 UTC and are retained for 30 days. A monthly snapshot is archived to cold storage for one year. Restore drills are performed quarterly to prove the backups are usable.

‎evals/corpus/code-review.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Code Review Policy
2+
3+
Every pull request requires two approvals before merge. Self-merge is not allowed. Continuous integration must be green. Changes that touch authentication require an additional security review.

‎evals/corpus/data-retention.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Data Retention
2+
3+
Customer records are retained for seven years to meet financial reporting obligations. Application logs are kept for 90 days. Personally identifiable information is purged within 30 days of account deletion.

‎evals/corpus/encryption-standards.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Encryption Standards
2+
3+
Data at rest is encrypted with AES-256-GCM. Data in transit uses TLS 1.3. Keys are rotated every quarter. Deprecated algorithms such as RSA-1024 and MD5 are prohibited across all services.

‎evals/corpus/incident-response.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Incident Response
2+
3+
When an incident is detected, the on-call engineer must acknowledge it within 15 minutes. Severity is classified as P0 (critical), P1 (high), P2 (medium), or P3 (low). A P0 incident requires immediate escalation to the incident commander and a status page update.

‎evals/corpus/kubernetes-deploy.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Kubernetes Deployment
2+
3+
Services deploy with a rolling update strategy and readiness probes. Every workload runs a minimum of three replicas. Major releases use a blue-green cutover so traffic shifts only after the new version is healthy.

0 commit comments

Comments
 (0)