Skip to content

Commit e20d40a

Browse files
gwittebolleclaude
gwittebolle
and
claude
committed
test: golden methodology vectors + CI runner
12 hand-checked vectors (tokens in -> expected co2/cost) covering every model family, cache_read/cache_write paths, excluded models and edge cases. tests/run-vectors.sh replays them against the same awk formulas as persist-session.sh (relative tolerance 1e-6). New CI workflow runs the suite on push/PR. These vectors are the shared contract with downstream consumers of this methodology. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 60d937b commit e20d40a

4 files changed

Lines changed: 317 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: ci
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
concurrency:
9+
group: ci-${{ github.workflow }}-${{ github.ref }}
10+
cancel-in-progress: true
11+
12+
jobs:
13+
methodology-vectors:
14+
name: methodology golden vectors
15+
runs-on: ubuntu-latest
16+
timeout-minutes: 5
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
# jq is preinstalled on ubuntu-latest; the runner only needs bash + jq + awk.
21+
- name: Replay golden vectors against data/factors.json + data/prices.json
22+
run: bash tests/run-vectors.sh

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,10 @@ Factors from [Jegham et al. 2025](https://arxiv.org/abs/2505.09598), a peer-revi
178178

179179
Factors are editable in `data/factors.json`. See [METHODOLOGY.md](METHODOLOGY.md) for the full scientific basis, formula, and equivalences.
180180

181+
### Golden vectors
182+
183+
The methodology is pinned by golden test vectors in [`tests/methodology-vectors.json`](tests/methodology-vectors.json): hand-computed expected CO2/cost values for known token breakdowns, replayed by `bash tests/run-vectors.sh` in CI on every push. Downstream consumers (such as TokenClimate) keep a copy of this file and verify weekly that their implementation produces the same numbers. If you edit `data/factors.json` or `data/prices.json`, update the vectors in the same commit, otherwise CI fails.
184+
181185
## Dependencies
182186

183187
- `jq` - JSON parsing

‎tests/methodology-vectors.json‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
{
2+
"_purpose": "Canonical golden vectors for the claude-carbon methodology. Source of truth shared with downstream consumers (TokenClimate SaaS keeps a byte-identical copy in tests/fixtures/ and checks it weekly). Any change to data/factors.json or data/prices.json MUST update these expected values in the same commit.",
3+
"_formulas": {
4+
"co2_grams": "((input_tokens + cache_creation_tokens) * factor_in + cache_read_tokens * (factor_in * cache_read_factor) + output_tokens * factor_out) / 1e6",
5+
"cost_usd": "(input_tokens * price_in + cache_creation_tokens * (price_in * cache_write_multiplier) + cache_read_tokens * (price_in * cache_read_multiplier) + output_tokens * price_out) / 1e6"
6+
},
7+
"_token_semantics": "input_tokens here is the UNCACHED input (Anthropic usage.input_tokens). cache_creation_tokens (cache write) is separate. Note: the plugin DB column input_tokens stores input + cache_write; these vectors keep them split, as captured from the JSONL.",
8+
"_constants": {
9+
"cache_read_factor": 0.08,
10+
"cache_write_multiplier": 1.25,
11+
"cache_read_multiplier": 0.1,
12+
"factors_gco2e_per_mtok": {
13+
"fable": { "input": 1000, "output": 6000 },
14+
"opus": { "input": 500, "output": 3000 },
15+
"sonnet": { "input": 190, "output": 1140 },
16+
"haiku": { "input": 95, "output": 570 }
17+
},
18+
"prices_usd_per_mtok": {
19+
"fable": { "input": 10, "output": 50 },
20+
"opus": { "input": 5, "output": 25 },
21+
"sonnet": { "input": 3, "output": 15 },
22+
"haiku": { "input": 1, "output": 5 }
23+
}
24+
},
25+
"_rounding": "All token counts are multiples of 10,000 so every expected value is EXACT at <= 3 decimals for co2_grams and <= 4 decimals for cost_usd. This makes the vectors implementation-agnostic: the bash plugin prints co2 at 4 decimals / cost at 6 decimals, the SaaS rounds co2 to 3 decimals / cost to 4 decimals, and both land on the exact value. Runners compare with a relative tolerance of 1e-6.",
26+
"_excluded_semantics": "Vectors with excluded=true use a model string without 'claude' (or matching factors.json exclude_models). expected_* is null: the bash plugin stores co2_grams=0 / cost_usd=0 with excluded=1 (raw tokens kept, no estimate); the SaaS returns null ('CO2 non connu'). Both mean: no number is fabricated.",
27+
"vectors": [
28+
{
29+
"id": "fable-mixed",
30+
"description": "Fable 5, full mix of all four token kinds",
31+
"model": "claude-fable-5",
32+
"input_tokens": 50000,
33+
"cache_creation_tokens": 200000,
34+
"cache_read_tokens": 3000000,
35+
"output_tokens": 30000,
36+
"expected_co2_grams": 670.0,
37+
"expected_cost_usd": 7.5
38+
},
39+
{
40+
"id": "opus-no-cache",
41+
"description": "Opus, plain input/output, no cache traffic",
42+
"model": "claude-opus-4-6",
43+
"input_tokens": 1000000,
44+
"cache_creation_tokens": 0,
45+
"cache_read_tokens": 0,
46+
"output_tokens": 100000,
47+
"expected_co2_grams": 800.0,
48+
"expected_cost_usd": 7.5
49+
},
50+
{
51+
"id": "sonnet-cache-read-dominant",
52+
"description": "Sonnet, cache_read dominant - shape of a real Claude Code session (dated model id resolved by family pattern)",
53+
"model": "claude-sonnet-4-5-20250929",
54+
"input_tokens": 20000,
55+
"cache_creation_tokens": 150000,
56+
"cache_read_tokens": 12000000,
57+
"output_tokens": 80000,
58+
"expected_co2_grams": 305.9,
59+
"expected_cost_usd": 5.4225
60+
},
61+
{
62+
"id": "haiku-mixed",
63+
"description": "Haiku, full mix of all four token kinds",
64+
"model": "claude-3-5-haiku",
65+
"input_tokens": 40000,
66+
"cache_creation_tokens": 10000,
67+
"cache_read_tokens": 500000,
68+
"output_tokens": 20000,
69+
"expected_co2_grams": 19.95,
70+
"expected_cost_usd": 0.2025
71+
},
72+
{
73+
"id": "opus-cache-write-only",
74+
"description": "Opus, cache_write only: full input factor for CO2, 1.25x input price",
75+
"model": "claude-opus-4-6",
76+
"input_tokens": 0,
77+
"cache_creation_tokens": 100000,
78+
"cache_read_tokens": 0,
79+
"output_tokens": 0,
80+
"expected_co2_grams": 50.0,
81+
"expected_cost_usd": 0.625
82+
},
83+
{
84+
"id": "sonnet-cache-read-only",
85+
"description": "Sonnet, cache_read only: 0.08x input factor for CO2, 0.1x input price",
86+
"model": "claude-sonnet-4",
87+
"input_tokens": 0,
88+
"cache_creation_tokens": 0,
89+
"cache_read_tokens": 1000000,
90+
"output_tokens": 0,
91+
"expected_co2_grams": 15.2,
92+
"expected_cost_usd": 0.3
93+
},
94+
{
95+
"id": "all-zeros",
96+
"description": "Known model, zero tokens everywhere",
97+
"model": "claude-sonnet-4",
98+
"input_tokens": 0,
99+
"cache_creation_tokens": 0,
100+
"cache_read_tokens": 0,
101+
"output_tokens": 0,
102+
"expected_co2_grams": 0.0,
103+
"expected_cost_usd": 0.0
104+
},
105+
{
106+
"id": "excluded-non-claude-model",
107+
"description": "Non-Anthropic model (e.g. local model behind ANTHROPIC_BASE_URL): no estimate is fabricated",
108+
"model": "glm-4.7-flash",
109+
"input_tokens": 100000,
110+
"cache_creation_tokens": 50000,
111+
"cache_read_tokens": 2000000,
112+
"output_tokens": 40000,
113+
"excluded": true,
114+
"expected_co2_grams": null,
115+
"expected_cost_usd": null
116+
},
117+
{
118+
"id": "fable-mythos-alias",
119+
"description": "Fable family via the mythos model id",
120+
"model": "claude-mythos-5",
121+
"input_tokens": 10000,
122+
"cache_creation_tokens": 0,
123+
"cache_read_tokens": 0,
124+
"output_tokens": 10000,
125+
"expected_co2_grams": 70.0,
126+
"expected_cost_usd": 0.6
127+
},
128+
{
129+
"id": "opus-heavy-session",
130+
"description": "Opus, large session with heavy cache traffic",
131+
"model": "claude-opus-4-7",
132+
"input_tokens": 30000,
133+
"cache_creation_tokens": 400000,
134+
"cache_read_tokens": 8000000,
135+
"output_tokens": 60000,
136+
"expected_co2_grams": 715.0,
137+
"expected_cost_usd": 8.15
138+
},
139+
{
140+
"id": "haiku-tiny-session",
141+
"description": "Haiku, small symmetric session, no cache",
142+
"model": "claude-haiku",
143+
"input_tokens": 10000,
144+
"cache_creation_tokens": 0,
145+
"cache_read_tokens": 0,
146+
"output_tokens": 10000,
147+
"expected_co2_grams": 6.65,
148+
"expected_cost_usd": 0.06
149+
},
150+
{
151+
"id": "haiku-cache-read-only",
152+
"description": "Haiku, large cache_read only",
153+
"model": "claude-3-5-haiku",
154+
"input_tokens": 0,
155+
"cache_creation_tokens": 0,
156+
"cache_read_tokens": 10000000,
157+
"output_tokens": 0,
158+
"expected_co2_grams": 76.0,
159+
"expected_cost_usd": 1.0
160+
}
161+
]
162+
}

‎tests/run-vectors.sh‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env bash
2+
# run-vectors.sh — Replay tests/methodology-vectors.json against the plugin's
3+
# cost/CO2 formulas (the exact math of scripts/persist-session.sh compute_co2
4+
# and scripts/recompute.sh) using the CURRENT data/factors.json + data/prices.json.
5+
# Exits 1 on the first relative deviation above the tolerance.
6+
#
7+
# bash 3.2 compatible (macOS default): no associative arrays, no mapfile.
8+
# Dependencies: jq, awk (same as the rest of the plugin).
9+
10+
set -u
11+
12+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
13+
FACTORS_FILE="${SCRIPT_DIR}/../data/factors.json"
14+
PRICES_FILE="${SCRIPT_DIR}/../data/prices.json"
15+
VECTORS_FILE="${SCRIPT_DIR}/methodology-vectors.json"
16+
REL_TOL="0.000001" # 1e-6
17+
18+
command -v jq >/dev/null 2>&1 || { echo "FAIL: jq is required" >&2; exit 1; }
19+
[ -f "$FACTORS_FILE" ] || { echo "FAIL: missing $FACTORS_FILE" >&2; exit 1; }
20+
[ -f "$PRICES_FILE" ] || { echo "FAIL: missing $PRICES_FILE" >&2; exit 1; }
21+
[ -f "$VECTORS_FILE" ] || { echo "FAIL: missing $VECTORS_FILE" >&2; exit 1; }
22+
23+
# Emission factors (gCO2e per Mtok) + cache_read energy fraction
24+
F_FAB_IN="$(jq -r '.models.fable.input' "$FACTORS_FILE")"; F_FAB_OUT="$(jq -r '.models.fable.output' "$FACTORS_FILE")"
25+
F_OPUS_IN="$(jq -r '.models.opus.input' "$FACTORS_FILE")"; F_OPUS_OUT="$(jq -r '.models.opus.output' "$FACTORS_FILE")"
26+
F_SON_IN="$(jq -r '.models.sonnet.input' "$FACTORS_FILE")"; F_SON_OUT="$(jq -r '.models.sonnet.output' "$FACTORS_FILE")"
27+
F_HAI_IN="$(jq -r '.models.haiku.input' "$FACTORS_FILE")"; F_HAI_OUT="$(jq -r '.models.haiku.output' "$FACTORS_FILE")"
28+
CRF="$(jq -r '.cache_read_factor // 0.08' "$FACTORS_FILE")"
29+
30+
# Prices (USD per Mtok) + cache multipliers
31+
P_FAB_IN="$(jq -r '.models.fable.input' "$PRICES_FILE")"; P_FAB_OUT="$(jq -r '.models.fable.output' "$PRICES_FILE")"
32+
P_OPUS_IN="$(jq -r '.models.opus.input' "$PRICES_FILE")"; P_OPUS_OUT="$(jq -r '.models.opus.output' "$PRICES_FILE")"
33+
P_SON_IN="$(jq -r '.models.sonnet.input' "$PRICES_FILE")"; P_SON_OUT="$(jq -r '.models.sonnet.output' "$PRICES_FILE")"
34+
P_HAI_IN="$(jq -r '.models.haiku.input' "$PRICES_FILE")"; P_HAI_OUT="$(jq -r '.models.haiku.output' "$PRICES_FILE")"
35+
CW_MULT="$(jq -r '.cache_write_multiplier // 1.25' "$PRICES_FILE")"
36+
CR_MULT="$(jq -r '.cache_read_multiplier // 0.1' "$PRICES_FILE")"
37+
38+
EXCLUDE_MODELS="$(jq -r '(.exclude_models // []) | join("|")' "$FACTORS_FILE")"
39+
40+
# Same exclusion rule as persist-session.sh is_excluded_model()
41+
is_excluded_model() {
42+
local model="$1"
43+
if ! echo "$model" | grep -qi "claude"; then return 0; fi
44+
if [ -n "$EXCLUDE_MODELS" ] && echo "$model" | grep -qiE "$EXCLUDE_MODELS"; then return 0; fi
45+
return 1
46+
}
47+
48+
# Relative-tolerance comparison (absolute when expected == 0). Returns 0 on match.
49+
close_enough() {
50+
echo "$1 $2 $REL_TOL" | LC_ALL=C awk '{
51+
actual = $1; expected = $2; tol = $3;
52+
diff = actual - expected; if (diff < 0) diff = -diff;
53+
ref = expected; if (ref < 0) ref = -ref;
54+
if (ref == 0) { exit (diff <= tol) ? 0 : 1 }
55+
exit (diff / ref <= tol) ? 0 : 1
56+
}'
57+
}
58+
59+
N="$(jq '.vectors | length' "$VECTORS_FILE")"
60+
FAILURES=0
61+
PASSED=0
62+
i=0
63+
while [ "$i" -lt "$N" ]; do
64+
ROW="$(jq -r --argjson i "$i" '.vectors[$i] | [
65+
.id, .model,
66+
(.input_tokens // 0), (.cache_creation_tokens // 0),
67+
(.cache_read_tokens // 0), (.output_tokens // 0),
68+
(if .excluded == true then "1" else "0" end),
69+
(.expected_co2_grams // 0), (.expected_cost_usd // 0)
70+
] | @tsv' "$VECTORS_FILE")"
71+
IFS="$(printf '\t')" read -r ID MODEL IN CW CR OUT EXCLUDED EXP_CO2 EXP_COST <<EOF
72+
$ROW
73+
EOF
74+
75+
# Mirror persist-session.sh compute_co2: exclusion first, then family pick.
76+
if is_excluded_model "$MODEL"; then
77+
CO2="0"; COST="0"
78+
if [ "$EXCLUDED" != "1" ]; then
79+
echo "FAIL ${ID}: model '${MODEL}' is excluded by the plugin but the vector is not marked excluded"
80+
FAILURES=$((FAILURES + 1)); i=$((i + 1)); continue
81+
fi
82+
# Excluded vectors expect 0/0 from the plugin (expected_* is null upstream).
83+
EXP_CO2="0"; EXP_COST="0"
84+
else
85+
if [ "$EXCLUDED" = "1" ]; then
86+
echo "FAIL ${ID}: vector marked excluded but model '${MODEL}' is not excluded by the plugin"
87+
FAILURES=$((FAILURES + 1)); i=$((i + 1)); continue
88+
fi
89+
FAMILY="sonnet"
90+
echo "$MODEL" | grep -qiE "fable|mythos" && FAMILY="fable"
91+
echo "$MODEL" | grep -qi "opus" && FAMILY="opus"
92+
echo "$MODEL" | grep -qi "haiku" && FAMILY="haiku"
93+
case "$FAMILY" in
94+
fable) FIN="$F_FAB_IN"; FOUT="$F_FAB_OUT"; PIN="$P_FAB_IN"; POUT="$P_FAB_OUT" ;;
95+
opus) FIN="$F_OPUS_IN"; FOUT="$F_OPUS_OUT"; PIN="$P_OPUS_IN"; POUT="$P_OPUS_OUT" ;;
96+
haiku) FIN="$F_HAI_IN"; FOUT="$F_HAI_OUT"; PIN="$P_HAI_IN"; POUT="$P_HAI_OUT" ;;
97+
*) FIN="$F_SON_IN"; FOUT="$F_SON_OUT"; PIN="$P_SON_IN"; POUT="$P_SON_OUT" ;;
98+
esac
99+
# Same awk expressions and printf precision as persist-session.sh
100+
CO2="$(echo "$IN $CW $CR $OUT $FIN $FOUT $CRF" | LC_ALL=C awk \
101+
'{printf "%.4f", (($1 + $2) * $5 + $3 * ($5 * $7) + $4 * $6) / 1000000}')"
102+
COST="$(echo "$IN $CW $CR $OUT $PIN $POUT $CW_MULT $CR_MULT" | LC_ALL=C awk \
103+
'{printf "%.6f", ($1 * $5 + $2 * ($5 * $7) + $3 * ($5 * $8) + $4 * $6) / 1000000}')"
104+
fi
105+
106+
OK=1
107+
if ! close_enough "$CO2" "$EXP_CO2"; then
108+
echo "FAIL ${ID}: co2_grams ${CO2} != expected ${EXP_CO2} (model ${MODEL})"
109+
OK=0
110+
fi
111+
if ! close_enough "$COST" "$EXP_COST"; then
112+
echo "FAIL ${ID}: cost_usd ${COST} != expected ${EXP_COST} (model ${MODEL})"
113+
OK=0
114+
fi
115+
if [ "$OK" = "1" ]; then
116+
echo "PASS ${ID}: co2=${CO2} g, cost=\$${COST}"
117+
PASSED=$((PASSED + 1))
118+
else
119+
FAILURES=$((FAILURES + 1))
120+
fi
121+
i=$((i + 1))
122+
done
123+
124+
echo ""
125+
if [ "$FAILURES" -gt 0 ]; then
126+
echo "${FAILURES}/${N} vector(s) FAILED (${PASSED} passed)."
127+
exit 1
128+
fi
129+
echo "All ${N} methodology vectors passed."

0 commit comments

Comments
 (0)