v1.4.0: AlphaMissense enrichment, config system, HTML report polish

Add AlphaMissense variant pathogenicity enrichment — enrichment-only
annotator (same pattern as gnomAD), pre-built SQLite cache on
HuggingFace, AM Score column in all three report formats. Build script
with Zenodo streaming and local TSV modes. PharmGKB rows display AM
scores as neutral with caveat across all renderers.

Add config file system — config.toml with per-source on/off toggles,
commercial mode safety switch (license.commercial = true auto-disables
non-commercial sources). CLI flags override config per-invocation.

Fix HTML report layout (#20) — table overflow with horizontal scroll,
sticky rsID column, description max-width, refs collapsed into
<details>, conditional Review Status column, stat card flex-wrap.
Repute row tints (red/green by significance). Sortable columns via
inline JS. Magnitude legend with scoring tables. Source floor note.

Add 25 unit tests for AlphaMissense build script (#24) covering TSV
parsing, gnomAD rsID join, chr prefix normalization, --no-gnomad path,
multi-allelic composite PK, batched insert, and end-to-end integration.

Code review fixes: AM caveat consistency, Content-Length download
integrity check, dead code removal, _connection() guards, disk space
preflight at 5x gz size, multi-allelic MAX aggregation tests.

Closes #20, closes #24

Author: dial481

CHANGELOG.md

@@ -2,6 +2,68 @@
 
 All notable changes are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.0]
+
+### Added
+- **AlphaMissense variant pathogenicity enrichment.** New
+  `AlphaMissenseAnnotator` enriches annotations with missense variant
+  pathogenicity scores from DeepMind's AlphaMissense (71M variants,
+  CC BY 4.0). Pre-built SQLite cache downloaded from HuggingFace via
+  `db update`. AM Score column in terminal, HTML, and JSON reports.
+  PharmGKB rows show AM scores as neutral with caveat (protein
+  structure impact only — tooltip in HTML, dimmed `*` footnote in
+  terminal, `am_caveat` field in JSON). `--no-alphamissense` flag to
+  skip.
+- **Config file system.** `config.toml` with per-source on/off toggles
+  and `license.commercial = true` safety switch that auto-disables
+  non-commercial sources (SNPedia). `allelix config show/set/reset`
+  CLI commands. CLI flags override config per-invocation.
+- `scripts/build_alphamissense_cache.py` — AlphaMissense cache build
+  script with Zenodo HTTPS streaming (default) and local TSV modes.
+  Joins against gnomAD cache for coordinate-to-rsID mapping.
+- AlphaMissense CC BY 4.0 attribution in HTML and JSON reports.
+- Magnitude scoring legend in HTML report (collapsible, per-source
+  scoring tables for ClinVar, PharmGKB, GWAS, SNPedia).
+- Source floor note in HTML report when per-source magnitude minimums
+  are active.
+- Repute row background tints in HTML report (red for pathogenic/risk,
+  green for protective/benign) derived from existing significance
+  field.
+- Sortable columns in HTML report (magnitude, gene, source, AM score)
+  via inline JavaScript.
+- ADR-0027 documenting the AlphaMissense enrichment cache architecture.
+- `scripts/run-tests.sh` — detached background test runner with log
+  rotation.
+
+### Fixed
+- HTML report table overflows viewport, columns clipped on left (#20).
+  Added `overflow-x: auto` container, sticky rsID column,
+  `max-width` on description cells, refs collapsed into `<details>`
+  toggle, conditional Review Status column (hidden when all empty),
+  stat card `flex-wrap`.
+- AlphaMissense build script has zero unit-test coverage (#24). Added
+  25 tests covering TSV parsing, gnomAD rsID join, chr prefix
+  normalization, `--no-gnomad` NULL-rsid path, multi-allelic composite
+  PK, batched insert, and end-to-end integration.
+- Download integrity: Content-Length check after downloads catches
+  truncated files.
+- Disk space preflight before decompressing `.sqlite.gz` caches uses
+  5x gz size (accounts for gz + decompressed tmp on disk
+  simultaneously).
+- `_connection()` guards on gnomAD and AlphaMissense annotators raise
+  `FileNotFoundError` with actionable message when cache is missing.
+- Dead `cache_exists()` removed from gnomAD and AlphaMissense loaders.
+- Legacy caches stamp remote signal instead of re-downloading on
+  `db update`.
+- README database sizes updated to match actual on-disk measurements.
+
+### Changed
+- `db update` display includes gnomAD and AlphaMissense in "Analyzing
+  against" annotator list.
+- Both build scripts (`build_gnomad_cache.py`,
+  `build_alphamissense_cache.py`) run `VACUUM` for smaller output
+  files.
+
 ## [1.3.1]
 
 ### Fixed
@@ -41,8 +103,8 @@ All notable changes are documented here. Format follows [Keep a Changelog](https
   non-finding filter degrades gracefully. Signal carries
   `cpic:unavailable` so recovery auto-triggers a refresh.
 - `scripts/build_gnomad_cache.py` — streaming VCF build script for
-  the gnomAD frequency cache. Downloads ~120GB over HTTPS, never saves
-  VCFs to disk, outputs ~1GB SQLite.
+  the gnomAD frequency cache. Downloads ~185GB over HTTPS, never saves
+  VCFs to disk, outputs ~6GB SQLite.
 - `scripts/extract_array_manifest.py` — extracts rsID superset from
   genotype files for filtered gnomAD cache builds.
 - gnomAD ODbL v1.0 attribution in HTML and JSON reports.
@@ -1294,6 +1356,7 @@ All notable changes are documented here. Format follows [Keep a Changelog](https
 - GitHub Actions CI matrix on Python 3.11 and 3.12.
 
 
+[1.4.0]: https://github.com/dial481/allelix/compare/v1.3.1...v1.4.0
 [1.3.1]: https://github.com/dial481/allelix/compare/v1.3.0...v1.3.1
 [1.3.0]: https://github.com/dial481/allelix/compare/v1.2.0...v1.3.0
 [1.2.0]: https://github.com/dial481/allelix/compare/v1.1.1...v1.2.0

README.md

@@ -3,9 +3,11 @@
 Open-source command-line toolkit for analyzing raw genotype files from consumer DNA testing services. Format-agnostic ingestion, database-agnostic annotation, offline-first.
 
 > **Status:** Production — six parser formats, four annotators (ClinVar +
-> PharmGKB + GWAS Catalog + SNPedia), dual-build ClinVar caches
-> (GRCh37 + GRCh38), HTML/JSON/terminal reports, methylation +
-> pharmacogenomics focused commands, report diffing. Build
+> PharmGKB + GWAS Catalog + SNPedia), two enrichment sources (gnomAD
+> population frequencies + AlphaMissense pathogenicity), dual-build
+> ClinVar caches (GRCh37 + GRCh38), HTML/JSON/terminal reports,
+> methylation + pharmacogenomics focused commands, report diffing,
+> persistent config with commercial-mode safety switch. Build
 > auto-detection from position data (ADR-0021). No regex on prose
 > anywhere in production. Release notes: [`CHANGELOG.md`](CHANGELOG.md).
 
@@ -26,9 +28,9 @@ python tests/generate_mock_data.py
 # Show summary statistics for a genotype file
 allelix stats tests/fixtures/mock_myhappygenes.txt
 
-# Download reference databases (ClinVar VCF ~100MB compressed; PharmGKB clinical
-# annotations ~1MB). Re-runs skip downloads when local + remote signals match;
-# use --force to refresh unconditionally.
+# Download reference databases. First run downloads all sources (~15GB
+# on disk with gnomAD + AlphaMissense). Use --no-gnomad / --no-alphamissense
+# to skip the large enrichment databases. Re-runs skip unchanged sources.
 allelix db update
 allelix db status   # see what's cached
 
@@ -76,6 +78,8 @@ Adding a new format means adding one file to `allelix/parsers/` and registering
 | CPIC (per-allele function table) | ✓ | Internal data source for the PharmGKB filter. Fetched from `api.cpicpgx.org` at `db update` time. Used to populate the `pharmgkb_allele_function` table — not surfaced to end users as its own annotator. |
 | SNPedia | ✓ | CC BY-NC-SA 3.0 US. **Optional — requires a one-time download** via `python scripts/scrape_snpedia.py`. Scrapes both `Category:Is_a_snp` (111,726 pages) and `Category:Is_a_genotype` (104,806 pages) from the MediaWiki API. Stores raw wiki markup; the annotator parses structured genotype templates at query time. If the SNPedia database is absent, analysis runs without it. For commercial use, pass `--exclude-snpedia` or skip the scrape step — either way, `analyze` runs using all other databases and omits SNPedia annotations. |
 | GWAS Catalog | ✓ | Public domain (EBI/NHGRI). Trait–SNP associations with p-values and effect sizes. Carrier rule (ADR-0007) requires the user to carry the risk allele. P-value magnitude scoring (ADR-0024) maps continuous p-values to the 0–10 scale; unknown-risk-allele entries fire on rsID match alone but are capped at 3.0. |
+| gnomAD | ✓ | ODbL v1.0. **Enrichment annotator** — adds population allele frequency context to existing annotations. Shows how common each variant is in the general population (~16M exome variants from 730K individuals). A pathogenic variant that 35% of people carry reads very differently from one seen in 0.001%. Pre-built cache downloaded via `db update` (~6GB on disk). Use `--no-gnomad` to skip. |
+| AlphaMissense | ✓ | CC BY 4.0. **Enrichment annotator** — adds DeepMind's protein-structure-based pathogenicity predictions to existing annotations. Scores 71M missense variants on a 0–1 scale: <0.34 = likely benign, >0.564 = likely pathogenic. Complements ClinVar's expert classifications with computational predictions — especially valuable for variants ClinVar hasn't reviewed yet. Pre-built cache downloaded via `db update` (~8GB on disk). Use `--no-alphamissense` to skip. |
 
 ### Known PharmGKB limitation: reference-genotype rows where ClinVar and CPIC both lack data
 
@@ -110,6 +114,39 @@ This is not a disclaimer afterthought. It is a design constraint that affects mo
 - Reference databases are downloaded via `allelix db update` and cached locally.
 - Analysis runs offline against local database caches. A brief freshness check runs before analysis by default (skipped with `--no-update`).
 
+## Configuration
+
+Allelix stores persistent configuration in `config.toml` (in the data directory, default `~/.local/share/allelix/`). A default config is created on first run.
+
+```bash
+# View current config
+allelix config show
+
+# Disable a source permanently
+allelix config set sources.gnomad false
+
+# Enable commercial mode (auto-disables non-commercial sources like SNPedia)
+allelix config set license.commercial true
+```
+
+CLI flags (`--no-gnomad`, `--no-alphamissense`, `--exclude-snpedia`) override the config for a single run. The config sets the baseline; flags override per-invocation.
+
+### Database sizes and download times
+
+Not all databases are equal in size. `allelix db update` downloads them all by default, but you can skip the large ones if disk space or bandwidth is a concern:
+
+| Database | On disk | Download time | What it adds |
+|---|---|---|---|
+| ClinVar (GRCh37 + GRCh38) | ~900MB | 1–2 min | Core clinical variant classifications. Required. |
+| PharmGKB + CPIC | ~6MB | seconds | Drug-gene interactions. |
+| GWAS Catalog | ~200MB | 1–2 min | Trait-SNP associations from genome-wide studies. |
+| gnomAD | ~6GB | 5–15 min | Population allele frequencies (how common is this variant?). |
+| AlphaMissense | ~8GB | 5–15 min | Missense pathogenicity predictions (how likely to break protein function?). |
+
+gnomAD and AlphaMissense are the largest but add the most interpretive context. gnomAD answers "is this variant rare or common?" — a pathogenic variant carried by 35% of the population reads very differently from one seen in 3 people. AlphaMissense answers "does this missense change likely damage the protein?" — especially valuable for the thousands of variants ClinVar hasn't reviewed yet.
+
+To skip either during download: `allelix db update --no-gnomad --no-alphamissense`. To disable permanently: `allelix config set sources.gnomad false`.
+
 ## Data Sources & Licensing
 
 Allelix source code is licensed under the **GNU Affero General Public License v3.0 or later** (AGPL-3.0-or-later). Allelix ships with **zero third-party data**. All reference databases are downloaded by the user at runtime via `allelix db update`. Each database retains its original license on the user's machine:
@@ -122,8 +159,9 @@ Allelix source code is licensed under the **GNU Affero General Public License v3
 | CPIC | cpicpgx.org | CC BY-SA 4.0 | Attribution required. Per-allele function data fetched from `api.cpicpgx.org` at `db update` time; used internally for the PharmGKB non-finding filter (ADR-0020), not surfaced as its own annotator. |
 | SNPedia | snpedia.com | CC BY-NC-SA 3.0 US | Attribution required, **non-commercial only**. Use `--exclude-snpedia` to omit. |
 | gnomAD | gnomad.broadinstitute.org | ODbL v1.0 | Attribution required. Population allele frequencies for context; not a clinical annotator. Use `--no-gnomad` to omit. |
+| AlphaMissense | zenodo.org/records/10813168 | CC BY 4.0 | Attribution required. Cheng et al., Science 2023. Missense variant pathogenicity predictions. Use `--no-alphamissense` to omit. |
 
-**Commercial users:** SNPedia content is non-commercial. Pass `--exclude-snpedia` to any analysis command, or skip the `scripts/scrape_snpedia.py` step entirely — either way, `analyze` runs using all other databases and omits SNPedia annotations automatically.
+**Commercial users:** SNPedia content is non-commercial. Set `allelix config set license.commercial true` to permanently disable non-commercial sources, or pass `--exclude-snpedia` per-invocation. Either way, `analyze` runs using all other databases and omits SNPedia annotations automatically. All other databases (ClinVar, PharmGKB, GWAS Catalog, gnomAD, AlphaMissense) are compatible with commercial use.
 
 ### SNPedia data download
 

allelix/annotators/__init__.py

@@ -6,6 +6,7 @@
 
 from typing import TYPE_CHECKING
 
+from allelix.annotators.alphamissense import AlphaMissenseAnnotator
 from allelix.annotators.base import Annotator
 from allelix.annotators.clinvar import CLINVAR_SUPPORTED_BUILDS, ClinVarAnnotator
 from allelix.annotators.gnomad import GnomadAnnotator
@@ -46,10 +47,12 @@ def get_annotators(
     gwas = GWASCatalogAnnotator(data_dir, filter_traits=gwas_filter_traits)
     snpedia = SNPediaAnnotator(data_dir, clinvar_ref_provider=clinvar.reference_for)
     gnomad = GnomadAnnotator(data_dir)
-    return [clinvar, pharmgkb, gwas, snpedia, gnomad]
+    alphamissense = AlphaMissenseAnnotator(data_dir)
+    return [clinvar, pharmgkb, gwas, snpedia, gnomad, alphamissense]
 
 
 __all__ = [
+    "AlphaMissenseAnnotator",
     "Annotator",
     "ClinVarAnnotator",
     "GWASCatalogAnnotator",

allelix/annotators/alphamissense.py

@@ -0,0 +1,165 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 dial481
+"""AlphaMissense variant pathogenicity enrichment.
+
+AlphaMissense is not a clinical annotator — it does not produce
+Annotation objects. It enriches existing annotations with missense
+variant pathogenicity predictions. The pipeline calls
+``bulk_lookup()`` after all annotators have run, and stamps each
+annotation's ``am_pathogenicity`` and ``am_class`` fields.
+
+License: CC BY 4.0. Attribution: Cheng et al., Science 2023
+(doi:10.1126/science.adg7492).
+"""
+
+from __future__ import annotations
+
+import logging
+import sqlite3
+from typing import TYPE_CHECKING, ClassVar
+
+from allelix.annotators.base import Annotator
+from allelix.databases.alphamissense_loader import (
+    ALPHAMISSENSE_CACHE_URL,
+    ALPHAMISSENSE_DB_FILENAME,
+    install_prebuilt_cache,
+)
+from allelix.databases.manager import download, get_database_info, head_request_headers
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+    from allelix.models import Annotation, Variant
+
+logger = logging.getLogger(__name__)
+
+_BULK_BATCH_SIZE = 900
+
+
+class AlphaMissenseAnnotator(Annotator):
+    """Missense variant pathogenicity enrichment from AlphaMissense.
+
+    Subclasses Annotator for ``db update`` / ``db status`` / ``is_ready()``
+    integration. ``annotate()`` always returns ``[]`` — AlphaMissense does
+    not participate in the per-variant annotation loop.
+    """
+
+    name: ClassVar[str] = "alphamissense"
+    display_name: ClassVar[str] = "AlphaMissense"
+    attribution: ClassVar[str] = "AlphaMissense"
+    requires_download: ClassVar[bool] = True
+
+    def __init__(self, data_dir: Path) -> None:
+        """Bind to the data directory."""
+        super().__init__(data_dir)
+        self._db_path = data_dir / ALPHAMISSENSE_DB_FILENAME
+        self._conn: sqlite3.Connection | None = None
+
+    def _connection(self) -> sqlite3.Connection:
+        if self._conn is None:
+            if not self._db_path.exists():
+                raise FileNotFoundError(
+                    f"AlphaMissense cache not found at {self._db_path}. "
+                    "Run `allelix db update` first."
+                )
+            self._conn = sqlite3.connect(self._db_path)
+        return self._conn
+
+    def setup(self) -> None:
+        """Download the pre-built AlphaMissense cache from HuggingFace."""
+        signal = self.fetch_remote_signal()
+        gz_path = self.data_dir / "alphamissense.sqlite.gz"
+        download(ALPHAMISSENSE_CACHE_URL, gz_path)
+        install_prebuilt_cache(
+            gz_path,
+            self._db_path,
+            source_url=ALPHAMISSENSE_CACHE_URL,
+            remote_signal=signal,
+        )
+        try:
+            gz_path.unlink()
+        except OSError:
+            logger.warning("Could not remove staged file at %s", gz_path)
+
+    def is_ready(self) -> bool:
+        """True when the AlphaMissense SQLite cache exists and is queryable."""
+        return get_database_info(self._db_path, "alphamissense") is not None
+
+    def version(self) -> str | None:
+        """Return the cached database version, or None."""
+        info = get_database_info(self._db_path, "alphamissense")
+        return info["version"] if info else None
+
+    def record_count(self) -> int | None:
+        """Return the number of variants in the cache, or None."""
+        info = get_database_info(self._db_path, "alphamissense")
+        return info["record_count"] if info else None
+
+    def close(self) -> None:
+        """Close the SQLite connection if open."""
+        if self._conn is not None:
+            self._conn.close()
+            self._conn = None
+
+    def fetch_remote_signal(self) -> str | None:
+        """Probe the HuggingFace asset URL for ETag or Last-Modified."""
+        headers = head_request_headers(ALPHAMISSENSE_CACHE_URL)
+        if headers is None:
+            return None
+        etag = headers.get("ETag") or headers.get("Etag")
+        last_modified = headers.get("Last-Modified") or headers.get("Last-modified")
+        if etag:
+            return f"etag:{etag.strip()}"
+        if last_modified:
+            return f"lm:{last_modified.strip()}"
+        return None
+
+    def cached_remote_signal(self) -> str | None:
+        """Return the remote signal stored at last successful download."""
+        info = get_database_info(self._db_path, "alphamissense")
+        if not info or not info["remote_signal"]:
+            return None
+        return info["remote_signal"]
+
+    def annotate(self, variant: Variant) -> list[Annotation]:
+        """Not used — AlphaMissense enriches, does not annotate. Always returns []."""
+        return []
+
+    def lookup(self, rsid: str) -> tuple[float, str] | None:
+        """Return (am_pathogenicity, am_class) for a single rsID, or None."""
+        conn = self._connection()
+        row = conn.execute(
+            "SELECT MAX(am_pathogenicity), am_class FROM alphamissense_scores WHERE rsid = ?",
+            (rsid,),
+        ).fetchone()
+        if row is None or row[0] is None:
+            return None
+        return (row[0], row[1])
+
+    def bulk_lookup(self, rsids: set[str]) -> dict[str, tuple[float, str]]:
+        """Return ``{rsid: (am_pathogenicity, am_class)}`` for found rsIDs.
+
+        When an rsID maps to multiple rows (different ref/alt at the same
+        position), returns the highest pathogenicity score and its class —
+        consistent with gnomAD's ``MAX(af)`` approach.
+
+        Batches into chunks of 900 to stay within SQLite's variable limit.
+        """
+        if not rsids:
+            return {}
+        conn = self._connection()
+        result: dict[str, tuple[float, str]] = {}
+        rsid_list = list(rsids)
+        for i in range(0, len(rsid_list), _BULK_BATCH_SIZE):
+            batch = rsid_list[i : i + _BULK_BATCH_SIZE]
+            placeholders = ",".join("?" * len(batch))
+            rows = conn.execute(
+                f"SELECT rsid, MAX(am_pathogenicity), am_class"
+                f" FROM alphamissense_scores"
+                f" WHERE rsid IN ({placeholders}) GROUP BY rsid",
+                batch,
+            ).fetchall()
+            for rsid, score, cls in rows:
+                if score is not None:
+                    result[rsid] = (score, cls)
+        return result