Auralis LLM Data Pipeline

A no-nonsense, multi-stage cleaning + tokenization pipeline for LLM pretraining corpora, built for the Auralis / Helix German-primary language model. It handles German, English, and code — with domain-specific cleaning that comes from actually staring at real web-crawl garbage, not from a tutorial.

This is not "download and hope". It's a data factory: every stage is a deliberate filter, and the code path is validated by an executor (only code that compiles survives).

Pure standard library + a couple of well-known packages. No internal dependencies, no hidden config — every script reads --input and writes --output. Take what you need.

The five stages

 raw text  ─▶  1. Fast Reject  ─▶  2. Structure Clean  ─▶  3. Dedup  ─▶  4. Quality Audit  ─▶  5. Mix + Tokenize  ─▶  .bin / .idx
                                                                                                  (code: AST-validated)

#	Stage	Script	What it does
1	Fast Reject	strict_filter_pretrain.py	Cheap, aggressive rejects: too short/long, low alpha ratio, high symbol/digit/repetition ratios, too many URLs, n-gram repetition, plus opt-in drops for German web boilerplate ("zwei Klicks für mehr Datenschutz", "Bewertung verfassen", cart/checkout/shipping), adult/gambling spam, speaker-label transcripts, bibliography dumps, and Fraktur/old-print OCR.
2	Structure Clean	structure_clean_pretrain.py	Strips HTML boilerplate, navigation, list/header noise, mojibake, chat markers (<|im_start|>), OCR artifacts, TOC fragments, commercial boilerplate. Reparagraphs to a target length and scores document structure.
3	Dedup	dedup_de_fresh.py	Two-tier dedup: exact SHA1 and fuzzy near-dup via MinHash-LSH (5-shingles, 64 permutations, Jaccard ≥ 0.85), deduping a fresh crawl against one or more reference corpora.
4	Quality Audit	quality_scores.py	Scores the surviving corpus so you can see what you kept, not just what you dropped.
5	Mix + Tokenize	mix_corpora.py → tokenize_for_pretraining.py	Mixes sources at target ratios, then tokenizes with EOS-per-document, writing a flat .bin plus an .idx of [offset, length] per doc (enables clean document-boundary snapping for val splits). tokens_per_byte is measured after tokenizing, not guessed.
—	Code track	filter_code_corpus.py	For code: an py_compile gate — only code that parses under Python 3 survives — plus SHA256 exact dedup. Executor = truth.

Why it's built this way (lessons baked in)

• Domain-specific German cleaning. The boilerplate/OCR regexes are tuned on real German Common-Crawl / FineWeb2-DE garbage. Generic English cleaners miss most of it.
• Executor-validated code. Syntax-checking code with compile() removes the single biggest source of broken code corpora (Python-2 mixed in, truncated files). It checks syntax, not semantics — but that alone is a huge quality jump.
• EOS-per-document tokenization. A missing/invalid EOS silently corrupts a whole corpus (a -1 becomes a giant uint32). The tokenizer guards against it explicitly.
• Measured, not guessed. tokens_per_byte is computed post-tokenization, so your token budget math is real.
• Source-disjoint by design. Keep train / val / eval sources disjoint (not just byte-disjoint) to avoid leakage — see the Auralis source-disjoint plan for the methodology.

Repo layout

scripts/data/   the pipeline scripts (run from the repo root)
configs/        data_paths.example.yaml  (copy → data_paths.yaml, set your data_root)

Stages 1-3 + the code track are fully standalone (Python stdlib + datasketch only) — run them directly. Stages 4-5 (audit / mix / tokenize) use a small shared helper and a configs/data_paths.yaml, so run those as modules from the repo root (python -m ...).

Quick start

pip install -r requirements.txt

# --- standalone cleaning stages (no config needed) ---

# 1. Fast reject (German, with the opt-in boilerplate/OCR drops)
python scripts/data/strict_filter_pretrain.py --input raw_de.jsonl --output kept_de.jsonl \
    --language german --v3-structure-filters --drop-web-boilerplate \
    --drop-commercial-boilerplate --drop-adult-gambling-spam --drop-old-ocr

# 2. Structure clean
python scripts/data/structure_clean_pretrain.py --input kept_de.jsonl --output-jsonl clean_de.jsonl

# 3. Dedup the fresh crawl against one or more reference corpora
python scripts/data/dedup_de_fresh.py --fresh clean_de.jsonl --ref reference_corpus.txt --out dedup_de.jsonl

# Code track: keep only code that compiles (Python 3) + SHA256 dedup
python scripts/data/filter_code_corpus.py --inp raw_code.jsonl --out clean_code.jsonl

# --- integrated stages (copy configs/data_paths.example.yaml → configs/data_paths.yaml first) ---

# 4. Audit what you kept
python -m scripts.data.quality_scores --help

# 5. Mix sources at target ratios, then tokenize to .bin/.idx (bring your own SentencePiece model)
python -m scripts.data.mix_corpora --help              # source ratios
python -m scripts.data.tokenize_for_pretraining --help # EOS-per-doc, writes .bin + .idx

Every script has --help with its full option set and sensible defaults.

Requirements

• Python 3.10+
• datasketch (MinHash-LSH dedup)
• sentencepiece (tokenization)
• numpy (binary token arrays)

License

Apache-2.0. Built as part of the Auralis / Helix project. If it saves you a week of writing data-cleaning regexes, that's the point.