MLOps CI/CD Templates

GitHub Actions + Vertex AI ML pipelines — shadow deploys, rollback, audit trail.

CI/CD Flow

Push to main
     │
     ▼
[ml_ci.yml] Run tests → Evaluate model → Gate on accuracy threshold
     │ (pass)
     ▼
[ml_cd.yml] Shadow deploy → Compare with production
     │ (agreement ≥ 80%)
     ▼
Canary deploy (10% traffic) → Monitor → Full rollout
     │ (any failure)
     ▼
Auto-rollback to previous stable version

Shadow Deploy Pattern

The new model runs alongside production but responses are not served to users. Outputs are compared; if agreement ≥ 80%, the model is promoted to canary.

This catches regressions that eval metrics miss (distribution shift, format changes).

Quick Start

pip install -r requirements.txt

# Run training pipeline (mock mode)
MOCK_MODE=true python pipelines/train_pipeline.py

# Run evaluation + gate
MOCK_MODE=true python scripts/evaluate_and_gate.py

# Run shadow deploy comparison
MOCK_MODE=true python scripts/shadow_deploy.py

# Simulate rollback
MOCK_MODE=true python scripts/rollback.py

# Log audit event
MOCK_MODE=true python scripts/audit_trail.py --event deploy --stage canary

Rollback Strategy

Rollback is triggered automatically if:

• Accuracy drops more than accuracy_drop_threshold (default 5%) vs baseline
• Shadow agreement rate < canary_min_agreement_rate (default 80%)
• Any deployment step fails in CI/CD

Previous model version is promoted back to 100% traffic via scripts/rollback.py.

Audit Trail

Every deployment event (train, eval, shadow, canary, rollback) is logged to:

• Live: BigQuery table mlops_audit.deployment_events
• Mock/Local: deployment_audit.jsonl

Configuration

Edit config/pipeline_config.yaml to set thresholds, GCP project ID, dataset URIs.