Skip to content

06-data-quality.md

Latest commit

 

History

History
35 lines (25 loc) · 1.33 KB

06-data-quality.md

File metadata and controls

35 lines (25 loc) · 1.33 KB
Raw

06 – Data Quality

Data Quality (DQ) is a first-class concern in this platform.

6.1 DQ dimensions

  1. Completeness – required fields present.
  2. Validity – values within expected ranges and code sets.
  3. Uniqueness – no duplicate transaction_id for a given source.
  4. Timeliness – events arrive within expected time windows.
  5. Consistency – cross-table relationships hold.

6.2 Example rules

  • transaction_id must be unique per source and day.
  • amount must be ≥ 0 and ≤ configured upper bound per product.
  • event_timestamp must be within ±3 days of load date (configurable).
  • patient_id must exist in patient reference tables.
  • Mandatory compliance flags must be derived for each row.

6.3 Implementation

  • Streaming DQ in Synapse Spark Streaming:
    • Invalid records are written to DLQ tables with error codes.
  • Batch DQ in Synapse / Snowflake:
    • Rule bundles executed as SQL; results written to DQ summary tables.
  • DQ metrics exported to Log Analytics and surfaced in dashboards.

6.4 SLI/SLO examples

  • DQ Pass Rate (critical rules) – target ≥ 97%.
  • DLQ Rate – target < 0.5% of incoming records.
  • Schema compatibility – 100% of events must conform to contracts or be rejected explicitly.

Breaches feed alerts to on-call engineers and data stewards.