Skip to content

data_qual_tests.md

Latest commit

 

History

History
82 lines (54 loc) · 3.18 KB

data_qual_tests.md

File metadata and controls

82 lines (54 loc) · 3.18 KB
Raw

Data Validation Overview

This document explains the purpose of each validation method used in this project, why it’s important, and how it fits into the overall data quality process.

1. Quick Sanitary Check (Pandas)

What it does:
Performs a fast, high-level health check on the dataset to catch obvious data quality issues before deeper analysis. It:

  • Counts missing values for each column.
  • Reports the number of duplicate rows.
  • Summarizes numeric columns for range and distribution.
  • Shows the number of unique values for categorical fields.

Why it matters:
These checks act as the first line of defense, helping identify structural issues (missing data, duplicates, anomalies) early, so you can decide whether to clean, filter, or request updated data before proceeding.

2. Pandas-Only Data Validator

What it does:
Runs rule-based validation entirely in pandas without extra dependencies. Common checks include:

  • Ensuring numeric fields contain valid numbers and meet minimum/maximum thresholds.
  • Verifying percentages are within expected bounds.
  • Checking for uniqueness and non-null constraints on key identifiers.
  • Confirming date columns are properly parsed and in the correct type.

Why it matters:
This method is fast, easy to integrate, and repeatable — making it a good option for lightweight pipelines or analysis scripts where full schema validation might be overkill.

3. Schema Validation (Pandera)

What it does:
Applies a formal schema to the dataset using Pandera. This enforces:

  • Column names, data types, and nullability.
  • Numeric constraints (e.g., values must be ≥ 0).
  • Date fields must be valid datetime64[ns] types.
  • Values must fall within defined ranges.

Why it matters:
A schema serves as a data contract. It guarantees that any dataset entering your workflow matches the expected structure, reducing the risk of downstream errors. This is especially useful in production pipelines and CI/CD workflows.

4. Unit Tests (Pure Python Asserts)

What it does:
Implements data quality checks as standard Python assert statements, which can be run in testing frameworks like pytest. Examples:

  • No negative sales amounts.
  • Unique OrderIDs.
  • Profit margins within acceptable bounds.

Why it matters:
Treating data quality as part of the testing process allows you to block bad data from being deployed or used in reports. This approach integrates naturally into software development workflows and supports continuous integration.

How They Work Together

  • Quick Sanitary Check: Initial triage to spot glaring problems quickly.
  • Pandas-Only Validator: More systematic rules without adding dependencies.
  • Schema Validation: Formal, enforceable rules for production data integrity.
  • Unit Tests: Ongoing automated verification as part of your development lifecycle.

Best Practices

  • Always coerce data types before running validations (e.g., convert "35%" to 0.35).
  • Run Quick Sanitary Check before any transformation to understand data health.
  • Use Schema Validation in production to enforce data contracts.
  • Integrate Unit Tests into CI/CD to catch issues early.