I'm a Senior SDET (SDET III) at UST specializing in Data Quality Engineering and ETL Automation Testing. With 4+ years of experience validating production-scale PySpark pipelines (10M+ records/day), I sit at the intersection of data engineering and QA — catching the bugs that hide inside your data, not just your code.
In 2026, I published ValidateX — a lightweight Python data quality validation framework — to PyPI. Because after years of writing the same validation boilerplate across projects, I decided to ship it as a library instead.
pip install validatexA lightweight, production-ready data quality validation framework for Python Supports Pandas & PySpark • 25+ built-in expectations • Weighted quality scoring • Modern HTML reports
import pandas as pd
import validatex as vx
suite = (
vx.ExpectationSuite("production_data")
.add("expect_column_to_not_be_null", column="user_id")
.add("expect_column_values_to_be_unique", column="user_id")
.add("expect_column_values_to_be_between", column="age", min_value=0, max_value=150)
.add("expect_column_values_to_match_regex", column="email", regex=r"^[\w.]+@[\w]+\.\w+$")
)
result = vx.validate(df, suite)
print(result.summary()) # Data Quality Score: 97/100
result.to_html("report.html") # Beautiful dark-theme HTML reportWhy ValidateX?
| ValidateX | Great Expectations | |
|---|---|---|
| Setup | pip install → validate in 5 lines |
Multi-step setup with contexts & stores |
| Quality Score | ✅ Weighted 0–100 | ❌ |
| Severity Levels | ✅ Critical / Warning / Info | ❌ |
| CI/CD CLI | ✅ Built-in | ❌ |
| Learning Curve | Minutes | Hours to days |
┌─────────────────────────────────────────────────────────────┐
│ ETL Testing → Validate PySpark pipelines at scale │
│ Data Quality → Schema checks, SCD-2, drift detect │
│ Test Automation → Selenium + Robot Framework + pytest │
│ Open Source → Building tools the data world needs │
│ CI/CD Integration → Jenkins, GitHub Actions, Airflow │
└─────────────────────────────────────────────────────────────┘
By the numbers from my 4+ years in production:
- 🔴 60% reduction in data quality issues through automated testing frameworks
- ⚡ 40% reduction in manual reconciliation effort via Python automation
- 📊 10M+ records/day validated across PySpark ETL pipelines
- 🧪 96% code coverage on ValidateX (66 tests passing)
Data & ETL
Testing & Automation
Cloud & Storage
🧪 ValidateX — Published on PyPI
Open-source Python data quality validation framework. Pandas + PySpark support, 25+ expectations, severity scoring, HTML reports, CLI for CI/CD integration.
pip install validatex
Python ETL pipeline for Indian stock market data — bulk ingestion via CSV/Excel, OHLCV schema normalization, API constraint handling, Streamlit control layer. A hands-on data engineering project focused on ingestion, transformation, and delivery.
- 🏆 Databricks — Data Governance Fundamentals (Jan 2026)
- 🏆 Databricks — Databricks Fundamentals (Nov 2025)
- 📜 Big Data Analytics with Hadoop & Apache Spark — LinkedIn Learning (Sep 2025)
- 📜 Selenium WebDriver with Python — Udemy (Apr 2025)
- 📜 Getting Started in Test Automation Engineering — LinkedIn Learning (Apr 2025)
I write about data engineering, ETL automation, and real-world pipeline challenges on Medium.
"Bad data is worse than no data — it gives you false confidence."
That's why I build systems that catch it before it reaches your dashboards.