Production-aware credit-card fraud detection. Temporal splits, cost-sensitive thresholds, calibrated probabilities, and a Streamlit dashboard for live transaction scoring.
Most fraud-detection portfolio projects load the Kaggle dataset, train XGBoost, hit "99% accuracy", and ship. That's a broken benchmark:
- 99.83% of the dataset is not fraud, so always-predict-not-fraud gives 99.83% accuracy and zero fraud caught.
- A random 70/30 split has the same fraud patterns in train and test, inflating every metric.
- F1 score doesn't care that a missed $10,000 fraud costs more than flagging a $40 grocery purchase.
This project does the things production fraud teams actually do.
Temporal splits. Trains on the earliest 60% of transactions,
validates on the next 20%, tests on the latest 20% — using the Time
column to preserve chronological order. This catches concept drift
that random splits hide.
Cost-sensitive thresholds. Each false negative costs the transaction amount. Each false positive costs a configurable customer-service fee (default $5). The decision threshold is tuned on the validation set to minimize expected cost in dollars, not to maximize F1.
Calibrated probabilities. XGBoost is overconfident by default — when it says 0.7, the actual fraud rate is rarely 70%. We post-hoc calibrate on the validation set with isotonic regression and report Brier scores on test.
Honest model progression. Three models, in order: a dummy baseline, a class-weighted logistic regression, and XGBoost. The dummy baseline exists specifically to make "99.83% accuracy" look ridiculous.
Real Kaggle credit-card fraud dataset (284,807 transactions, 0.173% fraud). Temporal 60/20/20 split. Thresholds tuned on the validation set to minimize expected dollar cost (FN = transaction amount, FP = $5).
XGBoost hyperparameters tuned via Optuna Bayesian search (50 trials) optimizing expected dollar cost on validation — not F1, not AUC.
| Model | Threshold | Recall | Precision | F1 | PR-AUC | Expected cost | Brier |
|---|---|---|---|---|---|---|---|
| Dummy (always-no) | 0.006 | 0.00 | 0.00 | 0.000 | 0.001 | $7,729 | 0.00132 |
| Logistic + weights | 0.999 | 0.76 | 0.71 | 0.736 | 0.744 | $2,753 | 0.02273 |
| XGBoost (tuned) | 0.086 | 0.80 | 0.57 | 0.667 | 0.806 | $2,623 | 0.00047 |
| XGBoost tuned + calib. | 0.783 | 0.75 | 0.90 | 0.818 | 0.806 | $2,678 | 0.00042 |
The story this benchmark tells:
-
Cost-aware tuning beats accuracy-aware tuning. Optuna optimized directly for expected dollar cost on validation. Result: tuned XGBoost cuts test-set cost by 29% vs the default XGBoost ($3,674 → $2,623) and now narrowly beats logistic regression ($2,623 vs $2,753).
-
The calibrated tuned XGBoost is the production winner. 75% recall with only 6 false positives (vs. 45 for raw tuned XGBoost, 23 for logistic). For a real bank, customer-friction from false alarms matters as much as recall — the calibrated version is the model you'd actually deploy. It also has the best Brier score (0.00042) and highest F1 (0.818).
-
Don't trust accuracy or F1 in fraud detection. The dummy baseline has 99.87% accuracy and is useless. Choosing models on F1 alone would have picked tuned + calibrated XGBoost — which is correct here, but only because we also tuned on cost. The real discipline: always evaluate fraud models on dollar cost with threshold sweeps, not single-number metrics.
-
Calibration matters separately from accuracy. XGBoost has a Brier score 50× better than logistic. That matters for any downstream use of the probability — risk scoring, reason codes, queue prioritization, step-up authentication thresholds.
Reproduce locally:
pip install -r requirements.txt
python scripts/tune_xgboost.py --trials 50 # ~5 min
python scripts/run_pipeline.py --tuned-params models/xgb_best_params.jsonFigures saved to reports/figures/.
git clone https://github.com/Abd-alrhman1/transaction-fraud-detector
cd transaction-fraud-detector
pip install -r requirements.txt
# 1) Run unit tests (no real data required)
python tests/test_core.py # expect 11/11 passing
# 2) Run the full pipeline on the bundled synthetic dataset
python scripts/run_pipeline.py --synthetic
# 3) Launch the Streamlit dashboard
streamlit run scripts/streamlit_app.pyDownload the Kaggle dataset (~144 MB) — free Kaggle account required:
# Option A: Kaggle CLI
pip install kaggle # then place ~/.kaggle/kaggle.json in your home dir
kaggle datasets download -d mlg-ulb/creditcardfraud
unzip creditcardfraud.zip -d data/
# Option B: Manual
# https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
# → Download → unzip → place creditcard.csv in data/
python scripts/run_pipeline.py --data data/creditcard.csvThe dataset is anonymized European credit-card transactions from 2013,
with PCA-transformed features V1..V28 plus raw Time and Amount columns,
and a binary Class label (1 = fraud).
Transaction (V1..V28, Time, Amount)
│
▼
┌────────────────┐
│ Model │ XGBoost trained on temporal split
│ predict_proba │ + isotonic calibration on validation
└──────┬─────────┘
│ P(fraud) ∈ [0, 1]
▼
┌────────────────┐ Threshold tuned on validation set
│ Threshold │ to minimize $ cost (FN = amount, FP = $5)
└──────┬─────────┘
│
▼
FLAG / PASS
transaction-fraud-detector/
├── src/fraud_detector/
│ ├── data.py # loader + temporal splits
│ ├── models.py # dummy / logistic / xgboost wrappers
│ ├── metrics.py # cost-sensitive evaluation
│ ├── calibration.py # reliability + isotonic wrap
│ └── plots.py # publishable matplotlib figures
├── scripts/
│ ├── run_pipeline.py # end-to-end training + eval
│ └── streamlit_app.py # live dashboard
├── tests/
│ └── test_core.py # 11 unit tests, no network
├── reports/ # generated by run_pipeline.py
│ ├── BENCHMARK.md
│ ├── results.json
│ └── figures/
└── data/ # creditcard.csv goes here (not in git)
- Adversarial validation: train a classifier to predict time period to surface features that drift between train and test.
- Anomaly detection (Isolation Forest) for unsupervised comparison.
- Feature attribution per prediction with SHAP, surfaced in the Streamlit demo.
- A small REST API around the calibrated XGBoost so the same model can be hit by the dashboard or a webhook.
Built by Abdalrhman Qasim as part of an AI/ML engineering portfolio focused on production-grade ML for the financial sector.
- Repo: github.com/Abd-alrhman1/transaction-fraud-detector
- Portfolio companion: OCI Arabic RAG Toolkit
MIT — see LICENSE.