Real-Time Facial Emotion Recognition System

A production-style Python application for real-time facial emotion recognition using a custom CNN trained on FER-2013. The project includes a reusable inference layer, webcam detection, training and evaluation pipelines, session analytics, CSV export, screenshot capture, and a Streamlit dashboard suitable for portfolio demos and interview walkthroughs.

Features

• Real-time webcam face detection with OpenCV Haar Cascades
• Multi-face emotion prediction with confidence scores
• Custom TensorFlow/Keras CNN with convolution, batch normalization, pooling, dropout, dense layers, and softmax output
• Folder-based FER-2013 training pipeline with data augmentation, class balancing, early stopping, checkpointing, and learning-rate scheduling
• Evaluation workflow with classification report and confusion matrix
• Streamlit dashboard with live camera preview, emotion distribution, session metrics, and a clean light UI
• Advanced features: emotion history tracking, CSV session export, trend analytics, screenshot capture, audio alerts, and emotion-based recommendations
• Configurable through .env, CLI flags, and typed dataclasses

Tech Stack

Python 3.11+, TensorFlow/Keras, OpenCV, NumPy, pandas, matplotlib, scikit-learn, Streamlit, python-dotenv, argparse, pathlib, and the standard logging module.

Architecture

emotion-recognition-system/
|-- app/
|   |-- main.py          # CLI webcam application
|   |-- detector.py      # Face detection
|   |-- camera.py        # Camera stream abstraction
|   |-- predictor.py     # Model loading and emotion prediction
|   |-- preprocess.py    # Image preprocessing and low-light enhancement
|   |-- config.py        # Environment-backed configuration
|   `-- utils.py         # Drawing, analytics, alerts, exports
|-- models/
|   |-- train.py         # Folder-based FER-2013 training pipeline
|   |-- architecture.py  # Custom CNN
|   |-- evaluate.py      # Test metrics and confusion matrix
|   `-- saved/
|-- dataset/
|-- notebooks/
|-- tests/
|-- streamlit_app.py
|-- requirements.txt
|-- README.md
|-- .env.example
`-- .gitignore

The app layer is separated from the model layer so the same predictor, detector, preprocessing, and analytics components can be reused by the CLI app, Streamlit dashboard, tests, or future API endpoints.

Installation

cd emotion-recognition-system
python -m venv .venv
.venv\Scripts\activate
python -m pip install --upgrade pip
pip install -r requirements.txt
copy .env.example .env

On macOS/Linux, activate with:

source .venv/bin/activate

Dataset Setup

Place the FER-2013 folder dataset under dataset/:

dataset/
|-- train/
|   |-- angry/
|   |-- disgust/
|   |-- fear/
|   |-- happy/
|   |-- sad/
|   |-- surprise/
|   `-- neutral/
`-- test/
    |-- angry/
    |-- disgust/
    |-- fear/
    |-- happy/
    |-- sad/
    |-- surprise/
    `-- neutral/

The training pipeline enforces this class mapping:

0 Angry
1 Disgust
2 Fear
3 Happy
4 Sad
5 Surprise
6 Neutral

The explicit class order prevents TensorFlow's default alphabetical directory ordering from changing the model labels.

Training

python -m models.train --dataset dataset --epochs 60 --batch-size 64

Training outputs are written to models/saved/:

• emotion_cnn.keras: trained model used by inference
• labels.json: emotion labels in model output order
• training_curves.png: accuracy and loss plots

FER-2013 is noisy and class-imbalanced. A custom CNN trained from scratch typically lands below modern pretrained architectures, but it is realistic for a portfolio project and demonstrates a complete deep learning pipeline.

Evaluation

python -m models.evaluate --dataset dataset --model models/saved/emotion_cnn.keras

This prints a classification report and saves confusion_matrix.png.

Real-Time CLI Usage

python -m app.main --model models/saved/emotion_cnn.keras --export-csv --audio-alerts

Controls:

• Press q to quit
• Press s to capture a screenshot

Streamlit Dashboard

streamlit run streamlit_app.py

The dashboard provides live webcam inference, Start/Stop detection controls, session analytics, and a simple generated emotion bar graph.

Configuration

Create .env from .env.example and adjust paths or camera settings:

MODEL_PATH=models/saved/emotion_cnn.keras
CAMERA_INDEX=0
CONFIDENCE_THRESHOLD=0.35
LOW_LIGHT_GAMMA=1.35

Testing

pytest

The included tests focus on preprocessing and session analytics. Model training and camera integration are intentionally exercised through runtime commands because they depend on local hardware and dataset availability.

Screenshots

Add demo screenshots here after training and running the app:

• Streamlit dashboard: docs/screenshots/dashboard.png
• Webcam inference: docs/screenshots/realtime_cli.png
• Training curves: models/saved/training_curves.png

Resume-Ready Description

Built a real-time facial emotion recognition system using Python, TensorFlow/Keras, OpenCV, and Streamlit. Implemented a custom CNN trained on FER-2013 with folder-based data loading, augmentation, callbacks, class balancing, evaluation reports, and reusable inference components. Designed a clean live dashboard with session analytics, emotion history tracking, a focused distribution chart, and low-light handling.

Future Improvements

• Replace Haar Cascades with a lightweight DNN detector such as MediaPipe or YuNet
• Add FastAPI inference endpoints for image and video frame uploads
• Support ONNX/TFLite export for lower-latency deployment
• Add experiment tracking with MLflow or Weights & Biases
• Add Docker packaging and CI checks
• Improve calibration and temporal smoothing for more stable predictions