Large-Scale E-Commerce Recommender System

A production-style recommendation engine built on the OTTO RecSys dataset (216M interactions, 1.8M products), implementing memory-efficient candidate generation, leakage-safe offline evaluation, and LightGBM LambdaRank reranking under a strict 7 GB RAM constraint.

Highlights

• Streamed the full 216M interactions across 12.9M sessions and 1.85M products in a single memory-safe, O(1)-RAM pass (line-by-line JSONL parsing with reservoir sampling) under a 7 GB RAM constraint, and drew a 250K-session (~4.2M-event, ~1.9%) dev sample from it for all downstream modeling.
• Candidate generation, temporal splitting, feature engineering, and reranker training all run on that 250K-session dev sample, not the full 216M-interaction corpus — the headline Recall@100/MRR@20 numbers below are measured on this dev-sample scale.
• Built a disk-backed sparse co-visitation pipeline achieving 80.8% Candidate Recall@100 for order prediction (on the dev sample).
• Developed a 20-feature LightGBM LambdaRank reranker, improving Orders E2E MRR@20 from 0.3100 to 0.7391 (on the dev sample).
• Designed leakage-safe temporal evaluation, deterministic candidate assembly, feature ablation experiments, and end-to-end CLI inference.

1. Architecture

The pipeline follows the classic two-stage recommendation architecture: Candidate Generation followed by Reranking.

graph TD
    subgraph Offline Context
        A[Raw Session Logs] --> B(Event History Parquet)
        B --> C[Co-visitation Graphs]
        B --> D[Global & Recent Popularity]
    end

    subgraph Inference Pipeline
        E[User Session Events] --> F[Heuristic Candidate Assembly]
        C -.-> F
        D -.-> F
        
        F -->|Top 100 Candidates| G[Feature Engineering]
        B -.-> G
        
        G -->|Candidate Features| H[LightGBM LambdaRank]
        H --> I[Top 20 Recommendations]
    end

Loading

2. Problem Statement

Modern e-commerce recommendation systems must process massive volumes of user sessions and output highly relevant predictions for subsequent interactions. In this project, the goal is to predict what items a user will click, add to cart, or order within the same session, using raw historical event logs.

This requires transitioning from simple baselines to scalable, memory-efficient candidate generation algorithms, and ultimately, building a Learning-to-Rank (LTR) model to maximize recommendation quality (measured via Recall@20 and MRR@20).

3. Dataset

The dataset consists of 216 million anonymized user interactions over a 4-week period.

• Scale: 12.9M sessions, 1.85M unique items.
• Interactions: Clicks (89.9%), Carts (7.8%), Orders (2.4%).
• Constraints: Due to the large data volume, all processing, joining, and aggregation must be aggressively partitioned to fit into limited RAM (7 GB max).

4. Pipeline

The recommendation pipeline is divided into the following stages:

1. Memory-Safe Preprocessing (src/data/): Converts massive JSONL logs into compressed Parquet files with downcasted numerical types.
2. Leakage-Safe Validation Split (src/data/temporal_split.py): Uses session-isolated graph construction and chronological validation. Historical sessions build the co-visitation graph, while held-out validation sessions are split chronologically into ranker training and final test sets.
3. Graph Candidate Generation (src/candidates/covisitation.py): Builds sparse item-to-item co-visitation graphs using memory-safe chunked pair aggregation, with objective-weighted variants evaluated as additional ranking signals.
4. Feature Engineering (src/features/feature_builder.py): Extracts 20 ranking features spanning graph statistics, candidate interactions, popularity signals, and temporal session context.
5. Learning-to-Rank (src/experiments/train_ranker.py): Trains pairwise LGBMRanker models for each objective (Clicks, Carts, Orders).

5. Memory Optimizations

Due to a strict 7 GB RAM constraint, the pipeline employs heavy optimization:

• Hierarchical Aggregation: The covisitation builder computes transition frequencies on small chunks before recursively merging them on disk.
• Lazy Evaluation: The Polars scan_parquet() engine is heavily utilized to push down filters and slice Parquet columns.
• Zero-Positive Filtering: When building ranker training data, sessions with no valid targets in the Top-100 candidates are discarded early to conserve RAM.
• Batched Feature Engineering: Generating 20 features across 103,000 sessions with 100 candidates each yields over 10 million rows. This is chunked in groups of 5,000 sessions to avoid Out-of-Memory (OOM) errors.

6. Experiments & Ablation

To prove the value of the Learning-to-Rank model, we tested baselines against the ML model and ran a feature-family ablation study for the Orders objective.

The features were grouped into:

• Temporal: time_since_last_interaction, session_duration
• Candidate Interaction: prefix_frequency, seen_in_prefix, event counts.
• Graph: Heuristic co-visitation scores and ranks.
• Session Context: Length and unique items.
• Popularity: Global fallback ranks.

7. Results

The LambdaRank LightGBM model resulted in a massive improvement over the heuristic co-visitation baselines.

Headline Metrics (Orders)

Model	E2E Recall@20	E2E MRR@20
Global Popularity	0.0521	0.0135
COV-001 Baseline	0.6375	0.3100
LTR-002 LambdaRank	0.8014	0.7391

Feature Ablation Impact

Note: Removing candidate-interaction and temporal features caused the largest performance drops, showing that session-specific behavioral signals contribute substantially more to reranking quality than graph scores alone.

8. Inference

A command-line interface is available for running inference on new sessions using the pre-trained LightGBM model.

# Example: Recommend top 20 items to purchase for a given session
python recommend.py --session test_session.json --objective orders

Session Summary
---------------
Events: 3
Unique Items: 2

Top-20 Recommendations
----------------------
Rank     Item ID       Score
1        1460571       3.5501
2        1500000       0.4908
3        109499        -2.7060
...

9. Lessons Learned

1. Feature Engineering > Model Complexity: LightGBM trained on temporal and interaction features substantially outperformed manually weighted graph heuristics, demonstrating the value of preserving behavioral signals for downstream learning-to-rank.
2. Leakage is Subtle: Preventing target leakage required explicit disjointness checks (sanity_check.py) between the history graph sessions and ranker training sessions.
3. Memory-Efficient Execution Matters End-to-End: Peak memory bottlenecks emerged during global pair aggregation and Python-native candidate assembly. Disk-backed hierarchical aggregation and batched Polars joins reduced memory usage enough to execute the complete pipeline under 7 GB RAM.

10. Setup / Reproducibility

Prerequisites

• Python 3.10+
• 7 GB RAM minimum for the development-scale pipeline
• ~30 GB free disk space
• OTTO Recommender Systems dataset

1. Clone the Repository

git clone https://github.com/ayushgupta-15/large-scale-ecommerce-recommender.git
cd large-scale-ecommerce-recommender

2. Install Dependencies

pip install -r requirements.txt

3. Prepare the Dataset

Download the OTTO Recommender Systems dataset and place the raw files under:

data/raw/
├── train.jsonl
└── test.jsonl

Raw and generated datasets are excluded from Git because of their size. Only train.jsonl is used by the pipeline below — test.jsonl isn't currently consumed by any script.

4. Run the Data Pipeline

Execute commands from the repository root, in this exact order. Each step depends on the parquet/artifact files written by the previous one:

# Reservoir-samples 250K sessions from the full 216M-interaction train.jsonl into
# data/interim/dev_sessions.parquet, while writing full-corpus stats (dataset_stats.json)
# via a single O(1)-memory streaming pass. All later steps operate on this dev sample,
# not the full corpus.
python prepare_data.py

# Leakage-safe 7-day temporal split: builds the co-visitation "historical" event pool
# from sessions entirely before the cutoff.
python src/data/temporal_split.py

# Splits each held-out validation session into a chronological prefix (features) and
# suffix (held-out ground-truth targets).
python src/data/session_splitter.py

# Further splits validation sessions chronologically into ranker-train / ranker-test.
python src/data/ranker_split.py

# Builds the COV-001 unweighted co-visitation graph from the historical event pool.
python -m src.candidates.covisitation

# Builds the COV-002 objective-weighted co-visitation graphs (one per objective).
python -m src.experiments.build_cov002 --objective clicks
python -m src.experiments.build_cov002 --objective carts
python -m src.experiments.build_cov002 --objective orders

# Extracts the 20 ranking features for the ranker train/test splits (one per objective).
python -m src.features.feature_builder clicks
python -m src.features.feature_builder carts
python -m src.features.feature_builder orders

5. Train and Export the Ranker

# Trains and evaluates COV-001/COV-002/LTR-001/LTR-002 for all three objectives,
# printing the Recall@20/MRR@20 comparison table.
python src/experiments/train_ranker.py

# Fits and saves the LambdaRank model + feature schema used by recommend.py
# (defaults to the "orders" objective).
python -m src.experiments.export_model

6. Run Inference

python recommend.py --session test_session.json --objective orders

This full sequence — steps 1 through 6 — was verified end-to-end from a clean checkout (only a synthetic train.jsonl present, no pre-existing data/interim, data/processed, or artifacts) and every command completed successfully in order.