ATML_UniTN

Project Overview

This project studies selective classification under simulated label missingness. The goal is to evaluate whether missingness information can improve abstention decisions compared with standard confidence-based and margin-based rejection.

The experiments consider three missingness regimes:

• MCAR: Missing Completely At Random
• MAR: Missing At Random
• MNAR: Missing Not At Random

The final project supports two evaluation protocols:

• clean: missingness is simulated only on training labels. Validation/calibration and test/evaluation labels remain fully observed.
• propagated_observed: the same missingness mechanism is propagated to validation/calibration and test labels. Threshold calibration is performed only on observed validation rows, and risk/coverage/AURC evaluation is performed only on observed test rows. True labels are retained internally only for controlled metric computation.

The original clean protocol is preserved, while the propagated-observed protocol is included as an additional evaluation protocol.

Methods

The final experimental setup compares six methods:

• baseline_a: extended label-space baseline using the missing-label token
• baseline_b_confidence: confidence-based abstention
• baseline_b_margin: margin-based abstention
• method_c: missingness-aware two-head model with scalar score
• method_c_tuned: tuned two-head Method C variant
• method_c_dual_threshold: two-head Method C variant with separate confidence and missingness thresholds

The Method C family is evaluated to test whether the learned missingness signal adds abstention-useful information beyond standard classifier uncertainty.

Evaluation

Evaluation is based on:

• achieved coverage
• selective risk
• risk--coverage curves
• AURC, the area under the risk--coverage curve
• coverage deviation from the target coverage
• post-hoc Method C missingness-signal diagnostics

Lower AURC indicates a better risk--coverage trade-off.

In the clean protocol, thresholds are calibrated and evaluated using fully observed validation and test labels.

In the propagated-observed protocol, thresholds are calibrated only on observed validation rows, and final risk/coverage/AURC metrics are computed only on observed test rows.

The target coverages are:

0.70, 0.75, 0.80, 0.85, 0.90, 0.95, 0.99

Key Results

The final clean evaluation grid includes 2 datasets, 3 split seeds, 6 methods, 36 aggregated experimental conditions, 648 method-runs, and 4,536 risk-coverage operating points.

The main hypothesis was that label-missingness information could improve abstention under MNAR missingness. The final results do not provide robust support for this hypothesis: standard confidence- and margin-based abstention remained stronger and more stable than the missingness-aware Method C family.

Regime	Method C-family wins vs best standard baseline	Mean AURC gap	Interpretation
MCAR	1 / 4	+0.00033	No expected gain from missingness
MAR	1 / 8	+0.00228	No robust gain
MNAR	3 / 24	+0.00164	Weak support at most

Positive AURC gaps mean that the best Method C-family variant had higher AURC than the best standard confidence/margin baseline.

By individual method, the 36 condition winners were concentrated in the standard abstention baselines: confidence rejection won 20 cells and margin rejection won 10. The remaining wins were isolated: method_c_tuned won 3 cells, method_c won 2, baseline_a won 1, and method_c_dual_threshold won none.

The main diagnostic finding is that the missingness head often learned the simulated missingness process, but usually did not add error-predictive information beyond standard uncertainty. In other words, learning where labels are missing is not necessarily equivalent to learning where the classifier will make mistakes.

Representative Risk-Coverage Curve

In this high-severity MNAR setting, Method C-family variants are locally competitive at some high-coverage operating points, but confidence and margin rejection remain competitive across the full curve.

Repository Structure

src/abstention_mnar/
  data/           dataset loading and preprocessing
  missingness/    MCAR, MAR, and MNAR missingness simulation
  methods/        abstention scoring rules and method definitions
  models/         tabular MLP and two-head architectures
  training/       training loops and early stopping
  eval/           coverage, selective risk, and AURC metrics
  diagnostics/    Method C missingness-signal diagnostics
  plots/          risk-coverage and AURC visualizations

scripts/
  run_grid.py             run the full experimental grid
  aggregate_results.py    aggregate seed-level outputs
  make_plots.py           generate final plots

tests/
  unit and integration tests

Setup

Create and activate a virtual environment, then install dependencies and the package.

python -m venv .venv

# Windows:
.venv\Scripts\activate

# macOS/Linux:
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

Running Experiments

Run the Full Final Grid

python scripts/run_grid.py --eval-protocol clean
python scripts/run_grid.py --eval-protocol propagated_observed

Together, these two commands execute all combinations of:

• datasets: rice, dry_bean
• split seeds: 0, 1, 2
• missingness regimes: MCAR, MAR, MNAR
• missingness rates: 0.10, 0.50
• missingness strengths for MAR/MNAR: 0.5, 1.5
• MNAR mechanisms: reference_margin, label_class_bias, region_label_interaction
• methods: 6 total
• target coverages: 7 total

The final grid contains two evaluation protocols: clean and propagated_observed. The runner executes one evaluation protocol per invocation. Running without --eval-protocol defaults to clean, so the full final grid requires running both commands above before aggregation.

Per protocol:

Seed-level conditions: 108
Method-runs: 648
Operating points: 4536
Aggregated condition cells: 36

Across both protocols:

Seed-level conditions: 216
Method-runs: 1296
Operating points: 9072
Aggregated condition cells: 72

Run a Small Subset

Useful for debugging:

python scripts/run_grid.py --dataset rice --seed 0 --limit 8

Resume Behavior

The experiment runner is resume-safe. Completed runs are skipped automatically.

To force re-running completed runs:

python scripts/run_grid.py --force

Aggregating Results

After experiments finish, aggregate results across seeds:

python scripts/aggregate_results.py

Generating Plots

Generate risk--coverage curves and AURC summary plots:

python scripts/make_plots.py

Output Structure

All outputs are stored under:

outputs/results/

The current outputs under outputs/ are the source of truth for the final report. Old outputs should not be used to update report numbers. Codex should inspect existing outputs only to verify reported values, not rerun the full experimental grid unless explicitly requested.

Seed-level results:

outputs/results/seed_level/
    aurc.csv
    operating_points.csv

Aggregated results:

outputs/results/aggregated/
    aurc_agg.csv
    operating_points_agg.csv
    method_c_diagnostics_agg.csv
    method_c_pmax_qmiss_bins_agg.csv
    method_c_qmiss_quantiles_agg.csv

Report tables:

outputs/results/reports/
    aurc_report.csv
    operating_points_report.csv

Plots:

outputs/results/plots/
    risk_coverage/
    aurc_summary/

Expected Sizes

After running the full final grid across both protocols, the expected seed-level files are:

aurc.csv              -> 1296 rows
operating_points.csv -> 9072 rows

After aggregation across seeds, the expected files are:

aurc_agg.csv              -> 432 rows
operating_points_agg.csv -> 3024 rows

At the condition-cell level, the final setup contains:

36 aggregated condition cells per protocol 72 aggregated condition cells across both protocols

These values are useful sanity checks for verifying that the full final grid has completed.

Full Pipeline

Run the complete workflow:

python scripts/run_grid.py --eval-protocol clean
python scripts/run_grid.py --eval-protocol propagated_observed
python scripts/aggregate_results.py
python scripts/make_plots.py

Notes

• The main research question is whether missingness-aware abstention improves over confidence-based and margin-based rejection.
• MNAR is the central regime, but MCAR and MAR are included as controls.
• The final setup includes three MNAR mechanisms to avoid evaluating Method C only under confidence-aligned missingness.
• The final comparison uses both the clean and propagated_observed protocols.
• The propagated-observed protocol narrows the average Method C-family vs Baseline B gap, especially under MAR/MNAR, but does not make Method C robustly better.
• Baseline B remains the strongest and most stable family overall.
• Method C diagnostics are used to distinguish between learning the missingness process and learning an abstention-useful error signal.
• The entropy score is implemented as an auxiliary option, but the final reported grid uses confidence and margin as the two standard Baseline B variants.