LLM Grading with Execution Feedback

Execution-grounded code evaluation pipeline using Qwen — grades student code based on what it actually does at runtime.

Overview

This project evaluates student programming solutions by combining source code, sandboxed execution, and runtime evidence (stdout, stderr, exit codes). The LLM generates structured JSON grading feedback grounded in observed behavior, not static inspection alone.

Impact

• Three evaluation conditions: code only, code + test results, code + failure traces
• Consistency analysis across multiple grading runs
• Sandboxed execution with timeout detection

Architecture

flowchart LR
    Code[Student Code] --> Sandbox[Sandbox Executor]
    Sandbox --> Signals[Runtime Signals]
    Signals --> Prompt[Prompt Builder]
    Prompt --> Qwen[Qwen LLM Judge]
    Qwen --> Grade[Structured JSON Grade]

Loading

Features

• Sandboxed execution of student submissions
• Runtime signal collection: stdout, stderr, exit code, timeouts
• Structured JSON grading output
• Multi-run consistency analysis

Quick Start

conda create -n qwen-grade python=3.10
conda activate qwen-grade
pip install -r requirements.txt
python run_problem.py problems/longest_consecutive

Project Structure

grader/
  sandbox.py            # Safe code execution
  llm_judge.py          # Qwen grading interface
  build_prompts.py      # Prompt construction
  parse_grades.py       # JSON grade parsing
  consistency.py        # Multi-run stability analysis

problems/               # Datasets and test cases
results/                # Raw outputs, parsed grades, metrics

Author

Jyotiradityasinh Chauhan

Portfolio · LinkedIn · GitHub