Feature request
Add optional monitoring hooks to Tianshou's trainers that track signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — logged alongside existing training metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors already (currently for GRPO/TRL-style training, should generalize).
Motivation
Reward hacking is a common failure mode in RL training, and there's currently no standard Tianshou hook for monitoring it. This comes from work on two reward-hacking-detection benchmarks (RHOB, Flight Recorder). One relevant finding: these signals reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking").
Proposing before attempting a PR — happy to share methodology/code if there's interest.
Feature request
Add optional monitoring hooks to Tianshou's trainers that track signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — logged alongside existing training metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors already (currently for GRPO/TRL-style training, should generalize).
Motivation
Reward hacking is a common failure mode in RL training, and there's currently no standard Tianshou hook for monitoring it. This comes from work on two reward-hacking-detection benchmarks (RHOB, Flight Recorder). One relevant finding: these signals reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking").
Proposing before attempting a PR — happy to share methodology/code if there's interest.