RESEARCH · RESEARCH · #3502
Theoretical Limits of Language Model Alignment
The paper derives information-theoretic limits on reward improvement under a fixed KL-divergence budget for KL-regularized alignment, giving a closed-form expression governed by a Jeffreys divergence term and a reformulation as a covariance under the base model that yields a practical estimator. It also shows the proxy-reward (reward hacking) gap grows with reward error and lower KL penalties, proves reward ensembling reduces reward hacking, and empirically finds best-of-N nearly reaches the theoretical KL–reward Pareto frontier while PPO and GRPO remain substantially suboptimal on safety and summarization tasks.
KEY POINTS
- The paper derives information-theoretic limits on reward improvement under a fixed KL-divergence budget for KL-regularized alignment, giving a closed-form expression governed by a Jeffreys divergence term and a reformulation as a covariance under the base model that yields a practical estimator.
- It also shows the proxy-reward (reward hacking) gap grows with reward error and lower KL penalties, proves reward ensembling reduces reward hacking, and empirically finds best-of-N nearly reaches the theoretical KL–reward Pareto frontier while PPO and GRPO remain substantially suboptimal on safety and summarization tasks.
- Provides a first-principles bound and practical estimator for how much RLHF-style KL-regularized alignment can improve reward, explaining empirical behaviors (e.g., best-of-N vs PPO/GRPO) and guiding algorithmic priorities to avoid costly suboptimality and reward hacking.
WHY IT MATTERS
Provides a first-principles bound and practical estimator for how much RLHF-style KL-regularized alignment can improve reward, explaining empirical behaviors (e.g., best-of-N vs PPO/GRPO) and guiding algorithmic priorities to avoid costly suboptimality and reward hacking.