Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #33139

Jeffreys divergence

Related event timeline, sources and context from the news index.

EVENT TIMELINE

1

RESEARCH · 1 SOURCE · Apple Machine Learning Research

Theoretical Limits of Language Model Alignment

The paper derives information-theoretic limits on reward improvement under a fixed KL-divergence budget for KL-regularized alignment, giving a closed-form expression governed by a Jeffreys divergence term and a reformulation as a covariance under the base model that yields a practical estimator. It also shows the proxy-reward (reward hacking) gap grows with reward error and lower KL penalties, proves reward ensembling reduces reward hacking, and empirically finds best-of-N nearly reaches the theoretical KL–reward Pareto frontier while PPO and GRPO remain substantially suboptimal on safety and summarization tasks.

7.0