RESEARCH · RESEARCH · #3029
StateBench: benchmark for long-context state tracking in language models
StateBench is a new benchmark that measures how well language models track evolving entity state across long documents, organized into four difficulty levels (r-trivial, aperiodic, soluble, general finite) corresponding to increasingly complex kinds of variable assignment. The authors report that recurrent and hybrid architectures (e.g., Gated DeltaNet) generalize farther than transformers on these tasks, while leading frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol/Luna still break down on the hardest, long-dependency cases.
KEY POINTS
- StateBench is a new benchmark that measures how well language models track evolving entity state across long documents, organized into four difficulty levels (r-trivial, aperiodic, soluble, general finite) corresponding to increasingly complex kinds of variable assignment.
- The authors report that recurrent and hybrid architectures (e.g., Gated DeltaNet) generalize farther than transformers on these tasks, while leading frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol/Luna still break down on the hardest, long-dependency cases.
- This matters because StateBench exposes practical long-context failure modes and provides a structured hierarchy to compare architectures (transformer vs recurrent/hybrid) on state-tracking tasks relevant to code and narrative understanding.
WHY IT MATTERS
This matters because StateBench exposes practical long-context failure modes and provides a structured hierarchy to compare architectures (transformer vs recurrent/hybrid) on state-tracking tasks relevant to code and narrative understanding.