RESEARCH · RESEARCH · #3446
Weighted State Souping for efficient long-context retrieval (PALM @ NeurIPS 2026)
The paper proposes weighted state souping, a method that learns query-dependent weights to compose cached state encodings from state space models (SSMs). The authors report substantial improvements on long-context tasks (single- and multi-hop QA, needle-in-a-haystack retrieval, variable tracking, tool use), present a sparse variant with near-constant encoding cost, and give theoretical analysis; the work was accepted to the PALM workshop at NeurIPS 2026.
KEY POINTS
- The paper proposes weighted state souping, a method that learns query-dependent weights to compose cached state encodings from state space models (SSMs).
- The authors report substantial improvements on long-context tasks (single- and multi-hop QA, needle-in-a-haystack retrieval, variable tracking, tool use), present a sparse variant with near-constant encoding cost, and give theoretical analysis; the work was accepted to the PALM workshop at NeurIPS 2026.
- Weighted state souping combines reusable SSM state representations with selective, query-dependent retrieval to improve long-context reasoning while keeping encoding costs low.
WHY IT MATTERS
Weighted state souping combines reusable SSM state representations with selective, query-dependent retrieval to improve long-context reasoning while keeping encoding costs low.