Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3446

Weighted State Souping for efficient long-context retrieval (PALM @ NeurIPS 2026)

The paper proposes weighted state souping, a method that learns query-dependent weights to compose cached state encodings from state space models (SSMs). The authors report substantial improvements on long-context tasks (single- and multi-hop QA, needle-in-a-haystack retrieval, variable tracking, tool use), present a sparse variant with near-constant encoding cost, and give theoretical analysis; the work was accepted to the PALM workshop at NeurIPS 2026.

KEY POINTS

  1. The paper proposes weighted state souping, a method that learns query-dependent weights to compose cached state encodings from state space models (SSMs).
  2. The authors report substantial improvements on long-context tasks (single- and multi-hop QA, needle-in-a-haystack retrieval, variable tracking, tool use), present a sparse variant with near-constant encoding cost, and give theoretical analysis; the work was accepted to the PALM workshop at NeurIPS 2026.
  3. Weighted state souping combines reusable SSM state representations with selective, query-dependent retrieval to improve long-context reasoning while keeping encoding costs low.

WHY IT MATTERS

Weighted state souping combines reusable SSM state representations with selective, query-dependent retrieval to improve long-context reasoning while keeping encoding costs low.

SOURCES & TIMELINE

1