RESEARCH · RESEARCH · #3792
Long-context degradation is a sampling problem; top-nsigma prevents collapse at high temperature
An arXiv paper (arXiv:2611.22148v1) tests 29 open-weight models (1B–122B) on ten long-context benchmarks including retrieval out to one million tokens and finds that samplers which keep their candidate pool small as temperature rises (notably top-nsigma) retain accuracy, while samplers that admit many low-probability tokens (top-p, top-k, Top-H) collapse at high temperatures. The authors report that pool size predicts judged quality (Spearman −0.77), give examples on Llama-3.1-8B and Qwen3.5 (e.g., top-p/top-k/Top-H solve 0/100 GitHub issues at temperature 2 vs. top-nsigma 46–62) and release code and data.
KEY POINTS
- An arXiv paper (arXiv:2611.22148v1) tests 29 open-weight models (1B–122B) on ten long-context benchmarks including retrieval out to one million tokens and finds that samplers which keep their candidate pool small as temperature rises (notably top-nsigma) retain accuracy, while samplers that admit many low-probability tokens (top-p, top-k, Top-H) collapse at high temperatures.
- The authors report that pool size predicts judged quality (Spearman −0.77), give examples on Llama-3.1-8B and Qwen3.5 (e.g., top-p/top-k/Top-H solve 0/100 GitHub issues at temperature 2 vs.
- top-nsigma 46–62) and release code and data.
WHY IT MATTERS
This shows that sampler design—particularly controlling candidate-pool size—can be as decisive as model training for preserving accurate long, agentic, or high-temperature generation, changing how practitioners should configure open models for long-context tasks.