RESEARCH · RESEARCH · #2550
HyScale paper introduces HyTopK and selection replay to speed long-context learned sparse attention training
HyScale (arXiv:2610.32057v1) presents a system for scaling long-context training of learned sparse attention by treating selection outputs as index sets and introducing HyTopK (on-the-fly exact top-k) plus selection replay to avoid full score tables and repeated selection. The paper reports up to 2.07× speedup training GLM-5.1 on 1M-token sequences versus Megatron-LM on 512 H20 GPUs, and 2.01× / 1.57× speedups for the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next at 1M tokens on 32 H800 GPUs.
KEY POINTS
- HyScale (arXiv:2610.32057v1) presents a system for scaling long-context training of learned sparse attention by treating selection outputs as index sets and introducing HyTopK (on-the-fly exact top-k) plus selection replay to avoid full score tables and repeated selection.
- The paper reports up to 2.07× speedup training GLM-5.1 on 1M-token sequences versus Megatron-LM on 512 H20 GPUs, and 2.01× / 1.57× speedups for the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next at 1M tokens on 32 H800 GPUs.
- This introduces practical system techniques that cut the selection bottleneck for million-token learned sparse attention, enabling substantially faster training of very long-context language models.
WHY IT MATTERS
This introduces practical system techniques that cut the selection bottleneck for million-token learned sparse attention, enabling substantially faster training of very long-context language models.