HyScale paper introduces HyTopK and selection replay to speed long-context learned sparse attention training
HyScale (arXiv:2610.32057v1) presents a system for scaling long-context training of learned sparse attention by treating selection outputs as index sets and introducing HyTopK (on-the-fly exact top-k) plus selection replay to avoid full score tables and repeated selection. The paper reports up to 2.07× speedup training GLM-5.1 on 1M-token sequences versus Megatron-LM on 512 H20 GPUs, and 2.01× / 1.57× speedups for the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next at 1M tokens on 32 H800 GPUs.