Co-Designing Sparse Attention: indexer and fused Top-k optimizations for long-context inference
A technical primer describes how to co-design model-level sparse attention and GPU kernels to make long-context transformer inference efficient. It analyzes the indexer (a batched matmul with a fused epilogue), Top-k selection, head/config knobs (KH, Hsz, query heads), and kernel-level optimizations (FP16/FP8, fused Top-k) and reports measured gains such as a 2.1x–2.4x speedup from reusing previous-step selections and a fused Top-k implementation.