Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #15861

Megatron-LM

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

RESEARCH · 1 SOURCE · arXiv cs.AI

HyScale paper introduces HyTopK and selection replay to speed long-context learned sparse attention training

HyScale (arXiv:2610.32057v1) presents a system for scaling long-context training of learned sparse attention by treating selection outputs as index sets and introducing HyTopK (on-the-fly exact top-k) plus selection replay to avoid full score tables and repeated selection. The paper reports up to 2.07× speedup training GLM-5.1 on 1M-token sequences versus Megatron-LM on 512 H20 GPUs, and 2.01× / 1.57× speedups for the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next at 1M tokens on 32 H800 GPUs.

7.0

CODING · 1 SOURCE · NVIDIA Developer

Megatron Core PR #7262 adds workflow for bitwise-deterministic pretraining

Megatron-LM PR #7262 introduces a workflow and tooling in NVIDIA Megatron Core to achieve and validate bitwise-deterministic large-scale pretraining, including ordered per-rank tensor fingerprinting for offline divergence localization. Kernel-level fixes (e.g., private output slots for grouped-GEMM writers and a fixed reduction order), recipe/kernel/module validation across parallelism settings (including FP8/FP4), and optimizations on Nemotron workloads reduced the determinism overhead to about 2% at 2,432 GPUs while preserving bitwise identity over 800 steps.

7.0