Megatron Core PR #7262 adds workflow for bitwise-deterministic pretraining
Megatron-LM PR #7262 introduces a workflow and tooling in NVIDIA Megatron Core to achieve and validate bitwise-deterministic large-scale pretraining, including ordered per-rank tensor fingerprinting for offline divergence localization. Kernel-level fixes (e.g., private output slots for grouped-GEMM writers and a fixed reduction order), recipe/kernel/module validation across parallelism settings (including FP8/FP4), and optimizations on Nemotron workloads reduced the determinism overhead to about 2% at 2,432 GPUs while preserving bitwise identity over 800 steps.