RELEASE · CODING · #1750
Megatron Core PR #7262 adds workflow for bitwise-deterministic pretraining
Megatron-LM PR #7262 introduces a workflow and tooling in NVIDIA Megatron Core to achieve and validate bitwise-deterministic large-scale pretraining, including ordered per-rank tensor fingerprinting for offline divergence localization. Kernel-level fixes (e.g., private output slots for grouped-GEMM writers and a fixed reduction order), recipe/kernel/module validation across parallelism settings (including FP8/FP4), and optimizations on Nemotron workloads reduced the determinism overhead to about 2% at 2,432 GPUs while preserving bitwise identity over 800 steps.
KEY POINTS
- Megatron-LM PR #7262 introduces a workflow and tooling in NVIDIA Megatron Core to achieve and validate bitwise-deterministic large-scale pretraining, including ordered per-rank tensor fingerprinting for offline divergence localization.
- Kernel-level fixes (e.g., private output slots for grouped-GEMM writers and a fixed reduction order), recipe/kernel/module validation across parallelism settings (including FP8/FP4), and optimizations on Nemotron workloads reduced the determinism overhead to about 2% at 2,432 GPUs while preserving bitwise identity over 800 steps.
- Bitwise determinism makes debugging, checkpoint-resume, and system-change validation reliable for trillion-parameter training runs, reducing costly nondeterministic failures while keeping overhead low.
WHY IT MATTERS
Bitwise determinism makes debugging, checkpoint-resume, and system-change validation reliable for trillion-parameter training runs, reducing costly nondeterministic failures while keeping overhead low.