Echo: error‑compensated 4‑bit all-reduce for tensor‑parallel LLM training and inference
The arXiv paper introduces Echo, a 4‑bit all‑reduce protocol for tensor-parallel LLM training and inference that moves 4‑bit data but sums in high precision and feeds per-call rounding errors forward so they cancel rather than accumulate. The authors prove the accumulated error is depth‑independent and report that Echo is 2.09× faster than 16‑bit NCCL all‑reduce on 1 GB messages on A100s, speeds time‑to‑first‑token and training step latency by up to ~1.16×/1.09× on idle interconnects (and larger gains under background traffic), keeps average accuracy within 0.4 points on Qwen3‑8B, and yields <0.9% higher final loss after 10B‑token pretraining — with no FP8/FP4 hardware required.