FP8

2026-10-15

MODELS · 2 SOURCES · Hugging Face · The Decoder

Black Forest Labs releases FLUX.3 Image Klein with bounding-box layout control in Diffusers

Black Forest Labs published FLUX.3 Image Klein to the Hugging Face Hub and ships open checkpoints that run in Hugging Face Diffusers. The model supports text-to-image and multi-reference image-to-image generation, adds bounding-box layout control for region placement and image edits, includes a distilled checkpoint (1024×1024 ~2s on an H100, 16 steps by default), quantized single-file transformer weights (bf16, fp8r, nvfp4), and LoRA/DreamBooth training scripts.

8.0

2026-10-15

RESEARCH · 1 SOURCE · arXiv cs.AI

Echo: error‑compensated 4‑bit all-reduce for tensor‑parallel LLM training and inference

The arXiv paper introduces Echo, a 4‑bit all‑reduce protocol for tensor-parallel LLM training and inference that moves 4‑bit data but sums in high precision and feeds per-call rounding errors forward so they cancel rather than accumulate. The authors prove the accumulated error is depth‑independent and report that Echo is 2.09× faster than 16‑bit NCCL all‑reduce on 1 GB messages on A100s, speeds time‑to‑first‑token and training step latency by up to ~1.16×/1.09× on idle interconnects (and larger gains under background traffic), keeps average accuracy within 0.4 points on Qwen3‑8B, and yields <0.9% higher final loss after 10B‑token pretraining — with no FP8/FP4 hardware required.

7.0

2026-10-06

CODING · 1 SOURCE · NVIDIA Developer

Megatron Core PR #7262 adds workflow for bitwise-deterministic pretraining

Megatron-LM PR #7262 introduces a workflow and tooling in NVIDIA Megatron Core to achieve and validate bitwise-deterministic large-scale pretraining, including ordered per-rank tensor fingerprinting for offline divergence localization. Kernel-level fixes (e.g., private output slots for grouped-GEMM writers and a fixed reduction order), recipe/kernel/module validation across parallelism settings (including FP8/FP4), and optimizations on Nemotron workloads reduced the determinism overhead to about 2% at 2,432 GPUs while preserving bitwise identity over 800 steps.

7.0

2026-09-15

MODELS · 1 SOURCE · arXiv cs.AI

ZGCM-1: Open 7B foundation model claiming extreme efficiency and 256K context for math and agentic search

The arXiv submission (arXiv:2609.13356v1) presents ZGCM-1, a fully open 7B dense foundation model trained from scratch with an emphasis on data, system, and algorithmic efficiency and long-context operation (up to 256K). The authors report that ZGCM-1-7B is competitive with larger models on general benchmarks and on challenging mathematical reasoning and agentic search suites, claim a ≈4.2× improvement in 16K pre-training time-to-loss, describe a training recipe (including interleaved gated sliding-window/full attention and an FP8 Muon optimizer), and open-source weights, checkpoints, code, data recipes, and W&B logs.

7.0