NEWS · MODELS · #2999
Transformers' proposed MIC fixtures would have caught UMT5 future-token attention bug introduced with PyTorch SDPA
On July 30, 2026, Transformers switched T5 to use PyTorch's scaled dot-product attention (SDPA), and that change allowed the UMT5 decoder to attend to future tokens during training. The project describes Model Integration Contracts (transformers-mic): small, pinned fixtures plus runnable usage paths and recorded baselines run on cheap CPU runners; replaying MICs over commits shows the UMT5 fixture would have flagged the regression from its introduction until its fix on September 28, 2026.
KEY POINTS
- On July 30, 2026, Transformers switched T5 to use PyTorch's scaled dot-product attention (SDPA), and that change allowed the UMT5 decoder to attend to future tokens during training.
- The project describes Model Integration Contracts (transformers-mic): small, pinned fixtures plus runnable usage paths and recorded baselines run on cheap CPU runners; replaying MICs over commits shows the UMT5 fixture would have flagged the regression from its introduction until its fix on September 28, 2026.
- MIC fixtures provide a lightweight, reproducible integration test that can catch regressions (including training-time data leaks) earlier across many model types and prevent regressions from sitting in main for months.
WHY IT MATTERS
MIC fixtures provide a lightweight, reproducible integration test that can catch regressions (including training-time data leaks) earlier across many model types and prevent regressions from sitting in main for months.