Mix, Don't Tune (ICLR 2026): bilingual pre‑training outperforms hyperparameter tuning for data‑constrained languages

KEY POINTS

1. The ICLR 2026 paper "Mix, Don't Tune" compares aggressive hyperparameter tuning versus mixing in a high‑resource auxiliary language (English) when pre-training models on a low‑resource target (Arabic).
2. Across ~1,000 pre-training runs and four model scales (150M–1.43B), the authors find mixing yields larger gains than hyperparameter tuning on validation loss and downstream tasks (mixing is roughly equivalent to adding 2–3× unique target data for validation loss and 2–13× for downstream accuracy), the advantage grows with model size, and validation loss underestimates the knowledge benefit of mixing; they recommend prioritizing mixing ratios and transferring hyperparameters via μP.
3. This shows mixing in high‑resource languages is a more effective and scalable lever than costly hyperparameter searches for improving pre-training in data‑constrained languages, changing practical choices for multilingual/low‑resource model training.