Mix, Don't Tune (ICLR 2026): bilingual pre‑training outperforms hyperparameter tuning for data‑constrained languages
The ICLR 2026 paper "Mix, Don't Tune" compares aggressive hyperparameter tuning versus mixing in a high‑resource auxiliary language (English) when pre-training models on a low‑resource target (Arabic). Across ~1,000 pre-training runs and four model scales (150M–1.43B), the authors find mixing yields larger gains than hyperparameter tuning on validation loss and downstream tasks (mixing is roughly equivalent to adding 2–3× unique target data for validation loss and 2–13× for downstream accuracy), the advantage grows with model size, and validation loss underestimates the knowledge benefit of mixing; they recommend prioritizing mixing ratios and transferring hyperparameters via μP.