RESEARCH · RESEARCH · #2485
Scaling in Games: continued pre‑training study (arXiv:2610.28698v1)
This arXiv preprint reports a controlled study of continued pre‑training (CPT) for embodied agents across 120 training configurations varying model size (0.8B–27B), data volume (50–1,600 hours), and data diversity (Minecraft-only to multi‑world corpora). The authors fit an empirical scaling law and find that more data improves in‑corpus performance, diversity aids transfer to related worlds but gives limited gains in distant worlds, larger models improve data efficiency yet can generalize worse under single‑world training, and CPT alone likely cannot close gaps in distant virtual environments.
KEY POINTS
- This arXiv preprint reports a controlled study of continued pre‑training (CPT) for embodied agents across 120 training configurations varying model size (0.8B–27B), data volume (50–1,600 hours), and data diversity (Minecraft-only to multi‑world corpora).
- The authors fit an empirical scaling law and find that more data improves in‑corpus performance, diversity aids transfer to related worlds but gives limited gains in distant worlds, larger models improve data efficiency yet can generalize worse under single‑world training, and CPT alone likely cannot close gaps in distant virtual environments.
- The paper quantifies how data volume, diversity, and model size interact for virtual-world embodied agents and shows limits of CPT, informing when scaling helps and when additional training stages are needed.
WHY IT MATTERS
The paper quantifies how data volume, diversity, and model size interact for virtual-world embodied agents and shows limits of CPT, informing when scaling helps and when additional training stages are needed.