RESEARCH · RESEARCH · #585
AutoData: agentic search discovers improved pre-training data selection (arXiv:2609.19754v1)
The arXiv preprint introduces AutoData, an agent that searches a program space of executable selection algorithms (scoring, stratification, stochastic rules) to optimize pre-training data selection using validation feedback from a proxy model. In an overnight search AutoData found a recipe that outperforms existing human-designed curation pipelines and transfers to larger scales, improving the downstream CORE metric.
KEY POINTS
- The arXiv preprint introduces AutoData, an agent that searches a program space of executable selection algorithms (scoring, stratification, stochastic rules) to optimize pre-training data selection using validation feedback from a proxy model.
- In an overnight search AutoData found a recipe that outperforms existing human-designed curation pipelines and transfers to larger scales, improving the downstream CORE metric.
- It demonstrates that data engineering for LLM pre-training can be automated via agentic search over executable selection programs, producing recipes that transfer to larger-scale training.
WHY IT MATTERS
It demonstrates that data engineering for LLM pre-training can be automated via agentic search over executable selection programs, producing recipes that transfer to larger-scale training.