Paper finds widespread methodological failures in agentic data science systems
This arXiv paper evaluates "agentic" data science systems beyond accuracy using (1) a large-scale analysis across 220 data science tasks, (2) a reusable test suite targeting six specific failure modes, and (3) qualitative comparisons to expert Kaggle solutions. It finds most agentic pipelines contain at least one methodological failure (e.g., data leakage, improper validation), that data leakage often correlates with higher reported performance, and that agents lack creativity and domain knowledge compared to top human practitioners.