NEWS · RESEARCH · #354
Skill-based agentic evaluation for real-time data science tasks (ground-truth-as-code, arXiv:2609.16487v1)
This arXiv paper proposes a framework for evaluating data-science agents on live, continuously updated data by encoding expected answers as executable reference functions ('ground-truth-as-code') that recompute answers from current data at evaluation time, combined with a format-agnostic, factoid-level judge that decomposes responses into atomic claims and scores precision, recall, and accuracy. Validated on an internal ML skill with a synthetic database, the method improved agreement with human annotators (29% higher MCC vs. a natural-language baseline) and reduced token usage per test case by 16%, and is aimed at agents that perform multi-source integration over non-stationary data.
KEY POINTS
- This arXiv paper proposes a framework for evaluating data-science agents on live, continuously updated data by encoding expected answers as executable reference functions ('ground-truth-as-code') that recompute answers from current data at evaluation time, combined with a format-agnostic, factoid-level judge that decomposes responses into atomic claims and scores precision, recall, and accuracy.
- Validated on an internal ML skill with a synthetic database, the method improved agreement with human annotators (29% higher MCC vs.
- a natural-language baseline) and reduced token usage per test case by 16%, and is aimed at agents that perform multi-source integration over non-stationary data.
WHY IT MATTERS
It addresses a practical evaluation gap for agents that compute answers over non-stationary live data by making references executable and format-agnostic, improving automated agreement with human judgment.