RESEARCH · RESEARCH · #2014
ACED-Bench: neuro-symbolic benchmark to evaluate how LLM agents acquire and act on causal evidence
The paper introduces ACED-BENCH, a neuro-symbolic benchmark that separates natural-language scenario text from hidden probabilistic graphical models to test whether LLM-based agents can design experiments, interpret finite evidence, and make causal decisions. The main leaderboard ACED-CORE contains 180 tasks (60 set-valued ancestry/descendancy questions and 120 constrained causal-decision tasks) and ACED-QUAL is a 150-question yes/no qualification; the authors evaluate seven contemporary and two earlier LLMs across three active agent designs and use an evidence-ledger analysis to localize investigation failures.
KEY POINTS
- The paper introduces ACED-BENCH, a neuro-symbolic benchmark that separates natural-language scenario text from hidden probabilistic graphical models to test whether LLM-based agents can design experiments, interpret finite evidence, and make causal decisions.
- The main leaderboard ACED-CORE contains 180 tasks (60 set-valued ancestry/descendancy questions and 120 constrained causal-decision tasks) and ACED-QUAL is a 150-question yes/no qualification; the authors evaluate seven contemporary and two earlier LLMs across three active agent designs and use an evidence-ledger analysis to localize investigation failures.
- Provides a scalable, precise benchmark for measuring whether LLM agents can actively design experiments and reason about causal evidence, highlighting concrete failure modes that matter for AI-assisted science.
WHY IT MATTERS
Provides a scalable, precise benchmark for measuring whether LLM agents can actively design experiments and reason about causal evidence, highlighting concrete failure modes that matter for AI-assisted science.