benchmarks

2026-10-31

RESEARCH · 1 SOURCE · arXiv cs.AI

Paper (arXiv:2610.38100v1) argues CLadder-generated tests embed superficial patterns that inflate LLM causal reasoning claims

A new arXiv paper (2610.38100v1) examines the CLadder procedural generator used to benchmark formal causal reasoning in LLMs and finds that CLadder embeds highly predictive superficial patterns. The study shows simple models (e.g., random forests) can perform well after training on generated samples, that commercial LLM performance on CLadder tests is brittle, and that removing text needed for formal causal reasoning often does not reduce model accuracy, suggesting LLMs may exploit generator-specific artifacts rather than true causal reasoning.

7.0

2026-10-16

RESEARCH · 1 SOURCE · arXiv cs.AI

ARIA: runtime inspection and amendment framework plus dataset for LLM-based multi-agent systems

Researchers from LabRAI introduce ARIA, a MAS-agnostic framework and learned model for runtime inspection and amendment of LLM-based multi-agent executions. They also release the ARIA dataset—built from over 24,000 trajectories across four multi-agent frameworks, six benchmarks, and eight host models—containing 3,953 execution-verified repair pairs; the trained model shows strong post-hoc and runtime repair performance and generalizes to unseen frameworks and host models; code is on GitHub.

7.0

2026-10-15

RESEARCH · 1 SOURCE · arXiv cs.AI

Measuring inference energy efficiency of 86 open-weight LLM checkpoints across benchmarks, reasoning modes, and hardware

An arXiv preprint (arXiv:2610.14389v1) reports direct measurements of inference energy for 86 open-weight model checkpoints across seven benchmarks and six hardware platforms. The study finds large energy-efficiency variation even among similarly capable models (most configurations use a median 2.6–4.8× more energy than a Pareto-optimal alternative, with up to 27×), reasoning increases energy by ~3.7× on average, generated-token count misorders 18–24% of model pairs as an energy proxy, FP8 yields ~25% energy savings with little score change, and changing hardware largely rescales energy without altering relative model rankings; the authors recommend reporting benchmark scores together with measured energy under a specified serving regime and output budget.

7.0

2026-09-18

RESEARCH · 1 SOURCE · arXiv cs.AI

little m: an AI agent to formulate industrial process optimization models

Researchers released 'little m', an AI agent that combines a domain-specific knowledge repository with LLM-driven interaction to translate messy, multimodal industrial specifications (text and process diagrams) into mathematical optimization models. They also introduced IPC-Bench, a 50-scenario multimodal benchmark for industrial process control; evaluations (automated structural checks and double-blind human review) show little m generates substantially more semantically correct formulations than state-of-the-art LLMs, though the paper does not evaluate solver feasibility, physical validity, or closed-loop performance.

6.0

2026-08-28

RESEARCH · 1 SOURCE · Apple Machine Learning Research

Agent Seer: Synthesizing realistic agent evaluation scenarios from tool specifications

Agent Seer is a method that synthesizes realistic evaluation scenarios for AI agents by using tool specifications—function names, natural-language descriptions, and typed parameter schemas—rather than relying on hand-crafted scenarios or live tool execution. The approach aims to scale scenario generation across tool ecosystems and avoid static benchmarks that cannot keep up with evolving APIs.

7.0