RideWay benchmark and Efficiency Utility metric for tool-using agents (arXiv:2609.17985v1)
The paper introduces RideWay, an efficiency-focused benchmark for ridehailing agents in a stateful tool-calling environment, and proposes Efficiency Utility, a success-gated metric that penalizes excess tool calls and user-facing turns relative to task-specific reference effort. Evaluated across 58 tasks and 24 models, the fitted human-calibrated penalty for extra turns is about twice that for extra tool calls; Efficiency Utility matches held-out human preferences with 78.7% accuracy overall (90.6% when trajectories differ in turns) but is at chance when comparisons differ only in tool-call counts, highlighting limits of count-based tool-use evaluation.