RESEARCH · RESEARCH · #1766
AgentHorizon benchmark released to evaluate agentic judges on long-horizon computer-use tasks
AgentHorizon is a new benchmark of 1,373 instruction–trajectory pairs drawn from 166 hours of human-recorded computer-use trajectories across three operating systems, with negative examples created by swapping closely related instructions. The paper (arXiv:2610.11050v1) releases three splits (AH, AH-S, AH-D) and evaluates eleven agentic judges—finding the best judge, GPT-5.5, reaches 80.9% balanced accuracy on the AH subset, tool-use helps some models but hurts open-weight models, and judges vary widely in their ability to spot hidden errors in long interaction histories.
KEY POINTS
- AgentHorizon is a new benchmark of 1,373 instruction–trajectory pairs drawn from 166 hours of human-recorded computer-use trajectories across three operating systems, with negative examples created by swapping closely related instructions.
- The paper (arXiv:2610.11050v1) releases three splits (AH, AH-S, AH-D) and evaluates eleven agentic judges—finding the best judge, GPT-5.5, reaches 80.9% balanced accuracy on the AH subset, tool-use helps some models but hurts open-weight models, and judges vary widely in their ability to spot hidden errors in long interaction histories.
- Automated judges are being used to train and evaluate agentic systems; AgentHorizon exposes how current judges struggle to verify long, multi-application trajectories and provides a benchmark to drive better verification methods.
WHY IT MATTERS
Automated judges are being used to train and evaluate agentic systems; AgentHorizon exposes how current judges struggle to verify long, multi-application trajectories and provides a benchmark to drive better verification methods.