AgentHorizon benchmark released to evaluate agentic judges on long-horizon computer-use tasks
AgentHorizon is a new benchmark of 1,373 instruction–trajectory pairs drawn from 166 hours of human-recorded computer-use trajectories across three operating systems, with negative examples created by swapping closely related instructions. The paper (arXiv:2610.11050v1) releases three splits (AH, AH-S, AH-D) and evaluates eleven agentic judges—finding the best judge, GPT-5.5, reaches 80.9% balanced accuracy on the AH subset, tool-use helps some models but hurts open-weight models, and judges vary widely in their ability to spot hidden errors in long interaction histories.