SkillShift-Bench: benchmark for skill learning under open-environment shifts
SkillShift-Bench is a new benchmark and suite of three self-hosted web environments (with code and a leaderboard) that evaluates skill learning for web agents under controlled Contextual, Perceptual, and Execution shifts. Experiments show systematic robustness degradation — lower success, higher interaction cost, and more skill failures (Execution Shift hardest) — and the paper frames shifted skill execution as a nonstationary local executability problem, offering a dynamic-regret analysis and a test-time adaptation method to improve execution without relearning skills.