RESEARCH · RESEARCH · #2614
SkillShift-Bench: benchmark for skill learning under open-environment shifts
SkillShift-Bench is a new benchmark and suite of three self-hosted web environments (with code and a leaderboard) that evaluates skill learning for web agents under controlled Contextual, Perceptual, and Execution shifts. Experiments show systematic robustness degradation — lower success, higher interaction cost, and more skill failures (Execution Shift hardest) — and the paper frames shifted skill execution as a nonstationary local executability problem, offering a dynamic-regret analysis and a test-time adaptation method to improve execution without relearning skills.
KEY POINTS
- SkillShift-Bench is a new benchmark and suite of three self-hosted web environments (with code and a leaderboard) that evaluates skill learning for web agents under controlled Contextual, Perceptual, and Execution shifts.
- Experiments show systematic robustness degradation — lower success, higher interaction cost, and more skill failures (Execution Shift hardest) — and the paper frames shifted skill execution as a nonstationary local executability problem, offering a dynamic-regret analysis and a test-time adaptation method to improve execution without relearning skills.
- Provides the first controlled benchmark revealing how common environment shifts degrade learned-web-agent skills and offers an adaptation approach and theoretical framing to address shifted skill execution.
WHY IT MATTERS
Provides the first controlled benchmark revealing how common environment shifts degrade learned-web-agent skills and offers an adaptation approach and theoretical framing to address shifted skill execution.