RESEARCH · RESEARCH · #1837
Work2Skill: framework, dataset (WorkDemo-350) and method for reusable GUI agent skills (arXiv:2610.12984v1)
The paper introduces Work2Skill, a framework for capturing human desktop GUI experience and inducing reusable agent skills. It presents WorkRecorder for synchronized multimodal collection (screen, mouse/keyboard, accessibility tree), the WorkDemo-350 dataset of 350 screened demonstrations, and an agentic skill-induction method that improves execution model scores versus a No Skill baseline (mean gains 4.67–10.54 on MyPCBench Bounded Action and 3.11–7.50 on a 286-task OSWorld-Verified subset); code and data are on GitHub.
KEY POINTS
- The paper introduces Work2Skill, a framework for capturing human desktop GUI experience and inducing reusable agent skills.
- It presents WorkRecorder for synchronized multimodal collection (screen, mouse/keyboard, accessibility tree), the WorkDemo-350 dataset of 350 screened demonstrations, and an agentic skill-induction method that improves execution model scores versus a No Skill baseline (mean gains 4.67–10.54 on MyPCBench Bounded Action and 3.11–7.50 on a 286-task OSWorld-Verified subset); code and data are on GitHub.
- Capturing synchronized human GUI demonstrations and inducing reusable skills helps agents generalize workflow rules across unfamiliar software, improving automated GUI task success and evaluation scores.
WHY IT MATTERS
Capturing synchronized human GUI demonstrations and inducing reusable skills helps agents generalize workflow rules across unfamiliar software, improving automated GUI task success and evaluation scores.