RESEARCH · RESEARCH · #3805
SkillBuffBench: benchmark paper for measuring agent Skill utility across harnesses and models
The paper introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much file-based Agent Skills help across three Skill-access modes (no Skill, curated Skill, and automatic invocation from an installed library), memory on/off, four agent harnesses and multiple models (proprietary state-of-the-art models from Anthropic, Google and OpenAI and several open-weight Chinese models). The design emphasizes deterministic verification, paired task-level bootstrap uncertainty, and separate reporting of cost, agent rounds and wall-clock time; the paper describes the benchmark and tasks (currently in materials science and biology) and notes that evaluation runs are in progress with results to be added later.
KEY POINTS
- The paper introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much file-based Agent Skills help across three Skill-access modes (no Skill, curated Skill, and automatic invocation from an installed library), memory on/off, four agent harnesses and multiple models (proprietary state-of-the-art models from Anthropic, Google and OpenAI and several open-weight Chinese models).
- The design emphasizes deterministic verification, paired task-level bootstrap uncertainty, and separate reporting of cost, agent rounds and wall-clock time; the paper describes the benchmark and tasks (currently in materials science and biology) and notes that evaluation runs are in progress with results to be added later.
- This matters because it evaluates Skill benefits under realistic deployment conditions (installed libraries, memory, multiple harnesses/models) instead of idealized, hand-picked settings, enabling more generalizable comparisons of skill-enabled agents.
WHY IT MATTERS
This matters because it evaluates Skill benefits under realistic deployment conditions (installed libraries, memory, multiple harnesses/models) instead of idealized, hand-picked settings, enabling more generalizable comparisons of skill-enabled agents.