Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3805

SkillBuffBench: benchmark paper for measuring agent Skill utility across harnesses and models

The paper introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much file-based Agent Skills help across three Skill-access modes (no Skill, curated Skill, and automatic invocation from an installed library), memory on/off, four agent harnesses and multiple models (proprietary state-of-the-art models from Anthropic, Google and OpenAI and several open-weight Chinese models). The design emphasizes deterministic verification, paired task-level bootstrap uncertainty, and separate reporting of cost, agent rounds and wall-clock time; the paper describes the benchmark and tasks (currently in materials science and biology) and notes that evaluation runs are in progress with results to be added later.

KEY POINTS

  1. The paper introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much file-based Agent Skills help across three Skill-access modes (no Skill, curated Skill, and automatic invocation from an installed library), memory on/off, four agent harnesses and multiple models (proprietary state-of-the-art models from Anthropic, Google and OpenAI and several open-weight Chinese models).
  2. The design emphasizes deterministic verification, paired task-level bootstrap uncertainty, and separate reporting of cost, agent rounds and wall-clock time; the paper describes the benchmark and tasks (currently in materials science and biology) and notes that evaluation runs are in progress with results to be added later.
  3. This matters because it evaluates Skill benefits under realistic deployment conditions (installed libraries, memory, multiple harnesses/models) instead of idealized, hand-picked settings, enabling more generalizable comparisons of skill-enabled agents.

WHY IT MATTERS

This matters because it evaluates Skill benefits under realistic deployment conditions (installed libraries, memory, multiple harnesses/models) instead of idealized, hand-picked settings, enabling more generalizable comparisons of skill-enabled agents.

SOURCES & TIMELINE

1