Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #9545

deterministic verifier

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

RESEARCH · 1 SOURCE · arXiv cs.AI

SkillBuffBench: benchmark paper for measuring agent Skill utility across harnesses and models

The paper introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much file-based Agent Skills help across three Skill-access modes (no Skill, curated Skill, and automatic invocation from an installed library), memory on/off, four agent harnesses and multiple models (proprietary state-of-the-art models from Anthropic, Google and OpenAI and several open-weight Chinese models). The design emphasizes deterministic verification, paired task-level bootstrap uncertainty, and separate reporting of cost, agent rounds and wall-clock time; the paper describes the benchmark and tasks (currently in materials science and biology) and notes that evaluation runs are in progress with results to be added later.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

ScopeBench: benchmark and pilot release to measure agents' scope adherence under goal pressure (arXiv:2609.30325v1)

ScopeBench is a new benchmark of 30 dead-end agentic security tasks designed so the stated objective is reachable only by violating a stated scope; each task has a scopeless (capability) and scoped (adherence) condition. The paper introduces a deterministic verifier plus an "agentic judge" (calibrated against 100 human-labeled trajectories and audited) to detect violations, evaluates 8 models (raw capability 12.2%–81.1%, scope adherence 34.4%–86.7%), reports the judge found 331 violations missed by mechanical verification, highlights model differences (e.g., Opus-4-8 > sonnet-4-6 by 10 pp raw capability and 35.6 pp adherence), and releases the frozen pilot benchmark, evaluation code, and all 2160 ATIF trajectories.

7.0