Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3031

Collinear AI publishes CWE-bench v2: 100 open tasks to evaluate LLM cyber defenses

Collinear AI released CWE-bench v2, an open public set of 100 deterministic Harbor tasks (Dockerized environments + verifiers) covering 91 CWEs for evaluating and training models on finding and fixing security weaknesses; the dataset, grader, and harness are open on Hugging Face and a held-out set is available on request (cyber@collinear.ai). Leaderboard results reported 11 evaluated models with Gemini 4 Argon leading (67.8% pass@1, 81.0% pass@4) narrowly ahead of Claude Opus 5.5, and cost/performance tradeoffs highlighted for DeepSeek V4.1 Flash, Opus, and Gemini.

KEY POINTS

  1. Collinear AI released CWE-bench v2, an open public set of 100 deterministic Harbor tasks (Dockerized environments + verifiers) covering 91 CWEs for evaluating and training models on finding and fixing security weaknesses; the dataset, grader, and harness are open on Hugging Face and a held-out set is available on request (cyber@collinear.ai).
  2. Leaderboard results reported 11 evaluated models with Gemini 4 Argon leading (67.8% pass@1, 81.0% pass@4) narrowly ahead of Claude Opus 5.5, and cost/performance tradeoffs highlighted for DeepSeek V4.1 Flash, Opus, and Gemini.
  3. An open, reproducible benchmark that spans many CWEs lets researchers and vendors measure and train LLMs on realistic vulnerability‑fixing tasks and compare cost/accuracy tradeoffs.

WHY IT MATTERS

An open, reproducible benchmark that spans many CWEs lets researchers and vendors measure and train LLMs on realistic vulnerability‑fixing tasks and compare cost/accuracy tradeoffs.

SOURCES & TIMELINE

1