RESEARCH · RESEARCH · #3031
Collinear AI publishes CWE-bench v2: 100 open tasks to evaluate LLM cyber defenses
Collinear AI released CWE-bench v2, an open public set of 100 deterministic Harbor tasks (Dockerized environments + verifiers) covering 91 CWEs for evaluating and training models on finding and fixing security weaknesses; the dataset, grader, and harness are open on Hugging Face and a held-out set is available on request (cyber@collinear.ai). Leaderboard results reported 11 evaluated models with Gemini 4 Argon leading (67.8% pass@1, 81.0% pass@4) narrowly ahead of Claude Opus 5.5, and cost/performance tradeoffs highlighted for DeepSeek V4.1 Flash, Opus, and Gemini.
KEY POINTS
- Collinear AI released CWE-bench v2, an open public set of 100 deterministic Harbor tasks (Dockerized environments + verifiers) covering 91 CWEs for evaluating and training models on finding and fixing security weaknesses; the dataset, grader, and harness are open on Hugging Face and a held-out set is available on request (cyber@collinear.ai).
- Leaderboard results reported 11 evaluated models with Gemini 4 Argon leading (67.8% pass@1, 81.0% pass@4) narrowly ahead of Claude Opus 5.5, and cost/performance tradeoffs highlighted for DeepSeek V4.1 Flash, Opus, and Gemini.
- An open, reproducible benchmark that spans many CWEs lets researchers and vendors measure and train LLMs on realistic vulnerability‑fixing tasks and compare cost/accuracy tradeoffs.
WHY IT MATTERS
An open, reproducible benchmark that spans many CWEs lets researchers and vendors measure and train LLMs on realistic vulnerability‑fixing tasks and compare cost/accuracy tradeoffs.