NEWS · MODELS · #3259
Vals benchmark finds open-weight Deepseek close to OpenAI's GPT-6 Sol on autonomous cyber attacks
Vals AI's CyberOffense benchmark simulated realistic corporate networks and ran solo AI agents starting from a public web address; OpenAI's GPT-6 Sol (Daybreak) achieved 80.8% of objectives, while the open-weight Deepseek V4.1 Flash 1017 reached 72.8% at much lower cost, with longer runtimes proving critical, Vals reports. The findings echo a UK AISI analysis and Anthropic tests showing narrower gaps between closed and open models, and Vals said the same models could be used defensively to find vulnerabilities.
KEY POINTS
- Vals AI's CyberOffense benchmark simulated realistic corporate networks and ran solo AI agents starting from a public web address; OpenAI's GPT-6 Sol (Daybreak) achieved 80.8% of objectives, while the open-weight Deepseek V4.1 Flash 1017 reached 72.8% at much lower cost, with longer runtimes proving critical, Vals reports.
- The findings echo a UK AISI analysis and Anthropic tests showing narrower gaps between closed and open models, and Vals said the same models could be used defensively to find vulnerabilities.
- Shows that open-weight models are approaching closed frontier models in autonomous cyber-offense capability at much lower cost, raising access, defense and policy risks and opportunities.
WHY IT MATTERS
Shows that open-weight models are approaching closed frontier models in autonomous cyber-offense capability at much lower cost, raising access, defense and policy risks and opportunities.