RESEARCH · RESEARCH · #2025
Open-weight models match top closed models on CTI-REALM (arXiv:2610.14180v1)
This paper (arXiv:2610.14180v1) evaluates 26 models (19 open-weight, 7 closed) on CTI-REALM, a benchmark where an agent converts threat-intel reports into working detections by querying live telemetry. Main findings: an open-weight model can match the strongest closed models at less than a tenth of their cost if allowed enough turns; the benchmark's published message-limit truncates some agents and biases results; relaxing the limit lets some open-weight models take the highest rewards. The authors introduce stage-level metrics that execute submitted detections against the telemetry store and recommend routing different workflow stages to the models that perform them best, and they report per-model token usage and that 'thinking' token share does not track reward.
KEY POINTS
- This paper (arXiv:2610.14180v1) evaluates 26 models (19 open-weight, 7 closed) on CTI-REALM, a benchmark where an agent converts threat-intel reports into working detections by querying live telemetry.
- Main findings: an open-weight model can match the strongest closed models at less than a tenth of their cost if allowed enough turns; the benchmark's published message-limit truncates some agents and biases results; relaxing the limit lets some open-weight models take the highest rewards.
- The authors introduce stage-level metrics that execute submitted detections against the telemetry store and recommend routing different workflow stages to the models that perform them best, and they report per-model token usage and that 'thinking' token share does not track reward.
WHY IT MATTERS
Shows that audited, lower-cost open-weight models can be competitive on a realistic security agent benchmark, and exposes a benchmark confound and practical improvements (stage metrics, routing) that affect model choice and evaluation.