Open-weight models match top closed models on CTI-REALM (arXiv:2610.14180v1)
This paper (arXiv:2610.14180v1) evaluates 26 models (19 open-weight, 7 closed) on CTI-REALM, a benchmark where an agent converts threat-intel reports into working detections by querying live telemetry. Main findings: an open-weight model can match the strongest closed models at less than a tenth of their cost if allowed enough turns; the benchmark's published message-limit truncates some agents and biases results; relaxing the limit lets some open-weight models take the highest rewards. The authors introduce stage-level metrics that execute submitted detections against the telemetry store and recommend routing different workflow stages to the models that perform them best, and they report per-model token usage and that 'thinking' token share does not track reward.