RESEARCH · RESEARCH · #2630
NewsBench: expert-grounded benchmark for epistemic quality in AI news reporting (arXiv:2610.33240v1)
NewsBench is an expert-grounded benchmark and two-phase evaluation pipeline for AI-generated news and current-events reporting that operationalizes source quality, factuality, and neutrality using guidelines set by senior journalists, policy, and intelligence practitioners. The release comprises ≈3,000 evaluation prompts (≈500 refreshed monthly), ~2,500 human expert-annotated items for calibration and validation, and a suite of automated AI judges validated against expert-adjudicated gold labels; applied to four frontier models, it reveals distinct, non-overlapping failure profiles across sourcing, factuality, and neutrality.
KEY POINTS
- NewsBench is an expert-grounded benchmark and two-phase evaluation pipeline for AI-generated news and current-events reporting that operationalizes source quality, factuality, and neutrality using guidelines set by senior journalists, policy, and intelligence practitioners.
- The release comprises ≈3,000 evaluation prompts (≈500 refreshed monthly), ~2,500 human expert-annotated items for calibration and validation, and a suite of automated AI judges validated against expert-adjudicated gold labels; applied to four frontier models, it reveals distinct, non-overlapping failure profiles across sourcing, factuality, and neutrality.
- Anchoring evaluation in expert editorial standards and validated AI judges provides a more reliable, scalable way to measure and compare how models handle sourcing, factuality, and neutrality in news reporting.
WHY IT MATTERS
Anchoring evaluation in expert editorial standards and validated AI judges provides a more reliable, scalable way to measure and compare how models handle sourcing, factuality, and neutrality in news reporting.