RESEARCH · RESEARCH · #3354
Google and Artificial Analysis launch the Multimodal Knowledge Agent (MKA) benchmark
MKA is a new benchmark, developed with Artificial Analysis, designed to evaluate how agents reason across visually complex files in realistic knowledge-work tasks; it includes a 200-task private evaluation set (plus 10 public tasks on Hugging Face), an open-source agent harness called Stirrup, and an autorater for automated scoring. The report shows large efficiency and perception differences across models, and notes Gemini 4 Carbon ranks #2 on quality while completing tasks at roughly 6× lower cost than the top-scoring model.
KEY POINTS
- MKA is a new benchmark, developed with Artificial Analysis, designed to evaluate how agents reason across visually complex files in realistic knowledge-work tasks; it includes a 200-task private evaluation set (plus 10 public tasks on Hugging Face), an open-source agent harness called Stirrup, and an autorater for automated scoring.
- The report shows large efficiency and perception differences across models, and notes Gemini 4 Carbon ranks #2 on quality while completing tasks at roughly 6× lower cost than the top-scoring model.
- MKA creates a more realistic, multimodal test for agent capabilities and exposes important quality-cost and perception trade-offs that will influence development and deployment of agent systems.
WHY IT MATTERS
MKA creates a more realistic, multimodal test for agent capabilities and exposes important quality-cost and perception trade-offs that will influence development and deployment of agent systems.