Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3354

Google and Artificial Analysis launch the Multimodal Knowledge Agent (MKA) benchmark

MKA is a new benchmark, developed with Artificial Analysis, designed to evaluate how agents reason across visually complex files in realistic knowledge-work tasks; it includes a 200-task private evaluation set (plus 10 public tasks on Hugging Face), an open-source agent harness called Stirrup, and an autorater for automated scoring. The report shows large efficiency and perception differences across models, and notes Gemini 4 Carbon ranks #2 on quality while completing tasks at roughly 6× lower cost than the top-scoring model.

KEY POINTS

  1. MKA is a new benchmark, developed with Artificial Analysis, designed to evaluate how agents reason across visually complex files in realistic knowledge-work tasks; it includes a 200-task private evaluation set (plus 10 public tasks on Hugging Face), an open-source agent harness called Stirrup, and an autorater for automated scoring.
  2. The report shows large efficiency and perception differences across models, and notes Gemini 4 Carbon ranks #2 on quality while completing tasks at roughly 6× lower cost than the top-scoring model.
  3. MKA creates a more realistic, multimodal test for agent capabilities and exposes important quality-cost and perception trade-offs that will influence development and deployment of agent systems.

WHY IT MATTERS

MKA creates a more realistic, multimodal test for agent capabilities and exposes important quality-cost and perception trade-offs that will influence development and deployment of agent systems.

SOURCES & TIMELINE

1