FUNDING · MODELS · #1800
Google releases Android Bench 2.0 with long‑horizon tasks, agentic evaluation, and continuous scoring
Google published Android Bench 2.0, a major update to its benchmark for evaluating AI models on Android development tasks that adds long‑horizon tasks (LHTs), agentic evaluation (using providers' agents), and a continuous, multi‑factor scoring system instead of binary pass/fail. The new dashboard lists recent models (e.g., Gemini 3.8/3.7, OpenAI GPT‑6, Anthropic Fable 5.1, Kimi K3, Qwen 3.8 Max); at the time of reporting Claude Opus 5.5 led LHT pass rates at 32%, with GPT 6 Astra at 28%.
KEY POINTS
- Google published Android Bench 2.0, a major update to its benchmark for evaluating AI models on Android development tasks that adds long‑horizon tasks (LHTs), agentic evaluation (using providers' agents), and a continuous, multi‑factor scoring system instead of binary pass/fail.
- The new dashboard lists recent models (e.g., Gemini 3.8/3.7, OpenAI GPT‑6, Anthropic Fable 5.1, Kimi K3, Qwen 3.8 Max); at the time of reporting Claude Opus 5.5 led LHT pass rates at 32%, with GPT 6 Astra at 28%.
- Android Bench 2.0 provides a finer‑grained, more realistic way to measure model performance on complex, multi‑step developer tasks and highlights where current models succeed or still struggle.
WHY IT MATTERS
Android Bench 2.0 provides a finer‑grained, more realistic way to measure model performance on complex, multi‑step developer tasks and highlights where current models succeed or still struggle.