RESEARCH · RESEARCH · #971
Epoch AI: cost to reach fixed benchmark scores fell ~47% per quarter; MIT finds smaller algorithmic gains (~3x/yr)
Epoch AI reports that market prices to achieve fixed scores on five AI benchmarks fell about 47% per quarter (≈13x/year) since 2023, citing examples like OpenAI's o3 vs GPT-5.6; the group warns its sample is limited and measures market price per benchmark score rather than pure algorithmic progress. MIT researchers (Hans Gundlach et al.) using broader model pricing data find 5–10x annual declines in benchmark cost but estimate actual algorithmic efficiency gains of ~3x/year after removing hardware and competition effects, and note some improvements come from higher test‑time compute rather than efficiency.
KEY POINTS
- Epoch AI reports that market prices to achieve fixed scores on five AI benchmarks fell about 47% per quarter (≈13x/year) since 2023, citing examples like OpenAI's o3 vs GPT-5.6; the group warns its sample is limited and measures market price per benchmark score rather than pure algorithmic progress.
- MIT researchers (Hans Gundlach et al.) using broader model pricing data find 5–10x annual declines in benchmark cost but estimate actual algorithmic efficiency gains of ~3x/year after removing hardware and competition effects, and note some improvements come from higher test‑time compute rather than efficiency.
- Falling benchmark costs can rapidly expand access to high‑performance AI and reshape economics for product development and deployment, but the split between hardware/market effects and true algorithmic gains affects long‑term expectations.
WHY IT MATTERS
Falling benchmark costs can rapidly expand access to high‑performance AI and reshape economics for product development and deployment, but the split between hardware/market effects and true algorithmic gains affects long‑term expectations.