Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

NEWS · CODING · #3427

JioHotstar benchmarks knowledge-base approaches for AI coding agents

JioHotstar evaluated six knowledge-base approaches for AI coding agents — including a filesystem baseline, Sourcebot, Graphify, CocoIndex Code, Repowise and an in‑house RAG pipeline — using subsets of SWE-bench and CodeScaleBench plus a custom QA harness and Claude Sonnet as the solver. They found structural (symbol/file) search worked well on concrete engineering tasks, semantic retrieval helped with ambiguous requirements, and adding retrieval layers did not consistently improve task completion or reduce token usage; the team recommends keeping filesystem tools as a baseline and choosing retrieval by task type.

KEY POINTS

  1. JioHotstar evaluated six knowledge-base approaches for AI coding agents — including a filesystem baseline, Sourcebot, Graphify, CocoIndex Code, Repowise and an in‑house RAG pipeline — using subsets of SWE-bench and CodeScaleBench plus a custom QA harness and Claude Sonnet as the solver.
  2. They found structural (symbol/file) search worked well on concrete engineering tasks, semantic retrieval helped with ambiguous requirements, and adding retrieval layers did not consistently improve task completion or reduce token usage; the team recommends keeping filesystem tools as a baseline and choosing retrieval by task type.
  3. The results guide practitioners building coding agents toward matching retrieval method to task type and retaining simple filesystem tooling as a reliable baseline.

WHY IT MATTERS

The results guide practitioners building coding agents toward matching retrieval method to task type and retaining simple filesystem tooling as a reliable baseline.

SOURCES & TIMELINE

1