NEWS · CODING · #3427
JioHotstar benchmarks knowledge-base approaches for AI coding agents
JioHotstar evaluated six knowledge-base approaches for AI coding agents — including a filesystem baseline, Sourcebot, Graphify, CocoIndex Code, Repowise and an in‑house RAG pipeline — using subsets of SWE-bench and CodeScaleBench plus a custom QA harness and Claude Sonnet as the solver. They found structural (symbol/file) search worked well on concrete engineering tasks, semantic retrieval helped with ambiguous requirements, and adding retrieval layers did not consistently improve task completion or reduce token usage; the team recommends keeping filesystem tools as a baseline and choosing retrieval by task type.
KEY POINTS
- JioHotstar evaluated six knowledge-base approaches for AI coding agents — including a filesystem baseline, Sourcebot, Graphify, CocoIndex Code, Repowise and an in‑house RAG pipeline — using subsets of SWE-bench and CodeScaleBench plus a custom QA harness and Claude Sonnet as the solver.
- They found structural (symbol/file) search worked well on concrete engineering tasks, semantic retrieval helped with ambiguous requirements, and adding retrieval layers did not consistently improve task completion or reduce token usage; the team recommends keeping filesystem tools as a baseline and choosing retrieval by task type.
- The results guide practitioners building coding agents toward matching retrieval method to task type and retaining simple filesystem tooling as a reliable baseline.
WHY IT MATTERS
The results guide practitioners building coding agents toward matching retrieval method to task type and retaining simple filesystem tooling as a reliable baseline.