RESEARCH · RESEARCH · #823
Graph-guided evidence navigation improves a local 9B model's reasoning on novels (exploratory arXiv result)
The arXiv preprint (arXiv:2609.22939v1) tests whether a frozen knowledge graph can help a local 9B model (qwen3.5:9b) answer questions about long novels. Across 30 detective novels and 234 multiple-choice questions, the best graph-based retrieval route reached 53.85% accuracy versus 46.15% for a recent-window baseline and ~51% for whole-book compression and vector retrieval; however, none of the fifteen graph vs baseline contrasts survive Holm correction, so the authors present these results as exploratory. Two more robust findings are that annotated evidence is enriched in the graphs' topological core (2.35× pooled) and that two graph-construction pipelines differ greatly in paragraph coverage (16% vs 73%), indicating pipeline bottlenecks that affect performance reporting.
KEY POINTS
- The arXiv preprint (arXiv:2609.22939v1) tests whether a frozen knowledge graph can help a local 9B model (qwen3.5:9b) answer questions about long novels.
- Across 30 detective novels and 234 multiple-choice questions, the best graph-based retrieval route reached 53.85% accuracy versus 46.15% for a recent-window baseline and ~51% for whole-book compression and vector retrieval; however, none of the fifteen graph vs baseline contrasts survive Holm correction, so the authors present these results as exploratory.
- Two more robust findings are that annotated evidence is enriched in the graphs' topological core (2.35× pooled) and that two graph-construction pipelines differ greatly in paragraph coverage (16% vs 73%), indicating pipeline bottlenecks that affect performance reporting.
WHY IT MATTERS
This work provides exploratory evidence that a frozen knowledge graph can enable a small local LLM to access long-range book evidence and highlights annotation and pipeline bottlenecks that determine whether graph methods help.