RESEARCH · RESEARCH · #2888
What Does an Ontology Catch? arXiv:2610.37809v1 — Comparing curated, LLM-generated and data-derived ontologies as checkers for LLM agents
This paper studies ontologies themselves as checkers of LLM agents answering questions over knowledge graphs, comparing curated, LLM-generated and data-derived schemas used as prompt context, as checkers, or both. Evaluated on public graphs, UniProt and a private graph of ~10^9 triples across open-weight and proprietary agents, the authors quantify trade-offs between caught errors and false alarms, show curated schemas can contradict their data and often flag many correct answers, and examine whether combining sources or giving agents checker-specific feedback helps.
KEY POINTS
- This paper studies ontologies themselves as checkers of LLM agents answering questions over knowledge graphs, comparing curated, LLM-generated and data-derived schemas used as prompt context, as checkers, or both.
- Evaluated on public graphs, UniProt and a private graph of ~10^9 triples across open-weight and proprietary agents, the authors quantify trade-offs between caught errors and false alarms, show curated schemas can contradict their data and often flag many correct answers, and examine whether combining sources or giving agents checker-specific feedback helps.
- This matters because many systems rely on ontologies to judge LLM outputs over knowledge graphs; the paper quantifies how different ontology sources trade missed errors for false alarms and questions the reliability of curated schemas as automated checkers.
WHY IT MATTERS
This matters because many systems rely on ontologies to judge LLM outputs over knowledge graphs; the paper quantifies how different ontology sources trade missed errors for false alarms and questions the reliability of curated schemas as automated checkers.