RESEARCH · RESEARCH · #2628
Paper proposes 'scoped semantic invariance' and KG grounding to improve LLM consistency
The new arXiv paper formalizes consistency for LLMs as 'semantic invariance' over equivalence classes of queries, proves that invariance under arbitrary meaning-preserving transformations is undecidable, and introduces scoped semantic invariance evaluated within bounded semantic domains. The authors operationalize this with knowledge graphs (KGs), reporting that inconsistencies concentrate in localized hotspot subgraphs (TREE, STAR, CYCLE patterns) across MITRE D3FEND and eight Wikidata-derived domains, and that selective, layered graph grounding improves correctness and cross-model consistency while indiscriminate grounding can cause regressions.
KEY POINTS
- The new arXiv paper formalizes consistency for LLMs as 'semantic invariance' over equivalence classes of queries, proves that invariance under arbitrary meaning-preserving transformations is undecidable, and introduces scoped semantic invariance evaluated within bounded semantic domains.
- The authors operationalize this with knowledge graphs (KGs), reporting that inconsistencies concentrate in localized hotspot subgraphs (TREE, STAR, CYCLE patterns) across MITRE D3FEND and eight Wikidata-derived domains, and that selective, layered graph grounding improves correctness and cross-model consistency while indiscriminate grounding can cause regressions.
- Provides a formal evaluation framework and practical KG-grounding strategies that identify structured inconsistency hotspots and reduce LLM disagreement, which is directly relevant to improving model reliability and evaluation.
WHY IT MATTERS
Provides a formal evaluation framework and practical KG-grounding strategies that identify structured inconsistency hotspots and reduce LLM disagreement, which is directly relevant to improving model reliability and evaluation.