RESEARCH · RESEARCH · #1775
Whose Ground Truth? Position paper urges modeling human interpretation ambiguity in AI
This position paper (arXiv:2610.10805v1) argues that many human-centered tasks involve inherently ambiguous human interpretations and that treating annotation variability as noise or collapsing it to a single ground truth omits important information. The authors call for modeling the space of plausible human judgments and for changes to how AI systems are represented, learned, evaluated, deployed, and governed to reflect meaningful ambiguity.
KEY POINTS
- This position paper (arXiv:2610.10805v1) argues that many human-centered tasks involve inherently ambiguous human interpretations and that treating annotation variability as noise or collapsing it to a single ground truth omits important information.
- The authors call for modeling the space of plausible human judgments and for changes to how AI systems are represented, learned, evaluated, deployed, and governed to reflect meaningful ambiguity.
- If adopted, this shift could change dataset practices, evaluation metrics, and governance for human-centered AI so systems better reflect diverse valid human judgments.
WHY IT MATTERS
If adopted, this shift could change dataset practices, evaluation metrics, and governance for human-centered AI so systems better reflect diverse valid human judgments.