RESEARCH · RESEARCH · #2881
Socio-demographic signals modestly aid pluralistic offensiveness modeling, especially after aggregation and with explicit reasoning
This paper (arXiv:2610.37896v1) evaluates when and how annotators' socio-demographic information helps pluralistic offensiveness prediction using three multi-annotator datasets and prompted/trained LLMs. The authors find demographic gains are small and inconsistent for individual predictions but clearer at group/population levels after aggregation; framing prediction as an explicit decomposition into a baseline plus a demographic-specific deviation and using explicit reasoning yields better baseline and shift estimates and improves final predictions compared with direct conditioning alone.
KEY POINTS
- This paper (arXiv:2610.37896v1) evaluates when and how annotators' socio-demographic information helps pluralistic offensiveness prediction using three multi-annotator datasets and prompted/trained LLMs.
- The authors find demographic gains are small and inconsistent for individual predictions but clearer at group/population levels after aggregation; framing prediction as an explicit decomposition into a baseline plus a demographic-specific deviation and using explicit reasoning yields better baseline and shift estimates and improves final predictions compared with direct conditioning alone.
- The work explains mixed prior findings and shows that socio-demographic signals can be useful for fairness-aware offensiveness modeling if evaluated at group levels and guided by explicit reasoning, informing annotation and modeling choices.
WHY IT MATTERS
The work explains mixed prior findings and shows that socio-demographic signals can be useful for fairness-aware offensiveness modeling if evaluated at group levels and guided by explicit reasoning, informing annotation and modeling choices.