RESEARCH · RESEARCH · #2268
FairStress benchmark released to evaluate demographic fairness and decision stability in LLMs
The team introduced FairStress, a 300-scenario benchmark across 29 domains (hiring, healthcare, lending, education, housing, criminal justice, etc.) that measures demographic fairness beyond final model choices by testing how models respond to matched persuasive "adversarial prefixes." FairStress is available for download and will be integrated into Olmo-eval; initial results show asymmetric effects of identical persuasive arguments across models including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and differences in explanations and internal activations across training stages.
KEY POINTS
- The team introduced FairStress, a 300-scenario benchmark across 29 domains (hiring, healthcare, lending, education, housing, criminal justice, etc.) that measures demographic fairness beyond final model choices by testing how models respond to matched persuasive "adversarial prefixes." FairStress is available for download and will be integrated into Olmo-eval; initial results show asymmetric effects of identical persuasive arguments across models including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and differences in explanations and internal activations across training stages.
- FairStress reveals hidden asymmetries in how LLMs change decisions when challenged, giving a more granular tool to detect unfair behavior that raw output parity can miss.
- FairStress: A deeper evaluation of fairness in LLM decision-making
WHY IT MATTERS
FairStress reveals hidden asymmetries in how LLMs change decisions when challenged, giving a more granular tool to detect unfair behavior that raw output parity can miss.