FairStress benchmark released to evaluate demographic fairness and decision stability in LLMs
The team introduced FairStress, a 300-scenario benchmark across 29 domains (hiring, healthcare, lending, education, housing, criminal justice, etc.) that measures demographic fairness beyond final model choices by testing how models respond to matched persuasive "adversarial prefixes." FairStress is available for download and will be integrated into Olmo-eval; initial results show asymmetric effects of identical persuasive arguments across models including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and differences in explanations and internal activations across training stages.