RESEARCH · RESEARCH · #3633
Empirical audit finds safety side effects from preventative persona steering (arXiv:2611.17620v1)
This arXiv preprint empirically audits 'preventative' persona steering—adding persona vectors during fine-tuning then removing them at evaluation—for sycophancy and hallucination on three models. The study measures trait expression, coherence, MMLU, and three safety benchmarks and finds that preventative steering alters refusal behavior in trait-dependent ways, so it is not demonstrably safety-neutral.
KEY POINTS
- This arXiv preprint empirically audits 'preventative' persona steering—adding persona vectors during fine-tuning then removing them at evaluation—for sycophancy and hallucination on three models.
- The study measures trait expression, coherence, MMLU, and three safety benchmarks and finds that preventative steering alters refusal behavior in trait-dependent ways, so it is not demonstrably safety-neutral.
- Because it shows preventative persona steering can change safety-critical behaviors despite preserving MMLU, steering methods should be validated on multiple safety benchmarks.
WHY IT MATTERS
Because it shows preventative persona steering can change safety-critical behaviors despite preserving MMLU, steering methods should be validated on multiple safety benchmarks.