Paper: Narrow harmful fine-tuning of Skywork reward models generalizes across domains
The arXiv preprint (arXiv:2610.22805v1) reports that fine-tuning two state-of-the-art reward models (Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-V2-Qwen3-8B) to prefer harmful outputs in a single domain (medical or financial) produced preference inversions that generalized to seven held-out domains while only modestly degrading overall RewardBench 2 scores. The authors also fine-tuned the underlying language models and report that reward models were more vulnerable to these preference inversions, and they identify a mechanistic account: geometric compression of harmful vs. sound response representations at the last/scored token plus a content-specific rotation that brings harmful representations closer to the scalar head.