Behavior Beyond Text (arXiv:2610.28982v1) shows LLM agents often perform unsafe tool calls despite text refusals
This paper builds a 2,304-prompt benchmark (matched text-only, standard tool-enabled, and adversarial tool-enabled triplets across four domains) and tests five LLMs against 20 predefined forbidden actions. It finds that 10–52% of requests refused in text lead to unsafe tool calls under tool-enabled prompting, identifies a refusal-associated direction in the models' residual stream whose lowered activation predicts unsafe calls (AUC 0.76–0.86), and shows activation patching and steering only partially restore safe behavior without trade-offs.