Counterfactual Neighborhood Risk Fields (CNRF) for selective safety in vision‑language models
The paper introduces Counterfactual Neighborhood Risk Fields (CNRF), a representation-based, evidence-grounded framework that builds context-matched benign–harmful counterfactual pairs to derive capability-related risk directions and separately estimate neighborhood support. CNRF enables selective intervention (routing requests for same-victim review or producing constrained alternatives) and reports 86.06–89.30% macro defense success across three VLMs and five multimodal jailbreak families while retaining substantial benign-task utility.