RESEARCH · RESEARCH · #724
ArXiv paper: fine-tuning alters LLM representations but EAP-identified causal components are decoupled
The new arXiv paper (arXiv:2609.21113v1) studies how fine-tuning changes internal LLM mechanisms and whether those representational changes align with task-relevant components identified by EAP (e.g., attention heads and logit-level activations). The authors report that EAP-identified components are localized to particular layers, yet their layer distribution is largely uncorrelated with layers showing the biggest representational shifts during fine-tuning, and overlap in EAP components across tasks does not reliably produce cross-task transfer and can sometimes cause performance degradation between dissimilar tasks.
KEY POINTS
- The new arXiv paper (arXiv:2609.21113v1) studies how fine-tuning changes internal LLM mechanisms and whether those representational changes align with task-relevant components identified by EAP (e.g., attention heads and logit-level activations).
- The authors report that EAP-identified components are localized to particular layers, yet their layer distribution is largely uncorrelated with layers showing the biggest representational shifts during fine-tuning, and overlap in EAP components across tasks does not reliably produce cross-task transfer and can sometimes cause performance degradation between dissimilar tasks.
- This matters because it separates where task-critical causal components reside from where representations change during fine-tuning, affecting interpretation, modular interventions, and expectations about transfer or catastrophic interference.
WHY IT MATTERS
This matters because it separates where task-critical causal components reside from where representations change during fine-tuning, affecting interpretation, modular interventions, and expectations about transfer or catastrophic interference.