RESEARCH · RESEARCH · #1652
Metonymic circuits reveal how Vision Transformers ground abstract concepts
This arXiv paper introduces the concept of "metonymic circuits": intermediate, concrete anchor concepts (e.g., fire) that bridge visual signals to abstract labels (e.g., angry) in Vision Transformers. Using Transcoders applied to CLIP and DINO encoders and a curated icon dataset, the authors trace structured circuits where perceptual primitives appear in early layers, object-like anchors precede abstract targets, images with rendered text use a distinct perceptual-to-textual route, and causal interventions indicate these intermediates are functionally involved in grounding abstract concepts.
KEY POINTS
- This arXiv paper introduces the concept of "metonymic circuits": intermediate, concrete anchor concepts (e.g., fire) that bridge visual signals to abstract labels (e.g., angry) in Vision Transformers.
- Using Transcoders applied to CLIP and DINO encoders and a curated icon dataset, the authors trace structured circuits where perceptual primitives appear in early layers, object-like anchors precede abstract targets, images with rendered text use a distinct perceptual-to-textual route, and causal interventions indicate these intermediates are functionally involved in grounding abstract concepts.
- Identifies and causally validates intermediate, interpretable features that link perception to abstract semantics in vision models, informing interpretability and dataset design.
WHY IT MATTERS
Identifies and causally validates intermediate, interpretable features that link perception to abstract semantics in vision models, informing interpretability and dataset design.