RESEARCH · RESEARCH · #2120
Goodfire's Nick Cammarata argues neural networks are largely interpretable in new essay
In a long essay titled "Neural Networks Are Already Clean," Goodfire researcher Nick Cammarata reinterprets classic examples of polysemantic neurons and argues that most neuron activations are understandable using scaled-up, automated neuron analysis; he reports ~90% of neuron poles in Qwen3-32B have a single main role and only 2–4% remain unexplained, and says earlier superposition interpretations were often mistaken. The essay has drawn enthusiastic praise from some (including OpenAI poster Roon) and caution from others (including DeepMind's Neel Nanda), and Goodfire says it sees similar results in larger models though independent confirmation is pending.
KEY POINTS
- In a long essay titled "Neural Networks Are Already Clean," Goodfire researcher Nick Cammarata reinterprets classic examples of polysemantic neurons and argues that most neuron activations are understandable using scaled-up, automated neuron analysis; he reports ~90% of neuron poles in Qwen3-32B have a single main role and only 2–4% remain unexplained, and says earlier superposition interpretations were often mistaken.
- The essay has drawn enthusiastic praise from some (including OpenAI poster Roon) and caution from others (including DeepMind's Neel Nanda), and Goodfire says it sees similar results in larger models though independent confirmation is pending.
- If validated, the claim that neuron-level circuits are mostly interpretable would materially affect AI safety and alignment by making internal model behavior — including potential deception — more visible and analyzable.
WHY IT MATTERS
If validated, the claim that neuron-level circuits are mostly interpretable would materially affect AI safety and alignment by making internal model behavior — including potential deception — more visible and analyzable.