Mechanistic interpretability: what we can actually see inside networks

The field has moved past activation visualisation. Sparse autoencoders, circuit tracing, and attribution patches now give us real traction on what models compute — and what they do not.

Mechanistic interpretability aims to reverse-engineer the computations that neural networks perform. Not to explain their outputs post hoc, but to identify the specific circuits, features, and algorithms that produce those outputs. The premise is that if we understand the mechanism, we can predict, verify, and ultimately trust the system's behaviour.

This is harder than it sounds, and the field has spent the last three years learning exactly how much harder. But recent results suggest we are past the point of "impressive demo" and into the territory of "useful tool." Here is where things stand.

Sparse autoencoders changed the game

The fundamental obstacle in interpretability is polysemanticity: individual neurons respond to multiple, seemingly unrelated concepts. A single neuron in GPT-4 might activate for both legal terminology and cooking recipes. This makes interpreting individual neurons about as useful as interpreting individual pixels in a photograph.

Sparse autoencoders (SAEs) address this by learning a much larger set of features from the activations of a smaller set of neurons. Each feature in the SAE is approximately monosemantic — it corresponds to a single, interpretable concept. Anthropic's work on Claude's middle layers identified features for specific entities, programming constructs, linguistic patterns, and safety-relevant concepts.

The technique is not perfect. SAEs introduce reconstruction error, and the features they find are not guaranteed to be the features the model actually uses. But they give us a vocabulary for talking about what is happening inside the network that is far more useful than "neuron 4,817 fired."

Circuit tracing at scale

Once you have features, the next question is how they connect. Circuit tracing follows the path from input to output, identifying which features in which layers contribute to a specific behaviour. The IOI (Indirect Object Identification) circuit in GPT-2 — which identifies the indirect object in sentences like "John and Mary went to the store. John gave the bag to ___" — was the first major success story.

More recent work has scaled this to larger models and more complex behaviours. Researchers have traced circuits for in-context learning, for multi-step reasoning, and for specific safety-relevant decisions like refusing harmful requests. The circuits are messier than the clean diagrams in papers suggest — there is significant redundancy and distributed computation — but they exist, and they are increasingly tractable to find.

The practical limitation is that circuit tracing is labour-intensive. Each circuit requires careful experimental design, multiple ablation studies, and significant compute. Automating this process is an active area of research, and progress is being made, but we are not yet at the point where you can point a tool at a model and get a circuit diagram.

Attribution and activation patching

Attribution methods answer the question: "which parts of the input were most responsible for this output?" Activation patching is the more precise version: replace a specific activation at a specific layer with the activation from a different input, and observe how the output changes.

These methods are the workhorses of modern interpretability. They are relatively cheap to run, they produce quantitative results, and they compose well with other techniques. The main pitfall is that they measure correlation with the output, not causation — a feature might be correlated with the correct answer without actually being used by the model to produce that answer.

Combining attribution with ablation (zeroing out a feature and measuring the effect) or with activation patching (substituting a feature from a different input) helps establish causation. The field has developed increasingly rigorous protocols for distinguishing genuine causal features from incidental correlations.

What we still cannot do

Honesty requires acknowledging the gaps. We cannot currently:

Interpret a model fully. We can find and trace specific circuits, but we cannot produce a complete description of what a frontier model computes. The search space is too large and the interactions too complex.

Predict novel behaviours from interpretability. We can explain why a model did something after the fact, but we cannot reliably predict what it will do in a novel situation from its internal structure alone. This is the key gap for safety applications.

Scale interpretability to frontier models. Most mechanistic interpretability work has been done on models up to about 7B parameters. Frontier models are 100x larger. The techniques scale in principle, but the compute and human effort required scales roughly linearly, which is prohibitive.

Guarantee that interpretability findings are complete. Finding a circuit for a behaviour does not mean you have found all the circuits. Models are redundant, and a behaviour that appears to be mediated by one circuit may have backup circuits that activate when the primary one is disrupted.

Why this matters for alignment

Interpretability is not alignment, but alignment without interpretability is blind. If we cannot inspect what a model is computing, we cannot verify that its behaviour generalises correctly, that its safety training is robust, or that its stated reasoning corresponds to its actual computation.

The current state is promising but insufficient. We have tools that give us real insight into specific behaviours in specific models. We do not yet have tools that give us comprehensive assurance about a model's behaviour across all inputs. Closing that gap is one of the most important open problems in AI safety.

The practical recommendation: use interpretability tools as one input into your assessment of a model's reliability, not as the sole input. Combine them with behavioural evaluation, red-teaming, and formal verification where possible. No single method is sufficient, but together they provide substantially more confidence than any one alone.