Research papers, myth by myth, Part 3: 'Nobody can see inside a model'

In May 2024, for one day, anyone could chat with a version of Claude that was obsessed with the Golden Gate Bridge. Ask it how to spend $10 and it suggested driving across the bridge and paying the toll. Ask it for a love story and you got a car that could not wait to cross its beloved bridge on a foggy day. Nobody had written a prompt telling it to do that, and nobody had retrained it. Researchers had found the place inside the model that stands for the bridge, and turned it up.

That demo is the most memorable result of a line of research that has quietly changed what we can say about the inside of a model. In this part I take one myth and test it against four papers.

The myth: “A model is a black box. Nobody can see what is inside it.”

The bottom line: we can now read millions of concepts inside real models and steer behaviour by changing them, so “nobody can see inside” is no longer true. What we see is still a partial map, expensive to draw, of one slice of the model.

What do I need to know first?

I use one everyday picture in this part, the one I used in the background post on how models work: a piano.

Weights are the strings. A model’s weights are set during training (the factory tunes the strings) and stay fixed while you use it.
A neuron is a key. It is one small calculation inside a layer: multiply the inputs by weights, add them up, pass the result through a simple function, and hand on one number.
An activation is how hard a key is pressed: the number a neuron outputs for the current piece of text.
A feature is a chord: a pattern across many neurons that stands for one concept, such as “this is Arabic script” or “this is about the Golden Gate Bridge”. A chord means something no single key does.
Interpretability is the research field that tries to explain how a model reaches its outputs. The flavour in this part is called mechanistic interpretability, because it looks for the actual mechanism, part by part.

Here is where people get tripped up. The obvious way to look inside a model is to read its neurons one at a time, the way you would label the keys on a piano. That mostly fails, and the reason it fails is the first paper.

What did the researchers do?

Four papers, one after another, each on a bigger model:

Toy Models of Superposition (Anthropic and Harvard, published September 2022). Tiny networks trained on made-up data, small enough to understand completely. Question: why do single neurons so often respond to several unrelated things?
Towards Monosemanticity (Anthropic, published October 2023). A small one-layer Transformer language model. Question: can a second, helper network pull the mixed-up neurons apart into clean concepts?
Scaling Monosemanticity (Anthropic, published May 2024). The same method on Claude 3 Sonnet, a production model at the time. Question: does it still work at full scale, and do the concepts actually control behaviour?
Gemma Scope (Google DeepMind, announced July 2024, blog post). The same kind of helper networks, trained on every layer of Gemma 2 2B and 9B, and released for anyone to use.

Why can’t we just read the neurons?

Because a model stores more concepts than it has neurons, so each neuron ends up carrying bits of several concepts at once. The paper calls this superposition.

Start with an everyday version. Imagine you have two light switches and five rooms to signal. With one switch per room you can only cover two rooms. But if you agree that “switch 1 fully on, switch 2 a little on” means room B, and other mixes mean the other rooms, you can signal all five, as long as you rarely need to signal two rooms at the same moment. Each switch now takes part in every signal. If you only ever look at switch 1 on its own, it seems to mean five different things.

Map that to the model: the switches are neurons, the rooms are concepts (features), and “rarely two at once” is what the paper calls sparsity. Most concepts are absent from most text. As the paper puts it, most tokens are not about Martin Luther King.

Now the paper’s own example. The team trained a tiny network to squeeze five features into two dimensions (think two neurons) and then rebuild them, with a ReLU at the end to filter the output.

When the five features were common (dense), the network did the sensible thing: it kept the two most important features, one per dimension, and dropped the other three entirely.
When the features were rare (sparse), it did something cleverer: it stored all five as directions arranged in a regular pentagon.

Here is the mechanism with small numbers. Suppose only feature B is present. B points mostly along neuron 1 and a little along neuron 2, so the two neurons read 0.95 and 0.31. To ask “is A present?”, the network measures how much the current point lines up with A’s direction. B overlaps A by 0.31, overlaps C by 0.31, and points away from D and E (minus 0.81). That small 0.31 overlap is interference: a false hint that A might be present. The ReLU at the end removes it: subtract a little, say 0.4, and set anything below zero to zero. A and C drop to zero, and B survives at 0.60. (The 0.95, 0.31 and 0.81 follow from the pentagon’s angles; the 0.4 is my illustrative choice.)

superposition in the toy model from Toy Models of Superposition: five concepts must fit into two neurons; when concepts are common the model keeps the two most important and drops three, when they are rare it packs all five as a pentagon, so every neuron carries a little of every concept and a ReLU filters out the small overlaps; read-out numbers are illustrative

That is the whole trick: accept a little interference in exchange for storing more concepts than you have neurons. And it explains the frustrating thing researchers kept seeing in real models. In the small language model from the second paper, one neuron responded to a mixture of academic citations, English dialogue, HTTP requests and Korean text. The paper’s word for a neuron like that is polysemantic (many meanings). The opposite, one neuron for one meaning, is monosemantic, which is where the next paper got its title.

How do you unpack a neuron into features?

You train a second, wider network whose only job is to rewrite the model’s neuron values as a much longer list of features, nearly all of them zero, and then rebuild the original values from that list. This is called dictionary learning, and the helper network is a sparse autoencoder (SAE).

An everyday version first. A smoothie tastes of something, but you cannot tell what is in it from one sip. Suppose you have a recipe book of 4,096 fruits, and a rule: explain every smoothie using as few fruits from the book as possible, and the recipe must taste like the original when you remake it. After practising on billions of smoothies, the recipes you write are short (“mostly banana, a little mango”) and each fruit in the book means one clear thing.

Map that to the real terms:

the smoothie is the list of neuron values at one point in the model for one token (its activations)
the recipe book is the dictionary, and each fruit in it is a feature
“as few fruits as possible” is the sparsity rule
“must taste like the original” is the reconstruction rule
writing the recipe is the encoder; remaking the smoothie from it is the decoder

Now the paper’s real set-up. Towards Monosemanticity took a one-layer Transformer whose MLP layer has 512 neurons. It trained sparse autoencoders on 8 billion samples of those 512 values, with dictionaries from 512 up to 131,072 features. Most of the analysis used one run with 4,096 features.

What comes out, step by step, on one of the paper’s own examples. Feed the model the end of a YouTube link, youtube.com/watch?v=dQw4w9WgXcQ:

On a token inside dQw4w9WgXcQ, many of the 512 neurons have non-zero values. Reading them one by one tells you little.
The encoder turns those 512 numbers into 4,096 feature values. Almost all are exactly zero. One that lights up strongly is a feature for base64 strings (the jumble of letters and digits used in links and encoded data).
The decoder rebuilds 512 values from the few active features, and they are close to the originals.

dictionary learning with a sparse autoencoder, as in Towards Monosemanticity: one neuron in a small model fires for four unrelated things, so the team trained an autoencoder that rewrites the layer's 512 neuron values as 4,096 features, almost all zero on any token, then rebuilds the 512 values from the few that fire; bar heights and lit cells are illustrative

The paper then checked a handful of features in detail and showed each one fires with high specificity for one thing: Arabic script, DNA sequences, base64 strings and Hebrew script. It also found that bigger dictionaries split concepts more finely: as the dictionary grew from 512 to 4,096 to 16,384 features, the number of base64-specific features went from 1 to 3 to many more.

How much of the layer did the features capture? In the main run, 79% of the drop in prediction error that the MLP layer provides was recovered when the layer’s real output was replaced by the autoencoder’s rebuilt version. The largest dictionary, 131,072 features, recovered 94.5%. The authors say these numbers “should be taken with a significant grain of salt”, which I come back to below.

Does it work on a real production model?

Yes. In 2024 Anthropic ran the same method on Claude 3 Sonnet and found millions of features, from concrete things like the Golden Gate Bridge to abstract ones like deception and bugs in code.

The details from Scaling Monosemanticity:

The team trained three dictionaries of roughly 1 million, 4 million and 34 million features.
They read the model’s residual stream at the middle layer only, halfway through the model.
On any one token, fewer than 300 features were active on average, and the rebuilt activations explained at least 65% of the variance of the real ones.

Anthropic’s blog post on the paper called it “the first ever detailed look inside a modern, production-grade large language model”. The features were richer than in the toy model: the Golden Gate Bridge feature (ID 34M/31164353) fired on mentions of the bridge, and Anthropic’s Golden Gate Claude post says it also responded to pictures of it. The paper also reports features related to unsafe code, bias, sycophancy, deception and power-seeking, and dangerous content. The authors add a caution I think is important: there is a difference between knowing about lies, being capable of lying, and actually lying.

Can you change behaviour by turning a feature up?

Yes, and that is what makes the features more than labels. If you hold a feature high and the model’s behaviour changes in exactly the way the label predicts, the label is probably right.

Everyday version first: a mixing desk. Each slider controls one instrument. If you push up the slider labelled “drums” and the drums get louder, you know the label is honest. If the violins got louder instead, you would know the label was wrong.

Map it: the slider is the feature’s activation, pushing it up is clamping the feature at a high value on every token, and the song is the model’s reply. This is called steering.

The paper’s examples:

Clamping the Golden Gate Bridge feature to 10 times its maximum activation made Claude 3 Sonnet start to self-identify as the Golden Gate Bridge.
Clamping a transit infrastructure feature to 5 times its maximum made the model mention a bridge where it otherwise would not.
Clamping a sycophantic praise feature to 5 times made Claude praise someone, in an over-the-top fashion, for claiming to have invented the phrase “Stop and smell the roses”.

The public demo, Golden Gate Claude, was the same idea in a chat window, online for a 24-hour period in May 2024 (Anthropic’s post is dated 23 May). Anthropic describes the change as “a precise, surgical change to some of the most basic aspects of the model’s internal activations”, and contrasts it with a system prompt and with fine-tuning: no extra text was added to the conversation and the weights were not retrained.

steering by turning one feature up: in the paper Anthropic clamped the Golden Gate Bridge feature in Claude 3 Sonnet to 10 times its maximum activation, with no prompt change and no retraining, and the model began to identify as the bridge; the public Golden Gate Claude demo turned the same feature up; the right-hand answers are Anthropic's descriptions of that demo, the normal answers are illustrative

Is this only a Claude thing?

No. Google DeepMind’s Gemma Scope trained the same kind of autoencoders on an open model and published them, so anyone can repeat this kind of work.

From the Gemma Scope blog post (31 July 2024) and the technical report:

Sparse autoencoders at every layer and sublayer output of Gemma 2 2B and 9B (and selected layers of 27B), more than 400 autoencoders with more than 30 million learned features in total, though DeepMind notes many likely overlap.
A newer design called the JumpReLU autoencoder, which DeepMind says makes it easier to balance spotting which features are present against estimating how strongly.
The cost: about 15% of the training compute of Gemma 2 9B, and about 20 pebibytes of saved activations, which DeepMind compares to about a million copies of English Wikipedia.
An interactive demo through Neuronpedia, so you can look up what a feature responds to without training anything.

Because Gemma 2 is an open-weight model, this matters beyond one lab. Outside researchers have already used it: the paper Do I Know This Entity? (November 2024) used Gemma Scope to find directions in Gemma 2 that track whether the model recognises an entity such as an athlete or a film, and showed they can push the model to refuse or to make things up. That is a direct bridge to the next part of this series.

So is the myth true?

Partly false now. “Nobody can see inside” was a fair description of the field ten years ago. It is not a fair description of 2024 onwards:

Researchers can extract millions of features from a production model and from open models, and many of them are readable concepts.
They can test a feature by turning it up and watching behaviour change in the predicted direction.
The tools for one family of open models are public, with a browser demo.

But the honest version is narrower than the headline. “We can see inside” means: we can see a partial map of some layers, built at great cost, which explains a large share of what those layers do but not all of it. The black box has windows now, though it is a long way from glass.

What did the papers not show?

The authors are unusually direct about the limits, and I think they belong in any summary of this work:

Not all the features. Anthropic writes that it does not believe it found anywhere near all the features in Sonnet, even in the one middle layer it studied, and that it is “quite likely that we’re orders of magnitude short”.
The cost does not scale yet. Finding all the features in all layers, they estimate, would need much more compute than training the model itself, which they call untenable. Gemma Scope’s 15% of a 9B model’s training compute shows the bill even at small scale.
Not all the behaviour. At least 65% of the variance explained also means up to 35% left out. Towards Monosemanticity adds that a “fraction of loss explained” figure can mislead, because the authors expect a long tail of rare features behind the last few percent.
Features are not mechanisms. A list of features says what concepts are present, not how they combine into an answer. Anthropic’s later circuit tracing work (March 2025) tries to trace those paths, and says that even on short, simple prompts it “only captures a fraction of the total computation”, taking a few hours of human effort per prompt of tens of words.
The toy results may not transfer. The pentagon geometry in the first paper comes with the authors’ own warning that it may be partly specific to the toy model.
A feature existing is not the model using it badly. A deception feature shows the model represents the concept, not that it is deceiving anyone.

how far the microscope has come, and how much is still dark: from five features in a toy model in 2022, to 4,096 features in a one-layer model in 2023, to 34 million features in the middle layer of Claude 3 Sonnet and 400 open autoencoders on Gemma 2 in 2024, while the authors say most features are still unfound and the cost is huge

Why does this matter if you build or use AI?

Do not treat interpretability as an audit you can buy today. It is research, and the authors say so. For your own systems, behaviour testing still does the heavy lifting: an eval set, logs and human review.
Expect it to show up in safety work first. Every paper here frames features for deception, sycophancy and dangerous content as the motivation. That is where model providers are most likely to use it.
If you want to look for yourself, use an open model. Gemma Scope and the Neuronpedia demo let a curious engineer see features on Gemma 2 without training anything.
Be wary of anyone who says a model’s reasoning is fully understood. The people doing the work do not say that.

Lock it in

Term                 What it is                                    Everyday picture
-------------------  --------------------------------------------  -----------------------------
Neuron               one small calculation in a layer              a piano key
Feature              a pattern across neurons = one concept        a chord
Superposition        more features than neurons, packed as         2 switches signalling 5 rooms
                     overlapping directions
Polysemantic neuron  one neuron that fires for unrelated things    one key in many chords
Sparse autoencoder   helper network: neuron values -> few          a smoothie -> a short recipe
                     active features -> rebuilt values             from a 4,096-fruit book
Steering             clamping a feature high to change behaviour   pushing one slider on a desk

My view: this is the most important change in how we can talk about models since the Transformer itself, because it turns “trust us” into something you can test. I would still describe it to a customer as a microscope that works on small samples, and I would not let it replace a single eval in a production system.

One honest gap: I do not know how well dictionary features hold up on the newest, much larger models with long reasoning traces. The published numbers are from Claude 3 Sonnet (2024), Claude 3.5 Haiku (2025) and Gemma 2. I have not seen public figures for how much of a current frontier model’s computation these methods explain.

What did we learn?

Neurons are a bad unit for reading a model, because models store more concepts than they have neurons and pack them as overlapping directions, which is called superposition.
A sparse autoencoder rewrites a layer’s neuron values as a long list of features, almost all zero on any token, and many of those features are readable concepts such as base64 strings or the Golden Gate Bridge.
Anthropic scaled this to Claude 3 Sonnet in 2024 with dictionaries of up to 34 million features, explaining at least 65% of the variance at the middle layer.
Turning a feature up changes behaviour in the predicted way, as Golden Gate Claude showed, which is evidence the labels are right.
Google DeepMind’s Gemma Scope released more than 400 autoencoders for Gemma 2, so outside researchers can do this work too.
The map is partial and expensive: the authors say they are probably orders of magnitude short of all features, and finding them all would cost more than training the model.

What comes next?

Part 4 takes the myth that hallucinations are random glitches and that a model has no idea when it is wrong. It uses the same kind of internal tracing from this part to show a “can’t answer” circuit inside Claude, and two papers on how well models know what they know and why training and grading reward guessing.

Sources

Toy Models of Superposition, Elhage et al., Anthropic and Harvard, published September 2022
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Bricken et al., Anthropic, published October 2023
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Templeton et al., Anthropic, published May 2024
Mapping the mind of a large language model, Anthropic blog, May 2024
Golden Gate Claude, Anthropic, May 2024
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2, Lieberum et al., Google DeepMind, arXiv, August 2024
Gemma Scope: helping the safety community shed light on the inner workings of language models, Google DeepMind blog, July 2024
Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models, Ferrando et al., arXiv, November 2024
Tracing the thoughts of a large language model, Anthropic, March 2025

Opinions are my own. Every fact in this post comes from the public documentation linked above.