Research papers, myth by myth, Part 1: 'An AI reads one word at a time, like we do'

When I explain how a chatbot works, someone usually describes it the way they read a book: it goes through the sentence one word at a time, keeps a running summary in its head, and by the end of the paragraph it “remembers” what it has read. It is a natural picture, because the answer appears on screen word by word. It is also how the leading language systems did work until 2017. Then a team at Google published a short paper with a confident title, and almost every model you use today is built on it.

This series takes one common myth about AI models at a time and asks what the research actually shows. I use the papers’ own examples and link every paper, so you can check me. Part 1 starts with the oldest myth and the most famous paper.

The myth: “An AI reads a sentence one word at a time, like we do, and keeps a running memory.”

The bottom line: it is half true. A modern model reads your whole prompt at once, with every word looking directly at every other word, and keeps all of it rather than a running summary. It only goes one step at a time when it writes.

What do I need to know first?

Five terms carry this whole post.

Token: the unit a model reads and writes, often a word or part of a word. “didn’t” might be two tokens, “did” and “n’t”. I say “word” below when the difference does not matter.
Vector: an ordered list of numbers. Inside a model, each token is represented by a vector, for example [0.2, -1.3, 0.7, ...] with hundreds or thousands of numbers. Words with related meanings end up with similar vectors.
Recurrent neural network (RNN): the design most language systems used before 2017. It reads one token per step and passes a single vector, the hidden state, from each step to the next. That hidden state is the “running memory” in the myth.
Attention: a way for one token to pull in information from other tokens, by scoring how relevant each one is and taking a weighted mix of them. “Self-attention” means the tokens of one sentence attending to each other.
Transformer: the design the paper introduced, built from stacked layers of self-attention, with no recurrence at all.

A quick example of the first idea: if the sentence is “I arrived at the bank after crossing the river”, the meaning of “bank” depends on “river”, eight words later. Any design has to get information from “river” to “bank” somehow. The question is how many steps that takes.

What did the researchers do?

In June 2017, Ashish Vaswani and seven co-authors, working at Google Brain and Google Research, posted Attention Is All You Need (published June 2017). The paper lists all eight as equal contributors. Their task was machine translation, English to German and English to French, where RNNs with an added attention step were the state of the art.

Their idea was to throw away the recurrence and keep only the attention. In their words, the Transformer is “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”. The model they described has an encoder of 6 layers that reads the source sentence and a decoder of 6 layers that writes the translation, with every token represented by a vector of 512 numbers. Google’s research blog explained the work in Transformer: A Novel Neural Network Architecture for Language Understanding (published August 2017).

Here is the difference in one picture, using the sentence from Google’s blog post.

how the two designs read: an RNN passes one running memory along the sentence, so 'animal' reaches 'it' only after five hand-offs; a transformer lets 'it' look straight at every earlier word in one step, for all words at once; writing is still one token at a time

What did they find?

How does an RNN read, and where does it struggle?

Think of a relay race where each runner can carry only one small bag. The first runner puts a note in it, hands it on, the next runner squeezes in their own note, and so on. By the eighth runner, the first note is crumpled under seven others. That bag is the RNN’s hidden state, and each runner is one step.

In the sentence “The animal didn’t cross the street because it was too tired”, the word “animal” goes into the bag at step 2. When the RNN reaches “it” at step 8, the only route back to “animal” is through that one vector, which has been rewritten five times since. The Google blog post puts it plainly: “the more such steps decisions require, the harder it is for a recurrent network to learn how to make those decisions.”

There is a second cost, and it is the one that changed the industry. Step 8 cannot start until step 7 has finished. GPUs and TPUs are fast because they do thousands of calculations at once, and a chain of dependent steps leaves most of that capacity idle.

What does self-attention do instead?

Self-attention lets every token look at every other token directly, in a single step. The paper’s Table 1 states it in computer-science shorthand: a recurrent layer needs a number of sequential operations that grows with the sentence length n, written O(n), while a self-attention layer needs a constant number, O(1). The longest path information has to travel between any two words drops from n steps to one.

Here is the paper’s example of what that buys. Google’s blog post shows two sentences that differ by one word:

“The animal didn’t cross the street because it was too tired.” Here “it” is the animal.
“The animal didn’t cross the street because it was too wide.” Here “it” is the street.

A person resolves this without noticing. A translator into French cannot avoid it, because “the animal” is masculine (l’animal, so “it” becomes il) and “the street” is feminine (la rue, so elle). The blog reports that the Transformer translated both sentences to French correctly, and it visualised where the encoder’s attention from “it” went: “the Transformer clearly identified the two nouns ‘it’ could refer to and the respective amount of attention reflects its choice in the different contexts.”

where 'it' looks in the two sentences from Google's 2017 blog post: in the 'tired' sentence most of the attention from 'it' goes to 'animal', in the 'wide' sentence to 'street', which is what a translator needs to choose il or elle in French; line thickness is illustrative

The paper’s own appendix shows the same behaviour on other sentences: a head in layer 5 of 6 that links the verb “making” to the distant words “more difficult” (Figure 3), and two heads that resolve what “its” refers to, with attention the authors describe as “very sharp for this word” (Figure 4).

How does one attention step work, with small numbers?

This is the part people find hardest, so let me use a library analogy that maps one to one. You walk in with a question (“books about tired animals”). Every book has a label on its spine. You compare your question with every label, decide how relevant each book is, and walk out with a mix of pages, mostly from the most relevant books. In attention:

Query (q): the question a token asks. Here, the vector “it” uses to ask “who am I referring to?”
Key (k): the label each token offers so that others can find it.
Value (v): the content a token hands over if it is picked.

All three are vectors the model computes from each token, using weights it learned in training. The paper’s formula is Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V. Here it is with toy numbers, four numbers per vector instead of 64, and only the two candidate nouns so the arithmetic stays small:

Query from "it":           q        = [1, 1, 0, 0]
Keys:                      k_animal = [2, 2, 0, 0]    k_street = [0, 0, 2, 2]
Values:                    v_animal = [1, 0]          v_street = [0, 1]

1. Match (dot product)     q·k_animal = 1x2 + 1x2 + 0 + 0 = 4
                           q·k_street = 0 + 0 + 0 + 0     = 0
2. Scale by √d_k (d_k=4)   4 ÷ 2 = 2        0 ÷ 2 = 0
3. Softmax                 e^2 = 7.39, e^0 = 1, total 8.39
                           animal 7.39 ÷ 8.39 = 0.88    street 1 ÷ 8.39 = 0.12
4. Mix the values          0.88 x [1, 0] + 0.12 x [0, 1] = [0.88, 0.12]

After this step, the vector for “it” is 88% “animal”. Two terms in that trace deserve a line each. A dot product is the matching score. Softmax turns scores into weights that add up to 1. The paper divides by √d_k because, as it explains, dot products of long vectors grow large and push softmax into giving almost all the weight to one token, where training barely learns.

What about the “wide” sentence? The query is not fixed. Earlier layers have already mixed “wide” into the vector for “it”, so in that sentence its query points towards the street’s key, and the weights flip.

one attention step with toy numbers: the query from 'it' is compared with each key by a dot product, scaled by the square root of the key size, turned into weights by softmax, and the weights mix the values, so the new 'it' is 88 percent 'animal'

Why several heads, and how does it know word order?

Multi-head attention runs several attention steps side by side, each with its own queries, keys and values, then joins the results. The paper uses 8 heads, each with vectors of 64 numbers (512 ÷ 8), so the total cost is similar to one big head. The point is that one head can track which noun a pronoun refers to while another tracks the verb’s object. The appendix figures show exactly this kind of division: different heads, different jobs.

Positional encoding fixes a problem attention creates. If every word looks at every other word at once, then “dog bites man” and “man bites dog” contain the same words and the same pairs, and pure attention cannot tell them apart. The paper adds a position signal to each token’s vector before the first layer, built from sine and cosine waves of different lengths, so position 1, position 2 and position 3 each get a distinctive pattern. The authors tried learned positions too and “found that the two versions produced nearly identical results”; they kept the waves because they may let the model handle sentences longer than any it saw in training.

What did it achieve?

On the standard WMT 2014 benchmarks, scored in BLEU, the large Transformer scored 28.4 on English to German, more than 2 points above the best previous results including ensembles, and 41.8 on English to French, a new single-model record. The base model trained in 12 hours on one machine with 8 NVIDIA P100 GPUs; the big model took 3.5 days. The paper calls this “a small fraction of the training costs of the best models from the literature”. The reason is the parallelism: a whole batch of sentences, every token at once.

What did later research add about how attention moves information?

Two papers from Anthropic’s interpretability team make the “no running memory” point sharper.

A Mathematical Framework for Transformer Circuits (published December 2021) describes the model as a shared workspace it calls the residual stream: each token has its own vector, every layer reads from it and adds to it, and “the fundamental action of attention heads is moving information. They read information from the residual stream of one token, and write it to the residual stream of another token.” Instead of one bag passed down a line, every token keeps its own notebook, and attention heads copy notes between notebooks.

In-context Learning and Induction Heads (published March 2022) found a concrete example of that copying. An induction head is a pair of attention heads that completes patterns: if the prompt contains [A][B] and later [A] again, it predicts [B]. Say a prompt reads “Mrs Patel runs the bakery. Ask Mrs”. A first head, the “previous token head”, tags “Patel” with “the token before me was Mrs”. At the second “Mrs”, the induction head searches for the token whose previous token was “Mrs”, finds “Patel”, and copies it forward. The paper stresses that this “applies regardless of what A and B are”, so it is an algorithm, not a memorised pair, which is why it works on names the model has never seen.

an induction head completing a pattern: a previous-token head first tags 'Patel' with 'came after Mrs'; when 'Mrs' appears again, the induction head looks for the token that followed 'Mrs' last time, finds 'Patel' and copies it, so the model predicts 'Patel'

The authors found that induction heads appear in a sudden “phase change” early in training, visible as a bump in the training loss, at the same moment the model gets much better at using earlier parts of its prompt, which is called in-context learning.

So is the myth true?

Half true. Split it into its two claims.

“Reads one word at a time”: false for a Transformer. It processes all the tokens of your prompt in parallel, and in every layer each token can attend to any earlier token directly. That is the whole point of the 2017 paper.

“Keeps a running memory”: false in the sense people mean. An RNN compresses everything into one hidden state. A Transformer keeps a separate vector for every token in its context window and looks back at all of them each time. Nothing is squeezed into one bag, which is also why a long prompt costs more: there is more to look at.

“One at a time” is true for writing. The paper’s decoder generates output “one element at a time” and is auto-regressive: “at each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next.” The decoder is masked so that a position cannot peek at words that come after it. The models in today’s chatbots keep that design, which is why answers stream onto your screen token by token. The authors even named it as unfinished business: “Making generation less sequential is another research goal of ours.”

So the honest correction is: reading is parallel, writing is sequential. Part 2 asks whether “one token at a time” also means “no planning”.

What did the paper not show?

It was a translation paper. It tested English to German, English to French and one parsing task. Chatbots, code and long documents came later, built by other teams on the same blocks. The paper did not show that the design would scale to today’s models; later work did.
Attention is not free. The same Table 1 shows the cost per layer grows with the square of the sentence length, O(n²·d). The authors noted it as fine for sentences and suggested restricting attention to a neighbourhood for very long inputs. That cost is why long context windows are expensive.
Attention pictures are hints, not explanations. The figures show where a head looks, not why the final answer came out. The authors used careful words such as “apparently involved in anaphora resolution”. Later interpretability work, including Part 2’s papers, exists because attention maps alone do not tell the full story.
Induction heads are best proven in small models. Anthropic’s own summary rates the evidence as “strong, causal” for small attention-only models and only “medium, correlational” for large models, where they could not knock the heads out cleanly.

Why does this matter if you build or use AI?

Order matters less than you think, content matters more. Because every token can see every other token, the model does not “forget” the start of your prompt the way an RNN would. It can still miss things in very long prompts, so put the instruction and the key facts where they are clear, and test.
Long prompts cost money and time. Attention compares tokens with each other, so more tokens means more work, and providers price by token. Trim what the model does not need.
Streaming is real, not theatre. The words appear one by one because they are produced one by one. The wait before the first word is the model reading your prompt; the speed after it is the writing.

If you want to see how this plays out in production systems, I cover prompts, context and cost in the Production-ready agents series.

Lock it in

Myth           "It reads one word at a time and keeps a running memory."
Verdict        Half true: reading is parallel, writing is sequential.
RNN            One hidden state passed step to step, like a relay with one bag.
Self-attention Every token looks at every other token directly, in one step.
Query / key /  The question a token asks / the label each token offers /
value          the content it hands over. Weights = softmax(q·k / √d_k).
Multi-head     8 heads in the paper, each free to track a different relation.
Positions      Added as sine and cosine patterns, or learned; results nearly identical.
Induction head [A][B] ... [A] -> [B]: attention copying a pattern from earlier in the prompt.

My view: the most useful thing this paper gives a non-specialist is the split between reading and writing. Once you hold that, a lot of everyday behaviour makes sense: why a model can connect a fact on page one to a question on page fifty, why long prompts cost more, and why the answer streams.

One honest gap: the paper explains how information can move between tokens, not what the model does with it. Saying “the head for ‘it’ looked at ‘animal’” is a description of a step, not an account of understanding. That gap is exactly what the next part’s papers try to close.

What did we learn?

Before 2017, leading language systems were RNNs that read one token per step and squeezed everything into a single running hidden state.
The Transformer, from Google’s “Attention Is All You Need”, drops recurrence: every token attends to every other token in one step, so a whole prompt is read in parallel.
One attention step compares a token’s query with every key, turns the scores into weights with softmax, and mixes the values; in the toy example “it” ends up 88% “animal”.
Multi-head attention runs several of these side by side, and positional encoding tells the model word order, which attention alone cannot see.
Anthropic’s later work shows attention heads move information between tokens, and induction heads copy patterns such as [A][B] … [A] -> [B].
The myth is half true: models read in parallel but still write one token at a time.

What comes next?

If a model writes one token at a time, is it “just predicting the next word”, with no plan? Part 2, Research papers, myth by myth, Part 2: ‘It only predicts the next word’, looks inside a real model with Anthropic’s 2025 circuit-tracing work and finds it choosing a rhyme before it writes the line.

Sources

Vaswani et al., Attention Is All You Need, arXiv 1706.03762 (published June 2017; HTML version with the appendix figures)
Jakob Uszkoreit, Google Research, Transformer: A Novel Neural Network Architecture for Language Understanding (published August 2017)
Elhage et al., Anthropic, A Mathematical Framework for Transformer Circuits (published December 2021)
Olsson et al., Anthropic, In-context Learning and Induction Heads (published March 2022)

Opinions are my own. Every fact in this post comes from the public documentation linked above.