abhinavbhatia
Sheet 02 / 05 — Blogs← Archive

Research papers, myth by myth

A series that takes twelve common myths about AI models and checks each one against the research papers that tested it, from Attention Is All You Need to Anthropic's work on planning, hallucination, hidden reasoning and agent safety. Simple examples, the papers' own examples, animated diagrams and a link to every paper.

· 5 min read · ~7 min listen· Research · AI models · Interpretability · AI safety

People say a lot of confident things about AI models. It only predicts the next word. Nobody can see inside it. Hallucinations are random. Its reasoning shows how it thinks. Most of these claims were reasonable guesses a few years ago, and researchers have since tested many of them. The answers are more interesting than the myths.

This series takes one myth at a time and checks it against the papers that tested it. Each part says what the researchers did, what they found, whether the myth survives, and, just as important, what the paper did not show. I use everyday examples first, then the papers’ own examples, such as the rhyme that ends in “rabbit” which Claude plans before it writes the line.

How is each part laid out?

Every part follows the same shape, so you can jump to what you need:

  • The myth, stated plainly, and the bottom line in one or two sentences.
  • What do I need to know first? A few terms, each with an everyday example.
  • What did the researchers do, and what did they find? In plain words, step by step, with small numbers and the papers’ own examples.
  • So is the myth true? A verdict: true, half true, or false, and why.
  • What did the paper not show? The limits the authors state themselves.
  • Why does this matter if you build or use AI? A short, practical section.
  • Lock it in, My view and One honest gap, then the sources, with a link to every paper.

Terms with a dotted underline open a short definition when you hover over or tap them, and the glossary lists them all.

What do the twelve verdicts look like?

TWELVE MYTHS, TWELVE VERDICTS P01 An AI reads one word at a time HALF TRUE reads in parallel, writes one by one P02 It only predicts the next word HALF TRUE writes token by token, but plans ahead P03 Nobody can see inside a model PARTLY FALSE millions of readable features found P04 Hallucinations are random glitches FALSE known causes, and tests reward guessing P05 Chain of thought shows the real reasoning FALSE helps, but often omits what drove it P06 Bigger models leap unpredictably MOSTLY FALSE smooth gains; jumps are partly scoring P07 More context is always better FALSE the middle gets missed; shots can jailbreak P08 Human feedback teaches the truth HALF TRUE it teaches what raters prefer P09 Passed safety training, problem gone FALSE backdoors and faking can survive it P10 An agent only does what it is told FALSE under pressure, many chose harm P11 Jailbreaks cannot be stopped cheaply FALSE AS STATED classifiers cut it, at a measured cost P12 Nobody knows how AI is used FALSE private studies exist; replacement unproven Read the verdict with its limit: every paper here states what it did not show.

twelve myths and what the research says: two are half true, the rest are false or mostly false, and each verdict comes with a limit the papers state themselves

The pattern is worth noticing. Most myths are not simply wrong; they describe the interface and miss the mechanism. A model does write one token at a time, but it plans ahead. It does say “I don’t know” less than it should, but the reason is in how it was trained and tested, not chance.

Which parts are there?

Part 1: ‘An AI reads one word at a time, like we do’.
Read Part 1. Attention Is All You Need (Google, 2017), plus Anthropic’s work on transformer circuits and induction heads: how a Transformer reads every word at once, and why it still writes one token at a time.

Part 2: ‘It only predicts the next word’.
Read Part 2. Anthropic’s Tracing the thoughts of a large language model (2025): the rabbit and habit rhyme planned before the line is written, concepts shared across languages, and mental arithmetic that differs from the model’s own explanation.

Part 3: ‘Nobody can see inside a model’.
Read Part 3. Toy Models of Superposition, Towards and Scaling Monosemanticity, Golden Gate Claude and Google DeepMind’s Gemma Scope: how researchers pull readable features out of a model, and how little of it they can read so far.

Part 4: ‘Hallucinations are random glitches’.
Read Part 4. Anthropic’s circuit tracing, Language Models (Mostly) Know What They Know and OpenAI’s Why Language Models Hallucinate: a known-entity path that misfires, uncertainty the model does carry, and tests that reward guessing.

Part 5: ‘A model’s chain of thought shows how it really reached the answer’.
Read Part 5. Google’s Chain-of-Thought Prompting and Anthropic’s Reasoning models don’t always say what they think: writing out steps helps, but the written steps often leave out what actually drove the answer.

Part 6: ‘Bigger models get smarter in sudden, unpredictable leaps’.
Read Part 6. Scaling Laws for Neural Language Models, DeepMind’s Chinchilla, and the emergent abilities debate: loss falls smoothly and predictably, and some apparent leaps come from how answers are scored.

Part 7: ‘More context is always better’.
Read Part 7. Lost in the Middle, Anthropic’s Many-shot jailbreaking and Google DeepMind’s Many-Shot In-Context Learning: where a long prompt gets ignored, and how the same mechanism helps or attacks.

Part 8: ‘Human feedback teaches a model the truth’.
Read Part 8. Anthropic’s helpful and harmless RLHF paper, OpenAI’s InstructGPT, sycophancy research, model-written evaluations and Constitutional AI: feedback teaches what raters prefer, including agreeable answers.

Part 9: ‘If it passed safety training, the bad behaviour is gone’.
Read Part 9. Sleeper Agents, Alignment faking and Natural emergent misalignment from reward hacking: behaviour that survives safety training, and why tests must match the setting a model runs in.

Part 10: ‘An agent only does what it is told’.
Read Part 10. Anthropic’s Agentic Misalignment (2025): sixteen models as email agents in simulated companies, what pushed them towards harmful actions, and why controls belong outside the model.

Part 11: ‘You cannot stop jailbreaks without making the model useless’.
Read Part 11. Red Teaming Language Models to Reduce Harms and Constitutional Classifiers: a measured trade-off between attack success, refusals of harmless requests and compute.

Part 12: ‘Nobody really knows how people use AI, and it is mostly replacing jobs’.
Read Part 12. Clio, the Anthropic Economic Index and Values in the wild: privacy-preserving evidence of real use, and what it does and does not show about jobs.

Where should I start?

If you are new to how models work, read Parts 1 to 4 in order: how a Transformer reads, what it computes, how researchers see inside it, and why it makes things up. If you build with models, Parts 5, 7 and 8 explain the behaviour you will meet every day: reasoning you cannot take at face value, long prompts, and agreeable answers. If you deploy agents, Parts 9 to 11 are about safety training, agents under pressure and guardrails. Part 12 is about how AI is used in practice.

For the engineering side of building agents, I recommend Anthropic’s Building effective agents and my own Production-ready agents series.

What comes next?

This is a living series. I add a part when an important paper changes what we can say about a myth, and I update the verdicts when newer work changes them. Start with Part 1: ‘An AI reads one word at a time, like we do’.

Opinions are my own. Every fact in this series comes from the public papers and pages linked in each part.