LATENT LIBRARY DEEP LEARNING Inside theTransformer How Large Language Models Read,Think and Write LATENT LIBRARY EDITORIAL

Inside the Transformer

How Large Language Models Read, Think and Write

Latent Library Editorial

IntroductionIntroduction

When you type a question into an AI assistant, the reply seems to arrive whole, like a message from a well-read person. Underneath, something far more mechanical and far more interesting is happening. Your text is cut into pieces, each piece becomes a list of numbers, those numbers flow through dozens of identical layers that let every piece "look at" every other piece, and at the end the system produces a probability for every possible next piece of text. It picks one, adds it to the text, and does the whole thing again.

The architecture that does this is the transformer, introduced by researchers in a 2017 paper titled "Attention Is All You Need". Almost every large language model since has been built on it, along with many image, audio and multimodal systems.

This book explains the transformer from the ground up, in the order data flows through it:

  1. Tokens: how text is cut into pieces.
  2. Embeddings: how pieces become numbers that carry meaning.
  3. Attention: how pieces exchange information, worked through with real numbers.
  4. The transformer block: the repeating unit that combines attention with other components.
  5. The residual stream and feed-forward layers: how information is carried, transformed and stored.
  6. Pretraining: how the model learns from vast amounts of text.
  7. The mechanics of training: the loss, perplexity, optimisers and the engineering of large runs.
  8. Scale: counting parameters, estimating compute and memory, and what scaling laws say.
  9. Fine-tuning: how a text predictor becomes a helpful assistant.
  10. Generation: how answers are produced, and what settings like temperature do.
  11. Efficient inference: the KV cache, quantisation, batching and other ways to make models fast and affordable.
  12. Beyond text: how the same architecture handles images, audio and several kinds of input at once.
  13. Interpretability: what researchers can and cannot yet see inside a trained model.
  14. Variations: rotary positions, encoder and decoder shapes, mixtures of experts and alternatives to attention.
  15. Reasoning, tools and retrieval: how systems built around the model extend what it can do.
  16. Measuring models: loss, benchmarks, human and model judges, and how to read a results table.
  17. Limits and frontiers: what these models cannot yet do well, and what researchers are trying.

Exercises with answers and a glossary follow the last chapter.

You need basic algebra and a willingness to follow small calculations. Every formula is explained in words, and every calculation uses numbers small enough to check by hand.

Chapter 1Tokens: Cutting Text Into Pieces

Why not characters or words?

A model needs a fixed vocabulary of units. Two obvious choices both have problems:

Modern models use subword tokens: common words are single tokens, while rare words are split into familiar pieces. "Unbelievably" might become "un", "believ", "ably". Typical vocabularies contain tens of thousands to a few hundred thousand tokens.

How a tokeniser is built: byte-pair encoding

A popular method, byte-pair encoding (BPE), builds the vocabulary from data:

  1. Start with individual characters (or bytes) as the vocabulary.
  2. Count every adjacent pair of symbols in a large text sample.
  3. Merge the most frequent pair into a new symbol and add it to the vocabulary.
  4. Repeat until the vocabulary reaches the desired size.

A toy illustration with the words low, lower, lowest, newer, newest:

After a few merges, "lowest" is tokenised as "low" + "est", and an unseen word such as "slowest" can still be built from "s" + "low" + "est".

Practical consequences

You have reached the end of the free sample.
The full book continues with 16 more chapters, plus everything else in its 80 pages.

Get the full book