How does ChatGPT work?
Tokens, Transformer, attention, context window: large language models explained.
本页目录
1. Predicting the next word
A simple goal, surprising results
A large language model (LLM) is trained with a modest goal: predict what comes next. Given "The cat is sleeping on the", it estimates the probability of each possible next word ("couch", "roof"…), picks one, then repeats. Entire paragraphs emerge from this repetition.
To correctly predict the next word across billions of texts, the model must capture grammar, facts, styles — which is why such a simple task produces capabilities that seem to go far beyond it.
What is the basic training objective of a large language model?
A language model generates a whole answer in one go, not word by word.
2. Tokens: how a machine reads text
From text to numbers
A computer only understands numbers. Text is split into tokens (whole words, word pieces, or punctuation), each turned into a list of numbers called an embedding, which places the word on a kind of map of meaning — popularized by word2vec (2013).
A unit that costs money
A token represents roughly three-quarters of a word in English; French or Chinese generally need more tokens for the same text. Models charge for, and limit, usage by tokens.
Text is first split into small pieces called ….
French text generally contains fewer tokens than the same text in English.
3. The Transformer and attention
The 2017 publication
On June 12, 2017, eight Google researchers published "Attention Is All You Need", introducing the Transformer architecture. Its central idea: the attention mechanism, which connects every word to every other word in the context to decide which ones matter most.
Why it works
Two advantages: the Transformer processes every word in parallel (fully exploiting GPUs) and handles long-range dependencies between distant words well. This paper has been cited more than 250,000 times.
What year was the Transformer architecture published?
In "The bank refused the loan because it was being cautious", what does the attention mechanism connect "it" to?
A key advantage of the Transformer is processing every word of a sentence in parallel rather than one at a time.
4. The context window
The model's working memory
A model can only "see" a limited amount of text at a time: its context window (your question + conversation history + the answer in progress). GPT-3 (2020) was limited to 2,048 tokens; recent models accept much more. Beyond this window, the model "forgets" what came before — it has no permanent memory unless the tool adds a dedicated memory feature.
How many tokens was GPT-3's (2020) context window limited to?
A language model keeps every past conversation in memory, even from months ago.
5. Why ChatGPT sometimes gets it wrong
Plausible, not verified
A hallucination is a fluent, convincing but false answer. It follows directly from how the model works: it produces plausible text, not verified text. In 2023, lawyers filed fake court decisions invented by ChatGPT with a New York court ($5,000 fine).
Ask for sources and check them. Turn on web search when available. Rephrase the question. Never use it as the sole source for an important decision.
Why can a language model "hallucinate" false information?
ChatGPT browses the internet in real time for every answer it gives.
Boss challenge: true or false about LLMs?
Sort each statement about how large language models work.