How does an AI learn?
Supervised, unsupervised, and reinforcement learning — and how ChatGPT is trained.
本页目录
1. The learning loop
4 steps repeated billions of times
1. Predict — The model receives data and proposes an answer. At first its parameters are random, so the answer is nonsense.
2. Measure the error — The answer is compared to the correct one using a loss function — a number showing how wrong the model is.
3. Correct — An algorithm computes how to adjust each parameter to reduce the error (gradient descent + backpropagation).
4. Repeat — Move to the next example. The error gradually decreases.
Put the 4 steps of the learning loop in order
At the very start of training, why is a model's first prediction nonsense?
2. Gradient descent
Walking down a mountain in the fog
Imagine being lost in fog on a mountain: you can't see anything, but you feel the slope under your feet and step in the direction that goes down the most. That's gradient descent: altitude represents the error, each step adjusts the parameters to reduce it.
Backpropagation (1986)
Computing the "slope" for billions of parameters is done by backpropagation, popularized in 1986 by Rumelhart, Hinton and Williams. The learning rate is the step size: too big and you overshoot the valley; too small and it takes forever.
What year was backpropagation popularized by Rumelhart, Hinton and Williams?
A learning rate that's too large can make training "overshoot the valley" (the minimum error).
3. Three main ways to learn
Supervised, unsupervised, reinforcement
Supervised: labeled examples are provided ("this image is a cat"). Unsupervised/self-supervised: no correct answer is given, the model finds structure on its own (or a word is hidden and it must guess it — this is how LLMs are pre-trained). Reinforcement: an agent receives rewards or penalties and learns the strategy that maximizes its gains — this is how AlphaGo beat Lee Sedol in March 2016.
Match each example to its learning type
Which variant of unsupervised learning hides a word in a sentence for the model to guess?
4. And what about ChatGPT?
Training in 3 stages
1. Self-supervised pre-training — The model reads huge volumes of text and learns to predict the next token. The most expensive stage.
2. Supervised fine-tuning — Humans write examples of good answers. OpenAI introduced this approach with InstructGPT in 2022.
3. RLHF (human feedback) — Evaluators rank several of the model's answers; a "reward model" learns these preferences and fine-tunes the assistant through reinforcement.
Put these 3 training stages of an assistant like ChatGPT in order
What name did OpenAI give to the supervised fine-tuning approach introduced in 2022?
5. Common learning pitfalls
4 classic pitfalls
Overfitting: the model memorizes instead of generalizing. Data bias: a model reproduces the imbalances in its training data. Label quality: "garbage in, garbage out". Energy cost: training the largest models uses thousands of processors for weeks.
How do you detect a model's overfitting?
If training examples are mislabeled, the model can learn errors from them.
Boss challenge: which type of learning?
Sort each situation by the type of learning it illustrates.