How is an AI built?
Data, GPUs, training costs, evaluation and deployment.
本页目录
1. A model's lifecycle
An industrial project, not programming
Building an assistant like ChatGPT isn't "programming intelligence" — it's a multi-stage project involving hundreds of people: data collection → pre-training → fine-tuning → alignment → evaluation → deployment.
Put the 6 stages of building an AI assistant in order
2. Gathering and preparing data
A massive corpus
GPT-3 (2020) was trained on roughly 300 billion tokens, drawn from a corpus of about 500 billion; over half came from Common Crawl (a public copy of much of the web).
Raw data isn't enough
It must be filtered for low-quality content and duplicates, stripped of personal information, balanced across sources and languages, and checked for legal issues — using copyrighted works to train AI is the subject of lawsuits (The New York Times v. OpenAI/Microsoft, December 2023).
Roughly how many tokens was GPT-3 (2020) trained on?
The quality of training data matters as much as its quantity.
3. Pre-training: GPUs and costs
Tens of thousands of GPUs
Pre-training relies on GPUs, capable of thousands of parallel calculations. AlexNet (2012) ran on 2 consumer graphics cards; a large model today uses tens of thousands of GPUs for weeks or months.
Published order-of-magnitude estimates
| Model | Estimated compute cost |
|---|---|
| GPT-2 (2019) | ~$50,000 |
| PaLM (2022) | ~$8 million |
| DeepSeek V3 (January 2025) | ~$5.6 million (claimed) |
Rank these models from lowest to highest estimated compute cost
What do the "scaling laws" published by OpenAI (2020) and later DeepMind's Chinchilla (2022) show?
4. Fine-tuning, alignment and evaluation
Making the model useful and safe
After pre-training, the model can "continue text" but can't yet answer like an assistant. Supervised fine-tuning trains it on examples of good conversations; alignment (RLHF) adjusts it based on human evaluators' preferences.
Benchmarks and red teaming
Before release, teams test the model with benchmarks (standardized test batteries) and red teaming (experts trying to trick it). These scores give an indication, not a guarantee of reliability in real use.
Sort each action by the stage it belongs to
A good benchmark score guarantees the model will be reliable in any real-world use.
5. Deployment, energy, and who builds AI
Open or closed
Once deployed via a site, app or API, some publishers release their model's "weights" (open models, like Mistral AI or DeepSeek), while others keep their models closed.
The energy cost
According to the IEA, data centers consumed roughly 415 TWh in 2024 (≈1.5% of global electricity), with a projection of roughly 945 TWh by 2030.
Who builds AI?
A handful of players dominate: OpenAI (founded December 2015), Anthropic (2021), Mistral AI (France, 2023), DeepSeek (China, 2023), among others. According to Stanford's AI Index 2026, over 90% of notable models come from industry rather than academia.
According to Stanford's AI Index 2026, where do over 90% of notable AI models come from?
An "open weights" model can be downloaded and run by anyone with the right hardware.
Boss challenge: which stage of building AI?
Sort each action by which stage of building an AI it belongs to.