Training — how AI learns | How AI Works

01 · The only objective

Training is a guessing game — play it

A model learns by predicting the next on trillions of examples, graded by . Play four rounds and notice how the shape of the distribution — sharp or flat — tells you what the model knows.

Round 1/4 — you are the model. Pick the next token.

The capital of France is▌

02 · Three phases

Predictor → assistant

Modern assistants are built in three passes: , supervised , and .

1 · Pretraining

trillions of tokens · months · thousands of GPUs

The model reads a huge slice of the internet and is graded on one game: guess the next token. Playing it well forces it to absorb grammar, facts, styles and reasoning patterns.

2 · Supervised fine-tuning

~10⁴–10⁶ curated examples · days

Humans write examples of great answers. The model learns the shape of being an assistant: answer the question, be clear, stop when done.

3 · Preference tuning (RLHF)

human rankings → reward model → RL

Humans compare pairs of answers and pick the better one. The model is tuned toward answers people prefer — more helpful, more honest, less harmful.

Pretraining loss vs tokens seen (log scale) — milestones appear as loss falls

Illustrative curve following the power-law shape of published training runs.

loss 5.4 · Spelling & frequent words

loss 4.3 · Grammar becomes reliable

loss 3.5 · Facts stick: capitals, dates

loss 2.9 · Code starts compiling

loss 2.5 · Multi-step reasoning emerges

loss 2.2 · Tool use & instructions

Same prompt — "How do I make my Wi-Fi faster?" — three checkpoints:

How do I make my laptop boot faster? How do I extend battery life? These are the questions readers send us most often. In this week's column…

Pure continuation: the prompt looks like a listicle, so it writes more listicle. Nobody taught it to answer.

03 · Human preferences as a signal

You rank, the model learns

RLHF turns millions of human comparisons into a steering force. Try one comparison yourself.

You are the human rater. Which answer is better?

"My code throws 'IndexError: list index out of range'. Help?"

Knowledge check

Did it stick?

1/3What single objective drives pretraining?

2/3Why did the model give a nearly flat distribution for 'the 17th digit of pi'?

3/3What does RLHF add that supervised fine-tuning doesn't?

Loading module…

Common questions this module answers

How does a large language model learn?

By next-token prediction at enormous scale: it reads trillions of tokens and is graded on guessing each next token. Getting surprise (loss) down on that simple game forces it to learn grammar, facts and reasoning patterns.

What is RLHF?

Reinforcement learning from human feedback: humans rank pairs of model answers, a reward model learns those preferences, and the LLM is tuned to produce answers the reward model scores highly — turning a raw text predictor into a helpful assistant.

What is the difference between a base model and an instruct model?

A base model just continues text — ask it a question and it may reply with more questions. An instruct model has been fine-tuned on conversations and human preferences so it actually answers, follows instructions and refuses harmful requests.