Explore versus exploit

Use when

• The same kind of choice repeats and you learn from each round.
• You suspect you have settled on something adequate without checking the alternatives.
• A new option appears and you must decide whether it deserves a trial.

Do not use when

• The choice happens once. There is nothing to exploit.
• Exploring is expensive and irreversible — the model assumes cheap trials.

Inputs required

• A repeating choice
• Some record of how options have performed
• A horizon

Procedure

1. 01

   Count the remaining rounds

   This is the variable that sets the answer. Many rounds left: explore. Few: exploit. The horizon, not the uncertainty, does most of the work.

2. 02

   Choose a rule

   Simplest workable rule: explore a fixed fraction of the time — say one round in ten — and take the best-known option otherwise. More refined rules exist; the fixed fraction captures most of the benefit.

3. 03

   Prefer optimism about the untried

   Treat an option with no data as better than it probably is. This biases toward trying things, which is correct when there is time to recover from a bad trial.

4. 04

   Shift as the horizon closes

   Near the end, stop exploring entirely. Information has no value once there are no rounds left to use it in.

5. 05

   Record every trial

   Exploration only pays if the result is kept. An untracked trial costs the same and returns nothing.