AI papers, pictured · Part 2 · cheat sheet

InstructGPT on one page

How roughly 40 people ranking answers taught GPT-3 to help

Ouyang et al., OpenAI, 2022 · arXiv:2203.02155 · “Training language models to follow instructions with human feedback”

The problem

GPT-3 is trained to predict the next word, so it continues your prompt instead of doing what you asked. The paper calls this misaligned, and a bigger model doesn’t fix it.

Copy good examplessupervised fine-tuning

Labelers write ideal answers to prompts (about 13k of them), and GPT-3 is fine-tuned to imitate them.

Learn a judgethe reward model

The model writes K = 4 to 9 answers to a prompt and a labeler ranks them, best to worst (33k prompts). Every pair is a lesson: K choose 2, up to 36 per ranking. A 6B model learns to score answers.

The paper, equation 1

loss (θ) = - \frac{1}{(\binom{K}{2})} E [\log σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))]

loss (θ) = - \frac{1}{(\binom{K}{2})} E [\log σ (r_{θ} (x, y_{w})

- r_{θ} (x, y_{l}))]

In words: score the winner (y_w) above the loser (y_l). σ turns the score gap into the chance a person would agree.

Practice against the judgereinforcement learning

Reinforcement learning (PPO) on 31k prompts: write an answer, get a score from the reward model, adjust. The objective adds two guard rails:

The paper, equation 2

objective (φ) = E [r_{θ} (x, y) - β \log (\frac{π_{φ}^{RL} (y | x)}{π^{SFT} (y | x)})]

+ γ E [\log π_{φ}^{RL} (x_{pretrain})]

objective (φ) = E [r_{θ} (x, y)

- β \log (\frac{π_{φ}^{RL} (y | x)}{π^{SFT} (y | x)})]

+ γ E [\log π_{φ}^{RL} (x_{pretrain})]

βThe leash (a KL penalty): stay close to the step-1 model, so it can’t game the judge. The paper calls that gaming over-optimization.
γKeep practicing on the original internet text. That’s PPO-ptx, the final InstructGPT: it wins back most of the “alignment tax” on old skills (SQuAD, DROP, HellaSwag, WMT French→English).

What it got

85%

of the time, people preferred InstructGPT 175B over GPT-3 175B (give or take 3%)

1.3B > 175B

the small InstructGPT beat GPT-3, a model over 100× bigger

21% vs 41%

how often it made things up vs GPT-3, on closed-domain tasks

< 2%

extra compute: 4.9 + 60 vs GPT-3’s 3,640 petaflop/s-days