AI papers, pictured · Part 3 · cheat sheet

RAG on one page

How AI learned to look things up before it answers

Lewis et al., Facebook AI Research, UCL and NYU, May 2020 (NeurIPS 2020) · arXiv:2005.11401 · “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”

The problem

A model like BART keeps everything it learned inside its weights (the paper calls that parametric memory). It answers closed-book: it can’t show a source, can’t learn new facts without retraining, and when it’s unsure it makes things up. The paper’s word: “hallucinations”.

BART, asked to define the middle ear“The middle ear is the part of the ear between the middle ear and the nose.” (Table 3, marked factually incorrect)

Build the librarynon-parametric memory

The December 2018 Wikipedia, cut into 100-word chunks: 21 million passages. Each one gets a list of numbers, and they all go into one fast search index (FAISS). This memory is plain text, so people can read it and edit it.

Retrieve the right pagesthe retriever (DPR)

Your question becomes a list of numbers, q(x); every passage already has one, d(z). Multiply them number by number and add it up: that’s the match score. A Maximum Inner Product Search (MIPS) grabs the top-K passages (K = 5 or 10 in training).

The paper, section 2.2

p_{η} (z | x) \propto \exp ({d (z)}^{⊤} q (x))

d (z) = {BERT}_{d} (z), q (x) = {BERT}_{q} (x)

In words: the passages whose numbers line up best with your question’s win.

Augment the questionadd the passage to the input

Each retrieved passage is simply joined to your question (concatenated), so the writer sees both.

Generate the answerthe generator (BART-large, 400M)

The generator writes a draft from each passage, and the drafts are blended by how much the retriever trusts each passage. Two ways to blend:

RAG-Sequence: one passage per answer

p (y | x) \approx \sum_{z \in top- k} p_{η} (z | x) p_{θ} (y | x, z)

RAG-Token: a passage per word

p (y | x) \approx \prod_{i}^{N} \sum_{z \in top- k} p_{η} (z | x) p_{θ} (y_{i} | x, z, y_{1 : i - 1})

p (y | x) \approx \prod_{i}^{N} \sum_{z \in top- k} p_{η} (z | x)

\times p_{θ} (y_{i} | x, z, y_{1 : i - 1})

ΣBlend the cards. Each draft counts as much as the retriever trusts its passage, p_η(z | x).
ΠFor every word. RAG-Token re-blends at each word, so one answer can draw on several passages. Given “Hemingway”, one passage brought “The Sun Also Rises” and another “A Farewell to Arms” (Figure 2).

Trained end to end: it only ever saw questions and answers. Nobody told it which passages were right; the question side of the retriever and BART learned together (the passage index stays fixed).

What it got

44.5

exact match on Natural Questions (real Google searches), a new state of the art; T5-11B answering from memory: 34.5

626M vs 11B

trainable parameters: RAG vs T5-11B, about 17× fewer

42.7% vs 7.1%

how often judges found RAG vs BART more factual, over 452 Jeopardy question pairs

70% · 68%

world leaders it got right after swapping in the 2016 or 2018 Wikipedia, with no retraining (12% and 4% with the wrong year’s)