Last week the InstructGPT paper went up on arXiv. OpenAI had put it on their site in January. The main claim is that people preferred answers of a 1.3B model tuned with human feedback over answers of the 175B GPT-3. I read it and realized that I use the words “trained with human feedback” and cannot say what the feedback physically is. Somebody clicks something and the answers get better. That is the level of magic I had.
Before InstructGPT the same group did this on Reddit summaries. In 2020 they put the raw clicks online, so I downloaded them and tried to repeat the mechanism on 4 CPU cores.
What a click is
The data is OpenAI’s summarize_from_feedback. Each row is a Reddit post, 2 short summaries of it and the one a person picked. Some rows also have a confidence score and notes, but no reason for the choice. The training part has 92,858 such comparisons and the validation part 83,802 on different posts.
Some pairs were labeled by several people. On a post about a girl he met on vacation 1.5 years ago and still has feelings for, 11 people compared the same 2 summaries and split 6 to 5. Over all repeated pairs in validation 2 labelers agree on 72.5% of comparisons, on the training part 74.7%. So “what humans want” is a vote. About every fourth time the second human votes the other way.
The cheapest reward model
Before training anything I wrote down the expectations. The first one was that a dumb rule already gets more than 55%. The rule is “the longer summary wins”. On validation it is right 58.0% of the time, ties left out. On average the picked summary has 28.4 words and the rejected one 26.6. Stiennon and colleagues, who collected this data, already controlled for length in their paper.
Then a linear model, logistic regression from scikit-learn 1.0.2. Every summary becomes a bag of words and word pairs plus the logarithm of its length. It learns from the difference between the 2 summaries which one was picked. This is the model of Bradley and Terry from 1952, the same formula as the logistic version of chess Elo ratings. My first run was worse than length alone, 56.9%, because I did not tune the regularization. After tuning it on a held out part of the training data it gave 58.3% on validation. Length alone in the same model gives 57.7%. So 103,587 word features added half a point. The weights are readable but not exciting. “ex”, “relationship”, “boyfriend”, “wedding” are positive. Many of the top words are about relationships and sex. “edit” is the strongest negative. An early OpenAI model sometimes invented a Reddit style “EDIT:” line at the end of its summary. People voted those down.
A real reward model, a small one
The papers use a big transformer with one number on top, so I wanted GPT-2 small. On this machine 1 training pair took about 7 seconds, so 16,000 pairs would take more than a day. I took MiniLM with 6 layers and 22M parameters, a sentence encoder from 2021. It reads the post and the summary together and has 1 linear output on top. The loss is the same as in the papers. It is minus log sigmoid of the reward of the picked summary minus the reward of the rejected one. I trained it on 16,000 pairs in 1 pass. The scores of the final model I checked again on torch 1.11 and transformers 4.17.
I tested it on 3,000 validation pairs. There it is right 60.3% of the time. On the same pairs length gets 57.5% and the linear model 58.1%. After 2,000 training pairs it was 57.8%, after 8,000 59.0%. It was still growing, but the learning rate also went down to zero at the end, so these middle points are not clean. On the 772 pairs where the 2 summaries differ in length by 10% or less, length cannot help. There it gets 58.4%, so it learned something besides length.
Among these pairs where the lengths differ, humans picked the longer summary in 57.5%. The reward model prefers the longer one in 69.0%. After the first 2,000 pairs it was 74.8%, 3 times in 4. Whatever goes with length, more detail or more copying, it rewards more often than the humans who clicked.
Pulling the answer toward the reward
InstructGPT pulls the answer toward the reward with reinforcement learning, PPO, which I cannot run here. The simpler way is best of n: generate n answers and keep the one with the highest reward. WebGPT used it in December.
My policy is plain GPT-2 small without any tuning. The prompt is the post and then “TL;DR:”. I took 16 samples per post at temperature 0.7, at most 48 tokens, cut at the first line break, for 100 validation posts. 134 of the 1,600 summaries are empty. For each post I counted over all subsets of n samples which one my reward model would keep.
| n | words | trigrams copied from the post | empty |
|---|---|---|---|
| 1 | 21.5 | 6.2% | 8.4% |
| 4 | 30.0 | 10.4% | 0.4% |
| 16 | 35.4 | 16.1% | 0 |
The chosen summary grows by 65%. In 54 of 100 posts the best of 16 runs into the 48 token limit. 48 of them stop in the middle of a sentence. A random sample hits the limit in 22.
I expected more copying, Ziegler and colleagues saw whole sentences copied in 2019. With an untuned GPT-2 I got them too. 17 of the 1,600 samples have at least 5 words and at least 90% of their trigrams come from the post, most of them a straight piece of it. The reward model puts them on average at place 2.8 of 16. 7 of them won their post. One post starts “I met this girl online, we planned a date but by talking with her I saw some traits I’m not really sure I like”. The winner for it is exactly this sentence. It is on topic and it says nothing the reader did not just read.
What I take from it
The expectations about length and copying held. The reward model came at 60.3%. I expected 60 to 68 for GPT-2 small. The model I had to use is more than 5 times smaller. The linear model I expected near 60% and it gave 58.3%.
None of this is new. Stiennon and colleagues in 2020 showed that optimizing the policy too hard against the reward model makes the summaries worse for humans. Next I want to see what PPO does with a reward like this, if I find a GPU.
What I did not check
- No PPO. Best of n only shows the direction the reward pulls.
- 1 training run, 1 seed, 16,000 of 92,858 pairs.
- The policy is untuned GPT-2 small, so the reward model chooses between texts of a model nobody tuned for summaries.
- Nobody judged whether the chosen summaries are better for a human. Longer and more copied is what I measured.