In January Jason Wei and his colleagues at Google put out a paper about chain of thought prompting. This week a second version came out. You give a model 8 example problems, but each example has the solution written out in words before the answer: “There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6.” On GSM8K, a set of school math word problems from OpenAI, this moves PaLM 540B from 17.9% to 56.5%.
There is one sentence in the paper that I read and skipped. Models smaller than about 100B parameters “produced fluent but illogical chains of thought, leading to lower performance than standard prompting”. I took it as natural that writing the steps helps, because it helps people. The small models looked like a detail. But the biggest model I can run at home is about 130 times smaller than that limit. For me this sentence is the part that matters. I wanted to see what a fluent but illogical chain looks like and what it does with the answer.
The setup
There is no GPU on this machine, only 4 cores of an AMD EPYC 7713 in a virtual machine and a 5 GiB memory limit. I took GPT-2 in 3 sizes: small with 124M parameters, medium with 355M and large with 774M. Everything runs on Python 3.9, torch 1.11 and transformers 4.18, in float32 with greedy decoding.
I wanted a 4th model, the 1.3B one from EleutherAI. In float32 its weights alone are 5.3 GB and torch 1.11 holds the checkpoint and the model at once, so it does not fit. In bfloat16 it fits, but this CPU has no hardware support for it and 40 tokens did not finish in 4 minutes. At that speed one pass over 500 problems would take days, so GPT-2 large stayed the biggest.
The prompt is the 8 examples from the paper, word for word. In standard mode each example ends with “The answer is N.” and nothing else. In chain mode the written solution comes before it. The questions:
- 200 problems from the GSM8K test set, chosen with seed 2022;
- 300 problems I generated, 100 each with 1, 2 and 3 steps of adding or subtracting small numbers. For example: “Sam has 9 pencils. He loses 5 pencils. How many pencils does Sam have now?”
GSM8K is too hard for these models anyway, so I wanted problems where a single step is trivial. The whole run took about 3 hours and 15 minutes. Most of it went to chain mode. GPT-2 small often loops (“So he has 16 glasses. So he has 16 glasses.”) until it hits my limit of 160 new tokens. GPT-2 small hit the limit on 215 of 500 problems.
The chain does not help
Out of 500 problems, standard mode was right 16, 12 and 19 times for small, medium and large. Chain mode was right 4, 11 and 10 times. For small the difference holds on paired answers (15 problems only standard got right, 3 only chain, McNemar p = 0.0075). For medium and large it does not. So I agree with the paper: at this size the chain does not help and for the smallest model it hurts. But both modes are right only a few percent of the time. Medium in standard mode answered all 100 of the 1 step problems with a number copied from the question and got none of them right.
What the chains look like
Here is the real answer of GPT-2 large to the pencil problem from above:
Sam started with 9 pencils. After losing 5 pencils, he had
9 - 5 = 6. After losing 2 more, he had6 - 2 = 4. The answer is 4.
I found it while looking through the right answers, where I expected the good chains to be. The answer is right. But 9 - 5 is not 6 and the problem has no second loss at all. “After losing 2 more” comes from the golf ball example in the prompt, where Michael loses 2 more balls on Wednesday. So the 2 wrong steps cancel each other and the number comes out right.
I counted every equation the models wrote in a form like a - b = c. Small got 7 of 292 right, medium 68 of 1078, large 80 of 550. On the 1 step problems large wrote the right expression, for example 9 - 5, in 28 of 100 cases. In 26 of them the result after it was wrong. The phrase “2 more” shows up in 19 of the 100 answers of large and in 22 of medium, every time in a question that does not have it. In the 2 cases where large did the subtraction right, this invented step broke the answer afterwards.
Then I looked at the right answers in chain mode. There are 25 over the 3 models. Only 1 contains the right step for its problem: medium writes 20 - 5 = 15 and right after it 20 - 15 = 15. In 7 there is no equation at all and the answer is a number repeated from the question or from a loop. The rest get there through wrong steps, like the pencils. Once large writes 9 - 3 = 6 and then 6 + 6 = 12. Both are correct, but in that problem Sam finds 3 more cookies, so neither step belongs to it.
Where my expectation was wrong
Before the run I wrote down that the final answer would match the last number of the chain in less than half of the cases. I thought a small model writes some chain and then answers something unrelated, so the chain is decoration.
It went the other way. In chain mode the answer equals the right side of the last equation the model wrote. For small this is 188 of the 189 answers with at least 1 equation, for medium 393 of 414 and for large 360 of 362. If I compute that last expression myself and compare, the answer equals the computed value only 6, 42 and 57 times.
So “illogical” means something different from what I thought. The answer follows the chain very closely, so a wrong 9 - 5 = 6 goes straight into the answer. The chain also costs time. On this CPU chain mode was 3.6 times slower than standard mode for large, simply because it writes more tokens. For small it was 8.6 times and the loops took 1418 of 1899 seconds.
What I did not check
- Models bigger than 774M. GPT-J 6B is 24 GB in float32 and does not fit here, so the 100B side of the paper I can only read about.
- Sampling many chains and voting. Wang and colleagues showed in March that it helps big models a lot. With greedy decoding I get one chain per problem.
- Other examples in the prompt. The “2 more” leak suggests that the model probably copies the shape of the examples, so different examples would give different mistakes. I did not test how much.
- A calculator. The paper already has one: it fixes the equations after generation. For LaMDA 137B on GSM8K chain mode goes from 14.8% to 19.5%. My models follow the last written number so closely that I would expect it to help a little, but it would not fix a wrong operation or the “2 more” step. I did not run it.
Each number here is from 1 greedy run, so the texts repeat exactly (the timings do not). But a few percent right out of 500 is a small base for anything finer than “it does not help”. For the models I can run I will keep the plain answer format. I am curious what a 6B model does with the same 8 examples, if I find a machine for it.