Sampling off or a fixed seed should give the same answer to the same prompt. That is how I have treated it all year, without checking. When a local generation did not match something I had saved earlier, I said “sampling” and ran it again, without asking if maybe something under the model had changed instead. On my machine something does change every few weeks, because I run pip install -U whenever a new transformers release has a fix I want. There was no single broken output that sent me to look. It was the end of the year and I wanted to know what a year of these upgrades did.
I did not keep my January outputs, so I rebuilt the January venv from what was current then. That is torch 1.10.1 from 15 December 2021 and transformers 4.15.0 from 22 December 2021. The December venv is torch 1.13.1, released last Thursday, with transformers 4.25.1. Python 3.9 in both, CPU only, float32, 4 cores of a virtual machine on an AMD EPYC 7713. The weights are GPT-2 small, 124M parameters, the same pytorch_model.bin file for every run.
The text did not move
I used 8 prompts, from “The capital of France is” to “Q: What is 17 times 23?”, with 60 new tokens each. Every prompt ran greedy in my own argmax loop, greedy through generate() and with sampling (top_k 50, torch.manual_seed(1234) before each prompt). I expected sampling to break first, because it depends on the whole distribution and not only on which token is on top.
Nothing broke. January and December gave the same tokens for all 8 prompts in every mode, sampling included. To the arithmetic question GPT-2 answers ” 17 times 23.” and then asks the question again, in a loop, in both venvs.
The logits did change. A repeated run in the same venv gave the same bits. December against January differs in 6 of 8 prompts, by at most 0.000107 on the first step.
Then I swapped one package at a time. transformers from 4.15.0 to 4.25.1 under the same torch changed zero bits. It did change the GPT-2 attention code in between, the mask value and how the scale is divided, but neither change moved a bit here. torch 1.11.0 gives exactly the bits of 1.10.1 and 1.12.1 gives exactly the bits of 1.13.1. I did not try 1.12.0, so the whole year of my venv sits somewhere between 1.11.0 and 1.12.1.
Where the step is
2 prompts kept their bits even across that step and I could not see why. I spent some time on MKL first. With MKL_VERBOSE=1 torch shows MKL 2020.0 doing the SGEMM calls, so MKL does run the matrix multiplications. But setting MKL_ENABLE_INSTRUCTIONS=SSE4_2 changed nothing at all, not one bit in either venv and I still do not know why.
Then I tried ATEN_CPU_CAPABILITY=default, which makes torch use its generic kernels instead of the AVX2 ones, like on an older CPU. The logits moved by up to 0.000122 against the normal December run. January and December became identical bit for bit. The change was inside some AVX2 kernel.
I dumped every layer on the prompt with the biggest change, the arithmetic one, 11 tokens. Embeddings and the first layer norm are the same. The first difference is the output of attention in block 0, 0.0000019, which grows to 0.00032 by the last block. Then I ran each operation alone on a fixed random input. Only softmax differs.
The 2 prompts that kept their bits have 5 and 7 tokens. All the others have 8 or more. I ran softmax on rows of every length from 1 to 64. Lengths 1 to 7 give the same bits in both versions. From 8 on, every length has rows that differ, except 23, which I think is luck.
8 floats is one AVX2 register. The diff of aten/src/ATen/cpu/vec/functional_base.h between the 2 tags shows the rest. Softmax sums the exponents with reduce_all, which keeps 8 partial sums in the register lanes and at the end folds them into one number. In 1.11.0 this fold goes lane after lane: lane 1 is added to lane 0, then lane 2 and so on up to lane 7. 1.12.1 added a faster fold for AVX2:
// 128-bit shuffle
Vec v1 = _mm256_permute2f128_ps(v, v, 0x1);
v = vec_fun(v, v1);
// 64-bit shuffle
v1 = _mm256_shuffle_ps(v, v, 0x4E);
v = vec_fun(v, v1);
// 32-bit shuffle
v1 = _mm256_shuffle_ps(v, v, 0xB1);
v = vec_fun(v, v1);It adds lanes 4 apart, then 2 apart, then neighbours. The numbers are the same but the order of additions is different and in float that can change the last bit. Rows shorter than one register never reach the new code. The generic kernels do not have it at all. Neither do builds made with MSVC, because the new fold sits under a GCC check, so on Windows this step probably does not exist.
Why the tokens survived
A shift of 0.0001 only matters if the first and second token are closer than that. 8 prompts is too few to say anything about it, so I took 200 sentence openings from Alice in Wonderland and generated 100 greedy tokens for each, 20,000 steps per run.
I did 6 runs and they give only 3 different sets of gaps. December repeated and December with 1 thread are the same to the bit. Both generic kernel runs are the same too and January is the third set. All 200 continuations were the same in all 6.
The smallest gap between the first and second token was 0.000366, at step 14 of the opening that starts with “So she set to work”. It was the same to the last bit in every run. No condition changed any gap by more than 0.000172, so the closest step had about twice the margin it needed.
Greedy GPT-2 falls into loops quickly and from the numbers a loop looks very sure of itself: the median gap is 0.70 over the first 10 steps and 6.27 over the last 10. A model that hesitates more would have more steps near zero, where the same 0.0001 has more chances to land.
1 thread instead of 4 also moved the first step logits, by up to 0.0000229 in both venvs and changed no token. I did not look at why.
What I did not check
- GPU and float16, where the reduction order depends on other kernels that I did not read.
- Batching on a server, which is what users of a hosted API meet. My runs are batch 1, alone on the machine.
- Another real CPU. AVX-512 has its own fold in the same file. I only imitated an older CPU with an environment variable.
- The commit itself. I read the diff between tags v1.11.0 and v1.12.1, not the pull request.
- Bigger models and longer outputs. 20,000 steps of a looping 124M model is not a page of text from something larger.
On this machine I can stop blaming sampling for a changed output, at least with GPT-2. But the closest step had only twice the margin and I would not promise the same for a model that loops less.