dimhold.by
← Writing

The same question to opus 5 gave 21 different answers in 64 calls, so I went looking for where the difference starts

October 17, 2026 · 15 min read

On 20 August I froze 6 tasks in a file and started asking claude-opus-5 the same questions through the Claude Code CLI. A grader checks each answer. I wanted a baseline for the “they made it dumber this week” posts, but the file showed a difference inside one sitting first. One task is 60 months of compounding MRR where the growth rate changes after month 6. The code computes 58937.5857. In 64 calls between 20 August and 12 September the model returned 21 different numbers. On the other 5 tasks it returned one answer per task. 1 call failed on a rate limit.

I always explained this to myself with temperature. The CLI does not expose it, so I never set it to 0 and I assumed that at 0 the answer would stop moving. I had also read “Defeating Nondeterminism in LLM Inference” by Horace He at Thinking Machines, from September 2025. It says that even at temperature 0 a server returns different text, because its kernels are not invariant to batch size. The batch depends on other people’s traffic. I believed that too and did not check it either. So this time I took a model where nothing is hidden and looked for the place where the difference starts.

Sampling off, float32

Qwen2.5 0.5B Instruct, transformers 5.16.1 and torch 2.14.0 on 4 cores of an AMD EPYC, greedy decoding, so no sampling at all. Qwen ships a repetition penalty of 1.1 in its generation config and I left it on in every run. 8 prompts with long answers, from hash tables to baking bread.

In float32 nothing moved. 5 repeats in a row gave the same logits bit for bit. 1, 2 and 4 threads also gave the same bits, which I did not expect. A batch of 2, 4 or 8 did change the logits in every one of the 8 prompts, but only in the fifth decimal place. The tokens stayed the same for all 200 and in a second series for all 400.

Greedy decoding takes the token with the highest score, so a flip needs the gap to the second token to be smaller than the noise. The smallest gap in 8 answers of up to 400 tokens was 0.000122, on the sum prompt. In a batch of 8 the same step had 0.000109. The batch never moved the gap by more than 0.00004.

bfloat16

I loaded the same weights in bfloat16. A repeated run at batch 1 again gave the same tokens, so the model is still deterministic with itself.

bfloat16 keeps 8 bits of precision and stores a logit around 17 with a step of 0.125. 2 tokens can get exactly the same score. float32 did not have one in 2782 steps, while in bfloat16 the top 2 were tied 38 times in 2709 steps. The bread prompt started with a tie, Certainly and Sure both at 17.625, while float32 had them at 17.693 and 17.598. Alone the run began with “Sure! Here’s a simple guide”. In a batch of 8 it began with “Certainly! Making bread at home”.

A batch in a plain forward pass moved the first logits by up to 0.000047 in float32 and by up to 0.29 in bfloat16. Through generate, 4 prompts of 8 went a different way in the batch of 8. float32 against bfloat16, both at batch 1, differed in 7 prompts of 8. The sum prompt asks for the sum of the numbers from 1 to 250 divisible by 3 or 7, which is 13482. Both runs gave the count instead of the sum. Only bfloat16 got the count right with 107, while float32 slipped on the multiples of 21 and wrote 106.

same prompt, sampling off: where the tokens first differ float32 repeat threads float32 batch 2, 4, 8 float32 batch 8 bfloat16 repeat bfloat16 batch 8 float32 against bfloat16 hash table = = = = = 84 URL in a browser = = = = 47 11 money runs out = = = = = = lighthouse story = = = = 121 121 PostgreSQL or SQLite = = = = = 22 sum, 3 or 7 = = = = = 219 float addition = = = = 134 144 bread = = = = 0 0 same tokens differ from this token
Columns 1 and 2 generate 200 new tokens, the rest 400. Token numbers count from 0. The batch changed the first logits in both formats: by up to 0.000047 in float32 and up to 0.29 in bfloat16. Only bfloat16 turned that into different text.

Here I got stuck for a while. 2 of the 4 batch flips happened where the gap between the top 2 tokens was 1.375, far from any tie. I could not see how noise of 0.29 overturns that. The answer was in my own script. I took the gaps from the raw logits, but the decoder chooses after the repetition penalty. In float32 the raw top token was not the chosen one in 396 steps of 2782. I ran the series again with the gap taken after the penalty and got the same tokens in all 32 runs. At those 2 steps the gap after the penalty was 0.09 alone and 0.034 and 0.023 in the batch. The other 2 flips were exact ties.

Back to opus

Then I asked claude-opus-5 to explain hash table collisions in about 150 words, 20 times. CLI 2.1.283, thinking off, no tools, an empty MCP config, a plain system prompt and a directory without any CLAUDE.md above it. I got 20 different texts, all starting with “A hash table” and covering chaining and open addressing. In 132 pairs of 190 the 2 texts share 11 words or less at the start.

After the local runs I read the 21 arithmetic answers as groups and not as noise. Those calls went through CLI 2.1.237 to 2.1.269 with thinking on. The grader accepts a relative error of 1e-6, about 6 cents. 46 of 64 answers are inside it. 31 of them match it to the cent, 16 rounded to 58937.59 and 15 truncated to 58937.58. Of the 18 outside the tolerance, 6 sit near 58928.2 and 7 miss by less than 20 cents.

opus 5, the same MRR question, 64 calls all answers 1 6 1 2 54 58905 58910 58915 58920 58925 58930 58935 58940 the group near the true value, by cent true value 58937.5857 grader tolerance 15 16 6 4 58937.2 58937.3 58937.4 58937.5 58937.6 58937.7
Filled dots passed the grader, relative error up to 1e-6. 20 August to 12 September 2026, 64 calls, 21 different numbers.

I cannot take the served model apart the way I took the small one. The CLI uses whatever sampling the service has, so sampling and batching end up in one number with everything else on the server. The local runs show only what is enough to change the text with sampling off: a tie in bfloat16 or neighbours in the batch. My guess about the arithmetic is that the model computes it in a long hidden reasoning, 691 to 6632 output tokens per call. An early different step would take the rest with it. The answers come in groups and that fits the guess, but I have not tested it.

For my own tools the result is modest. The grader already compares numbers with a tolerance and I keep it. The critic that grades my drafts is noisy in the same way. On one draft it gave the same failing verdict 30 times of 30. It named 160 different complaints, 114 of them only once.

What I didn’t check

  • A GPU. All local runs were on a CPU and the kernels Thinking Machines describe are GPU kernels. Yuan and colleagues (arXiv 2506.09501) measured bf16 against fp32 and batch size on GPUs with 7B models. My bfloat16 result is the same precision effect on a much smaller model.
  • Served models through an API at scale. Atil and colleagues (arXiv 2408.04667) measured that spread for 5 models.
  • Temperature 0 on the served model. The CLI has no such option.
  • Why a batched forward pass keeps the bread tie at 17.625 while the batch of 8 through generate breaks it by 0.125. The longest prompt needs no padding and behaves the same way. In a batched forward pass it is equal bit for bit, through generate it differs in the fifth decimal.
  • What calculation gives 58928.2. It came back 6 times, so it is probably a stable wrong path, but I did not find the step.