Somebody posts a benchmark, one version is 8 percent faster, the discussion moves on to why. I have posted numbers like that myself. What I had never done is find out what my own machine says when there is nothing to find, so I wrote the smallest version of that: 1 function, declared twice under 2 names, timed against each other. The true difference between them is 0 by construction.
They alternate inside each round, A then B, so that any drift in the machine hits both. 200 rounds, a loop over 200000 doubles, a quiet Linux server with 4 cores and nothing else on it.
The first round took 2.118 milliseconds. The settled median is 0.593. That is 3.57 times, the only part of this everybody already knows about.
Now the part I did not know. Take 1 round of A with the 1 round of B beside it. That is what a benchmark run once actually is. Out of 180 settled pairs, 19 of them, 10.6 percent, hand you a winner by more than 5 percent. The worst single pair puts identical code 32.9 percent ahead of itself.
So a person who runs each version once, sees 8 percent then writes it up is not doing anything unusual. They are sampling a distribution that produces a 5 percent lie 1 time in 9.
How many rounds buy how much
The fix is repetition, which everybody says. The useful question is how much repetition buys how much certainty. I cut the settled rounds into windows and asked how far the median wanders between windows of the same size.
1 round wanders 35.6 percent. 3 rounds wander 5.1. 5 rounds 2.8, 10 rounds 2.3, 30 rounds 0.2. The shape is the useful part: almost all of the benefit is in the first handful of repetitions. The last stretch from 10 to 30 is what buys the precision you need to defend a small claim.
The 2 percent that would not sit still
The medians of A and B came out 2.04 percent apart. Identical code. 2 percent is exactly the size of the difference people write blog posts about, so I went looking for the cause. My guess was position: A always ran first in the round and B second, so the second one might land on a warmer cache.
I swapped them. In the swapped order the sign flipped, minus 1.84 percent, which fits the guess. Then I ran both orders again. The normal order gave plus 2.24 percent, then minus 0.07. The swapped order gave plus 0.12.
3 runs out of 5 show the ordering effect and 2 show nothing at all. I cannot even call the explanation stable. A 2 percent difference on this machine is not a small effect I could chase with a better test. It is unattributable at this sample size. The honest report is that I do not know where it came from.
What I did not check
Whether any of this transfers. This is 1 tight numeric loop with no allocation, so the garbage collector never runs. The biggest source of jitter in a real program is absent from the whole experiment. I did not touch frequency scaling on the host, which moves cores around under exactly this kind of load. And I did not repeat any of it on a second machine, which means the noise floor I measured belongs to this box on this evening.
The narrow claim is a habit rather than a number. Before believing a difference, measure the noise floor of the setup that produced it by comparing something to itself, then treat anything under that floor as nothing. Here the floor is around 2 to 3 percent at 200 rounds. Single runs are worthless up to 33.