Load average 8 on a 4 core box, so the box needs more cores. I have said that sentence in a meeting. Nobody objected. What I had never done is make the number go up on purpose in more than 1 way, so I did that on a quiet 4 core server with nothing else on it.
The first run puts 4 processes into a pure arithmetic loop, which is as close to a definition of processor load as I can build. The second run starts 8 processes writing to disk. Both push the load average past 8. Only 1 is about the processor.
The number lags by more than people think
Before the interesting part, the boring part that turned out not to be boring. The 4 loops start at a load of 0.18. The processor sits at 100 percent from the first second, so the machine is fully busy immediately.
After a full minute of 4 saturated cores the 1 minute average reads 2.67. Not 4. The number is a damped average with a 60 second constant, so a minute in you are seeing about 63 percent of what is happening. The rest of it only arrives after 3 or 4 minutes.
That cuts both ways during an incident. A machine that just went under water looks 2 thirds as bad as it is. A machine you just rescued looks bad for another 3 minutes.
The same 8, built out of different things
Then the disk. 8 processes writing 300 megabytes each in a loop, bypassing the page cache so the writes actually go to the device.
Load 8.10, with the processor doing real work 36.5 percent of the time. It waits on the disk 63.3 percent. 7 of the 8 writers sit in uninterruptible sleep at any moment, which is the state Linux counts alongside the runnable ones. That decision is the whole reason the number is not a processor metric.
The pair is the point. 8.19 and 8.10 are the same alarm on a dashboard. One of them is fixed by more cores. The other one is fixed by a faster disk or by writing less, while the cores sit 2 thirds idle.
Where I tripped
My first attempt at the disk half did not measure the disk. I had the 8 writers call fdatasync but let the page cache take the writes. The run came back with a load of 8.19 at 99 percent processor and no blocked processes, which is the left box in that figure. I had written a memory copy benchmark and called it disk pressure.
The fix was to bypass the cache so the writes reach the device. That is the honest version of the number on the left: it is real, it is 8.19, it says nothing at all about a disk.
What I did not check
Whether the kernel samples this often enough to be trusted at 5 second resolution, since it computes the average on a timer rather than on every change, so my sampling could easily be reading its rounding. Whether uninterruptible sleep on network storage counts the same way, which is the case that matters in a datacenter. And what any of this looks like inside a container, where the load average belongs to the host while the limits belong to the cgroup. I expect that to be its own kind of misleading.
The narrow claim is about the sentence I used to say. A load of 8 on 4 cores does not mean the processor is the bottleneck. Before saying anything about cores, look at what the processes are doing: the same 8 can be 4 cores of arithmetic or 7 processes waiting on a device that will not go faster because you bought a bigger machine.