The story everybody tells about an outage is that the retries finished it off. A service gets slow, every client tries again, the extra load lands on a machine that is already behind. The thing never comes back until you turn the clients off. I believed it well enough to repeat it in a design review. I had never seen the numbers with the retries removed.
So I built the smallest rig where the effect could be seen. A service that handles 20 requests at once for 20 milliseconds each, which is a ceiling of 1000 per second. A queue of 100 in front of it. Anything arriving when the queue is full gets a 503 immediately. A client that opens requests on a schedule instead of waiting for the previous one to finish, with a timeout of 100 milliseconds, because that is what a crowd of users looks like.
Then I ran the load past the ceiling with the retries switched off, which is the arm I had never bothered to run.
At 800 per second everything succeeds. At 1000 per second everything still succeeds, at a median of 58 milliseconds. At 1200 per second, which is 20 percent more load, useful output falls from 909 per second to 65. That is a factor of 14 for a factor of 1.2. No client is retrying anything.
The service is not broken during this. It completes 6067 requests in the 6.6 seconds of the run, which is about 920 per second, close to the ceiling it always had. It is simply that 5627 of those 6067 completions, 93 percent, are written to clients that had already given up and gone. The queue holds 100 requests and the service drains 1000 per second, so a request landing at the back of the queue waits about 100 milliseconds, exactly the timeout the client set. Everything past that point is work for nobody.
Now switch the retries on
Same rig, same loads, each failure retried up to 2 times.
The traffic goes from 7200 attempts to 20701, which is 2.88 times. At 1500 offered it is 2.95 times. The 503s go from 1133 to 14521. And the useful output goes from 65 per second to 76.
Up. Not down. The retries in this rig bought 11 more successes per second while tripling the traffic, because the queue was already full. The extra attempts got refused at the door in a microsecond each, without touching the 20 worker slots. My story had the cause and the passenger swapped: the collapse was the timeout meeting the queue. The retry storm rode on top of it without making the number that matters any worse.
I want to be careful about how far that carries. The rejection here is cheap, a single syscall and a 503, with a bounded queue behind it. In a service where the overflow path costs real work, a database connection or a thread, the same 3 times traffic goes straight into the part that is already the bottleneck. I would expect the sign to flip there. That is the version of the experiment I have not run.
The number I nearly missed
The server looked healthy the whole time. Requests completed, the completion rate stayed near the ceiling, latency inside the service never moved, because each unit of work still took its 20 milliseconds. I only saw the problem after adding a counter for responses written to a client that had already disconnected. Before that counter existed, the graph on the service side said 920 per second and the graph on the client side said 65. Both were true.
What I did not check
Backoff, which is the obvious next arm: these retries fire immediately. Spacing them out is exactly the fix the folklore recommends. A retry budget, which caps attempts as a fraction of traffic rather than per request. A closed loop client, where users wait instead of piling on, since that alone removes most of the amplification. And a real network between the 2 sides, which adds its own queue that I did not model at all.
The narrow claim is about which knob to reach for. If the queue in front of a service can hold more work than the client timeout will wait for, the service will spend the outage finishing requests for people who left, whether or not anybody retries.