A request of ours hung and the thread that made it hung with it. I said what I had said before, that a client without a timeout waits forever, so set one. It is the advice I would still give. I had just never checked whether the first half of it is true. It is not.
The kernel has an opinion about how long to wait. You get that opinion for free.
The setup is 2 machines and 1 address that goes nowhere. 192.0.2.0/24 is the range reserved for documentation, so packets sent there are dropped somewhere upstream. No answer ever comes back. A connect to 192.0.2.1:80 therefore does the one thing I wanted to time: it waits for a reply that will not arrive.
On the server, with tcp_syn_retries at its default of 6, it waited 135.3 seconds and then failed with ETIMEDOUT. Nothing in my code chose that number.
The doubling is the whole mechanism. The first packet goes out. If nothing answers the next one goes after 1 second, then 2, then 4 and so on, so with 6 retransmissions the sum of the waits is 127 seconds. That model is worth exactly as much as its predictions, so I changed the setting and timed it again. At 2 retries it gave up after 7.2 seconds against a predicted 7. At 3 retries it took 19.4 seconds against a predicted 15.
So the model is right at the short end and runs over by 4.4 and then 8.3 seconds as the tail gets longer. My first guess was the kernel remembering the previous attempts: Linux caches round trip estimates per destination. A polluted estimate would stretch every later run. It is not that. There is no entry for the address at all. Flushing the cache gives 19.4 seconds again, the same number to the tenth. Where the extra seconds come from I do not know.
The half that really is forever
The connect timeout is the half you are given. The other half you are not.
I started a local server that accepts the connection, reads the request and never answers. A client with nothing set of its own was still waiting when I stopped it at 30 seconds. There is no packet to retransmit here and no failure for the kernel to detect, because the connection is healthy and the peer is simply quiet. Nothing below the library has any reason to act. The same client with 2000 milliseconds set died at 2.0 seconds.
That is the shape of the trap. The failure everybody thinks about, a machine that is down, is the one already handled, badly but bounded. The failure nobody sets a timer for is a peer that took your request, said nothing and now holds a thread of yours until the process dies.
Where I tripped
The first run of this took 0.0 seconds and reported success. I ran it at home. My home network answers 192.0.2.1 on port 80, an address that by standard belongs to nobody. It answers 10.255.255.1 the same way. There is no black hole to measure on a network where something always picks up, so the measurement moved to a machine with a clean route. The honest version of my first result is that I measured my router instead of the kernel.
What I did not check
Where the extra 8.3 seconds come from. Whether the numbers are the same on Windows on a clean route, which I could not test because I do not have one at home. Whether a library timeout that fires mid request leaves the socket in a state the pool will reuse, which is the failure I actually worry about in production and it needs a different setup than this one. And retries: every client I know layers a retry over the timeout, so the number a caller waits is the product of 2 settings, not either of them.
The narrow claim is this. Nobody is counting after the connection is up. Before it is up the kernel is counting to something around 2 minutes with a number nobody in the code has ever seen.