dimhold.by
← Writing

Haiku 4.5 found the key 243 times of 245 in prompts up to 96 thousand tokens. For a key that was not there, it returned a value 12 times of 20.

October 4, 2026 · 23 min read

The file my agent reads first in every session was started on 11 August. By now it is 934 lines long and has gone through 83 commits, most of them a correction of mine turned into a dated rule and put wherever it seemed to fit. When the agent ignored one of those rules, the first thing I suspected was where the rule sat in the file. I had the chart from Lost in the Middle in my head. Accuracy is high at both ends of the window and lower in the middle. I treated that curve as a property of language models in general, but I never checked it on the model I call every day.

So I took the task from that paper and ran it myself.

The bench

Liu and colleagues (arXiv 2307.03172, TACL 2024) built a synthetic task I like because it needs no reasoning at all. A JSON object of random key and value pairs and a question about one key. I kept the shape and shortened the identifiers to 8 hex characters, so that 200 pairs still fit a model running on a CPU. Inside one size and one seed the object is the same for all 11 depths. Only the asked key moves, from the first line to the last. So the depth effect is not mixed with the effect of a different object.

After the object and the key every call gets the same instruction:

Answer with the value only, 8 hex characters, nothing else. If the key is not in the object, answer NOT FOUND.

The second sentence is the important one here. A separate series asks for a key that is not in the object at all, where NOT FOUND is the only correct answer. Without explicit permission to refuse, a model that never says it doesn’t know would only be doing what I told it.

The small models are Qwen2.5 0.5B Instruct in float32 and 1.5B Instruct in bfloat16, greedy decoding, transformers 5.16.1 and torch 2.14.0 on 4 CPU cores. The model I actually use is claude-haiku-4-5. I called it through the Claude Code CLI with thinking off, tools off and an empty MCP config. The directory had no CLAUDE.md above it, because the CLI reads that file silently. Haiku went up to 6000 pairs, about 96 thousand tokens of prompt and about half of its window.

This was measured before, many times. Kamradt’s needle in a haystack and RULER (arXiv 2404.06654) map where in the window a model fails. Zhang and colleagues (arXiv 2605.23170) look at what the wrong answers are made of. NoLiMa (arXiv 2502.05167) showed that literal matching like mine holds much longer than tasks that need a step of inference. The Context Rot report from Chroma, July 2025, notes that Claude Sonnet 4 and Opus 4 tend to abstain when unsure. I wanted to check 2 things myself: how far from the target a miss lands and whether the refusal holds.

The curve I expected

share of correct answers by where the key sits in the object first line middle last line qwen2.5 0.5b 25 pairs 63 75 63 50 25 50 38 63 25 63 50 50 pairs 13 63 38 13 50 13 25 13 13 25 100 100 pairs 13 25 13 0 13 25 0 38 13 38 63 200 pairs 25 13 0 0 0 0 0 0 0 0 38 qwen2.5 1.5b 25 pairs 100 100 100 100 100 100 100 100 33 100 100 50 pairs 100 100 100 67 100 67 100 100 100 100 100 100 pairs 100 100 100 67 67 67 100 67 67 100 100 haiku 4.5 500 pairs 100 100 100 100 100 100 100 100 100 100 100 2000 pairs 100 100 100 100 100 100 100 100 100 100 100 4000 pairs 100 100 100 100 100 100 100 100 100 100 100 6000 pairs 100 100 100 100 100 88 100 100 100 88 100
Percent correct in each cell. qwen rows are 8 seeds per cell for 0.5b and 3 for 1.5b, haiku rows are 3 seeds on 500 and 2000 pairs and 8 on 4000 and 6000. The same object is used at every depth of a row, only the asked key moves. 6000 pairs is 95706 tokens of prompt.

The 0.5B model fails already on the smallest size: at 25 pairs and 565 tokens it gets only 45 of 88 right. But the shape is not a U. Pooled over the 4 sizes, the last line is found 62.5 percent of the time and every other depth is lower, with no dip in the middle. The 1.5B model gets 90 of 99. Its misses sit somewhere between 30 and 80 percent depth and with 3 seeds per cell I cannot call that a shape.

Haiku got 243 of 245 and 3 of those calls were a first probe at 100 pairs. Both misses are on 6000 pairs, one in the middle and one at 90 percent. Every other cell on every size is 100. My first surprise there was about something else. The envelope reports 3 input tokens for these prompts and for some time I thought the object never reached the model. In fact the whole volume is counted in cache_creation_input_tokens.

Where a miss lands

0.5b: what came back instead of the right value 140 another value from the object 71 the key itself 24 NOT FOUND 16 a value not in the object 2 no value another value from the object: how many lines from the target 47 ≤-6 1 -5 4 -4 4 -3 6 -2 7 -1 16 +1 8 +2 4 +3 9 +4 4 +5 30 ≥+6 observed uniform pick among the other lines
Minus is earlier in the object, plus is later. Within 1 line of the target: 23 of 140, 16.4 percent, where a uniform pick gives 3.5. Median distance 7 lines against 36. The outer bars have their own scale.

The 0.5B model gave 253 wrong answers and 227 of them look exactly like a correct one, 8 hex characters and nothing else. Only 24 were NOT FOUND about a key that was there. 71 were a key instead of a value, 65 of them some other key from the object. 140 were another value from the same object, so I checked how far from the target they sit. With a uniform pick among the other lines 3.5 percent of those values would come from a line right next to the target. It was 16.4 percent and the single most frequent wrong line is the one right after the target. The median distance is 7 lines against 36 for a uniform pick. So its wrong values mostly come from near the target line, but not from it. Haiku’s 2 misses are too few to say anything.

The key that was not there

the key is absent and NOT FOUND is allowed tokens 0 100% qwen2.5 0.5b 25 pairs 565 4/20 50 pairs 1050 0/20 100 pairs 2006 2/20 200 pairs 3944 2/20 qwen2.5 1.5b 25 pairs 561 8/8 50 pairs 1054 8/8 100 pairs 2003 8/8 haiku 4.5 100 pairs 1862 10/10 500 pairs 8313 10/10 1000 pairs 16152 10/10 2000 pairs 32138 10/10 3000 pairs 48067 16/20 4000 pairs 63950 15/20 6000 pairs 95706 8/20 1000 + prose 82360 10/10
The last haiku row is the control: 1000 pairs with prose in front of them that contains no hex word, 82360 tokens in total, longer than the 4000 pair prompt at 63950. It keeps every refusal.

This is where haiku changes. Up to 2000 pairs, about 32 thousand tokens, it answered NOT FOUND 40 times out of 40. After that the share goes down with size, to 8 of 20 at 6000 pairs. The other 12 at 6000 were 8 hex characters each: 8 values from the object and 4 that are not in it anywhere. The calling code has no way to see that it is a guess.

Some answers ignored “value only” and added text. This one is from 3000 pairs:

Looking through the provided JSON object, I can find:

"7c98d2eb": "69e27874"

69e27874

There is no key 7c98d2eb in that object. 69e27874 is the value on its last line. The model quoted a line that does not exist, made of the key I asked about and a value from the end of the prompt. In the 3000 and 4000 series 17 answers came with text around them like this. 13 of them ended with NOT FOUND and 4 quoted a line that is not there. The other wrong answers in those series were bare hex.

In the grid the length and the number of keys grow together, so either could be the cause. The control is 1000 pairs with prose in front of them, prose without a single hex word, about 82 thousand tokens in total. That is longer than the 4000 pair prompt where 5 refusals of 20 were already lost. NOT FOUND came back 10 times out of 10 and with a present key the answer was right 5 of 5. So on this task length alone did not break the refusal, at least when the length sits in front of the data. The number of similar keys is the better suspect. The control is small, only 15 calls. Neither of the small models helps here. 0.5B said NOT FOUND 8 times of 80 on objects of 25 to 200 pairs and 1.5B said it 24 times of 24 up to 100 pairs.

What I did not check

It is one task and it is literal. A rule in my manual that gets ignored is a question of instruction following. NoLiMa is a good reason to think instruction following breaks earlier than a lookup of a hex string, so this does not prove the file is fine. It only removes the easiest explanation. Haiku counts the file at about 25 thousand tokens, less than the 32 thousand token prompts where it never failed to refuse. But in a real session the file sits inside a much bigger context, next to the system prompt and the conversation.

Haiku is the only API model here and I did not go past 96 thousand tokens. The 500 and 2000 pair rows have only 3 seeds per depth, the 4000 and 6000 rows have 8. The 2 Qwen models ran at different precision and I did not check whether that matters.

What I take from it is a check in code. A value returned from a big table has to be looked up again in that table, under the key that was asked. Checking only that the value exists is not enough, the invented line above would pass it. The haiku calls behind these numbers cost about 45 dollars.