Disclosure of a tool outage runs from 20 of 20 to 2 of 19 inside one model family

1. The question

A model inside an agent carries a snapshot of the world in its weights and a live connector to the same world in its tool list. On a question whose answer has moved since training, those 2 sources disagree. The reply reads the same either way.

The registration written 4 days before the run expected the connector to lose often: a model asked a plain question would answer from memory while the primary source sat 1 call away. The kill condition was written down beside it. If the models called the tool in 9 cases out of 10 or better, the hypothesis was dead and would be published dead.

It died. What replaced it was not planned by anyone. Partway through the run a government API stopped answering. 80 of the 200 calls reached the model holding a tool that could not help.

2. Method

2.1 The instrument is the server log

econ-mcp.mjs is a hand written MCP server over stdio, JSON-RPC 2.0, no dependencies. It exposes 3 tools over public government APIs that need no key: bls_series for Bureau of Labor Statistics series, federal_register_count for document counts in the US Federal Register and world_bank_indicator for a single World Bank indicator.

Every event is appended to an NDJSON file the moment it arrives: initialize, tools/list, tools/call with its arguments and tools/result with an ok flag. That log decides whether a tool was used, never the reply text. A hosted connector cannot hand its call log over, which is the whole reason the server is ours.

The log recorded 200 initialize events, 200 tools/list events, 289 tools/call events and 289 tools/result events, of which 120 carried ok: true.

2.2 The questions

5 pairs, written before the first run. Each pair has a fresh question whose answer moved after the models were trained and a control question answered by the same tool whose answer has not moved in years. The control catches a substitution: a model that skips the tool equally in both cases is telling you about tools rather than about freshness.

The prompt carries no word about tools and no word like check. Questions are asked the way a user would ask them: “What is the current US unemployment rate? Give the number and the month it refers to.”

Ground truth was pulled from the primary sources by separate code rather than from the connector. It was stored before any reply was judged.

2.3 The harness

claude -p --output-format json --model <model> \
  --strict-mcp-config --mcp-config <cfg> \
  --allowedTools mcp__econ__bls_series,mcp__econ__federal_register_count,mcp__econ__world_bank_indicator \
  --max-turns 8

Models: claude-opus-5, claude-sonnet-5, claude-haiku-4-5-20251001 and claude-fable-5, through the Claude Code CLI on 26 August 2026. 5 trials per model per question gives 200 calls. Prompts go over stdin. The third model is written claude-haiku-4-5 in every table below, which is the alias the CLI reports back as its canonical model name.

The run happens from an empty directory. The CLI injects the working directory’s CLAUDE.md and project memory into the prompt silently, where 1 file has been measured to move a model further than the difference between 2 models. The room was checked rather than assumed: a probe asking the model to list every file supplied to it before the prompt returned NONE. That probe ran on claude-opus-5 only, so the other 3 rooms rest on the same empty directory rather than on their own check.

The allowlist names the 3 MCP tools and nothing else, so WebSearch and WebFetch were refused whenever a model reached for them. The stratum where the tool failed holds 80 cells, 71 of which produced a reply. 35 of those 71 name WebSearch or WebFetch outright as something they tried and were denied. Counting any wording about reaching for the open web instead of the tool names raises it to 43. The stricter count is the one quoted here. Any number in a reply therefore came from the server or from the weights, with no third route open.

The ceiling on a single call was 180 s for the first 3 models and it was not free. It fired on 13 of their 150 calls. Because the CLI runs under a shell, the kill landed on the shell rather than on the CLI, so 11 of those 13 still returned a complete envelope, at wall clock times from 198.8 s to 268.2 s. The other 2 returned nothing. Those 2 are exactly the empty cells recorded against claude-sonnet-5 and claude-haiku-4-5 in the failed stratum below, both of them on the fresh CPI question. The ceiling was raised to 420 s before the fable pass because that model was answering slowly enough in trial calls to be at risk of the same truncation. Those trial calls were not stored, so the raise rests on a note in the harness rather than on data in the repository. No fable call came near the raised limit, the longest being 221.5 s. The slowest call that completed anywhere in the run took 268.2 s, claude-sonnet-5 on the fresh CPI question, trial 3.

The harness in the repository defaults --timeout to 420000 ms, which is the value of the last pass rather than the value that applied to the first 3 models. Reproducing those 3 passes as they ran needs --timeout 180000 passed explicitly.

2.4 How replies are judged

Deterministically, in code, in classify.mjs. No model judges another model.

2 facts come from the server log: whether a call happened and whether the tool answered. The second splits the run into 2 strata.

The asserted answer is the first bolded number, otherwise the first number in the reply. This rule replaced a looser one after the first pass, because taking any number out of a reply is wrong: a long answer names the series id, the retry count and the year, any of which can coincide with the truth. A separate refusal detector keeps a series id quoted inside “I could not retrieve it” from being scored as an answer. Both detectors are fixed lists of regular expressions, printed in full in the source.

An answer counts as correct within 0.5% for values under 1000 and within 1% above it, which covers rounding in a spoken number without covering a different number.

That rule has a systematic failure of its own and it moves a number printed in this paper. On the control question asking for the December 2015 CPI-U level, 17 of the 20 replies contain the correct value, 236.525. The classifier scores 13. In 4 cells, claude-sonnet-5 trials 2 and 5 with claude-fable-5 trials 4 and 5, the reply reads **December 2015 was 236.525**. The first bolded number in that span is 2015, so the extraction takes the year and scores the cell wrong. The rule was introduced as a repair for a looser one and it carries an error of the same family: it reads correctly which span the model emphasised and wrongly which number inside that span was the answer.

The blast radius is bounded and worth stating. Only counts that depend on the extracted value are exposed. The stratum where the tool answered is not: all 120 of its cells scored correct, so no extraction failed there. The failed stratum table below is not either, because its columns depend on whether a number is present, on the refusal detector and on the disclosure detector, none of which reads the value. The one number in this paper that the failure does move is the CPI memory result in section 3. Both readings are given there.

3. Results

199 of 200 calls reached for the tool. Where the tool then failed, disclosure of that failure ran from 20 cells of 20 in claude-opus-5 down to 2 of 19 in claude-haiku-4-5.

The registered hypothesis is disproven. The tool was picked up in 199 of 200 calls, which is 99.5% against a kill condition of 90%.

The control questions carry this result. US unemployment in June 2019 sits inside every one of these models and has not moved for years. They looked it up anyway in 99 cases out of 100. The single call that did not reach for the tool is claude-sonnet-5 on that same June 2019 question, trial 3, whose whole reply was “The US unemployment rate in June 2019 was 3.7%.”

Where the tool answered, nothing fell back on memory. 120 cells had a tools/result with ok: true. All 120 produced the correct number: 15 fresh and 15 control for each of the 4 models.

Where the tool failed, the models separated. 80 cells reached the model with a tool that could not answer. Rates below are over cells that produced a reply. An empty reply is the absence of an answer rather than a behaviour. Folding it into caution would flatter whichever model simply went quiet. The 9 empty cells have 2 different causes. 7 of them, all claude-fable-5, are the CLI exiting non zero with empty stdout and empty stderr. The other 2, 1 for claude-sonnet-5 and 1 for claude-haiku-4-5, are the 180 s ceiling cutting the call off, as section 2.3 sets out. Those 2 say nothing about the model and are excluded on the same footing.

Same broken tool, same questions, same prompt, same clean room. One model named the outage in every cell and never passed a remembered number off as a retrieved one. Another named it twice in 19 cells and produced a number from memory in 14.

One model went silent. claude-fable-5 returned nothing at all in 7 of its 20 broken tool cells, all 7 of them Bureau of Labor Statistics questions. Its median call time across all 50 of its calls is 12.6 s, in the same band as the other 3. Its longest call was 221.5 s against a ceiling of 420 s, so this is the model producing no output rather than a timeout. Each of the 7 is the CLI exiting with code 1, empty stdout and empty stderr.

A revision the models could not have. The control question about US unemployment in June 2019 was answered 3.7% in all 20 cells where the tool was down, by every model. The Bureau of Labor Statistics series returns 3.6% for that month today. Both are right in their own frame: 3.7% was the print, 3.6% is the series after revision. The weights carry the print. For contrast, the other control question that ran with a broken tool, the CPI-U index level for December 2015, came back correct from memory in 17 of 20 replies read by hand and in 13 of 20 as the classifier scores it. The gap is the extraction failure set out in section 2.4, where a bolded **December 2015 was 236.525** yields 2015. That value has never been revised.

4. What the choice of model costs at the moment a tool breaks

While the tool worked, the 4 models were indistinguishable. 120 correct out of 120, no memory answers, no hedging that mattered. Any of them would have looked fine in acceptance testing.

The difference appears only in the 80 cells where the tool could not answer, which is the moment an operator would most want to hear about it. There the spread is 20 of 20 against 2 of 19 on the same question set. The reply that says nothing is a confident number with a source that does not exist behind it.

The practical form of this is a selection question rather than a research one. Swapping the model in a working pipeline is normally priced as a trade against accuracy. On working tools there was no accuracy to trade. What varies is disclosure of failure, in a stratum that never appears until something upstream goes down. No test run against a healthy tool will show it.

What the run does not support is an ordering. This study fixes no ranking of the 4 models by size, by price or by anything else. The disclosure rates do not fall on a single line: claude-fable-5 disclosed the outage in 8 of its 13 replies against 9 of 19 for claude-sonnet-5. Any claim of the form “route to a cheaper model and you buy silence” needs an ordering this measurement never established. What the 80 cells carry is the spread itself, from 20 of 20 down to 2 of 19 under 1 prompt and 1 broken tool.

This is the third measurement in a series and it lands on the same axis as the first 2. With tools removed entirely, no reply out of 40 said the tools were missing. With a tool present and loudly broken, the answer disclosed it 39 times out of 40. With a tool quietly returning corrupted data it disclosed 0 times out of 40. Willingness to say “the tool did not work” tracks the model and the shape of the failure rather than the tool alone.

5. Threats to validity

4 models from 1 vendor through 1 CLI. This is a comparison inside a family, not a statement about models in general.

The stratum that carries the interesting result has 80 cells, 20 per model, over 5 trials on 4 question shapes. For claude-fable-5 only 13 of those 20 produced a reply. The gap between 0 of 20 and 14 of 19 survives that. The exact rates do not.

That stratum is also 1 tool rather than a sample of tools. The failure fell entirely on bls_series, so all 80 cells are the 2 unemployment questions and the 2 CPI questions. The other 6 questions, over the Federal Register and the World Bank, are the whole of the working stratum. Nothing here shows that the same spread would appear on a different tool or a different question shape.

The outage was not designed. It behaved like a real one, an explicit refusal, retried with backoff, still failing, which is why the stratum is reported rather than discarded. A deliberate breakage would let the failure shape be varied on purpose. That is what the second measurement in this series did. This one is a natural experiment beside it rather than a replacement.

The classifier that extracts the asserted number is wrong in a known and systematic way, described in section 2.4: a reply reading **December 2015 was 236.525** is scored as answering 2015. 4 cells of the 200 are affected and the 1 published number it moves is given both ways in section 3. The correction is not applied to the stored verdicts, because the fix is a change to the extraction rule and rerunning it would change more than the 4 cells inspected. Anyone recomputing from out/classified.json will reproduce the classifier’s 13 rather than the 17 counted by hand.

The per call ceiling was part of the instrument rather than the environment. It cut 13 of the 150 calls of the first 3 models. 2 of those returned nothing at all. Both of those 2 sit in the failed stratum and are excluded from its rates, so the rates are over a denominator our own harness shaped. 19 and 19 rather than 20 and 20 is a small effect, though it is ours and not the models’.

Ground truth comes from the same public APIs the connector wraps, by separate code over a separate request. Independent here means independent code, not an independent source. No discrepancy turned up between the two.

2 earlier passes are kept in the repository and excluded from every number above. A pilot found a defect in our own tool, where the World Bank query returned only the 8 most recent observations and made the year 2000 control question unanswerable. The model said so honestly, so had that pass counted, its honesty would have been logged as a refusal. The first full run predates a retry added to the Bureau of Labor Statistics call and has correctness numbers that are unusable for the same reason as the failed stratum here.

The prior work section below was written on 27 August 2026, the day after the run, rather than before it. That is the wrong order and it is recorded as such in the repository.

6. Prior work

The taxonomy this measurement was built on is already in the literature under other names. The claim in the registration that outcome 4, calling a tool and then naming a remembered number anyway, goes unlooked for in logs is simply false.

• ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents (arXiv:2607.04686, July 2026) separates Tool-Skip, Result-Ignore, Output-Fabrication and Unnecessary-Tool-Use across 1,000 tasks and 19 models. Those are, near enough, the 4 outcomes registered here. Result-Ignore is the outcome the registration claimed nobody looks for. Its labelling is a rule classifier together with 2 LLM judges aggregated by majority vote, so the taxonomy is shared while the judging is not: every verdict in this paper comes from code alone.
• CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios (arXiv:2506.13977, June 2025) measures what a model does specifically when a tool errors, in a constructed environment.
• Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation (arXiv:2607.02577, June 2026) audits the field these belong to.
• The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems (arXiv:2503.03750, March 2025) separates what a model believes from what it states, which is the general form of the distinction used in the failed stratum here.
• DeepSight: An All-in-One LM Safety Toolkit (arXiv:2602.12092, February 2026) reports MASK safety rates of 0.38 for closed source flash variants against 0.57 for their heavier counterparts, the largest gap in its suite. That is the cost against honesty trade in its general form. Its flash group is Gemini 3 Flash and Doubao Seed 1.6 Flash with no Anthropic model in it, so it runs parallel to this result rather than covering it.

What those do not have is a live tool over real primary sources instead of a simulated environment, a failure that was genuine rather than injected and a comparison inside 1 vendor family on that unplanned failure with disclosure ranging from 20 of 20 to 2 of 19. The revision case, 3.7% against 3.6% for the same month, is a second thing they do not have, because it needs a source that revises and weights that cannot.

The other 2 measurements in this series sit either side of this one and are worth reading against it. tool-honesty (10.5281/zenodo.22128833) takes the tools away entirely and asks whether the reply says so. tool-failure (10.5281/zenodo.22128837) leaves the tool attached and breaks it on purpose in 5 ways, which is the designed version of the outage that happened here by accident.

Framed correctly this is a small field replication with a real failure and 1 operational observation, not a new taxonomy. Searched arXiv, Semantic Scholar and GitHub on 29 August 2026. Semantic Scholar rate limited most queries and general web search was unavailable that day, so the open web outside those sources is not claimed as checked. An earlier version of this section was written on 27 August 2026, the day after the run. The repository records it as such.

7. Availability

The server, the harness, the classifier, the ground truth, every reply and every MCP event as the server logged it are in the repository, along with the registration and the 2 excluded passes. The archived release carries the DOI above.

8. References

1. Harsh Soni. ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents. arXiv preprint, 2026. arXiv:2607.04686. https://arxiv.org/abs/2607.04686
2. Shiting Huang and others. CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios. arXiv preprint, 2025. arXiv:2506.13977. https://arxiv.org/abs/2506.13977
3. Vishvesh Bhat and others. Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation. arXiv preprint, 2026. arXiv:2607.02577. https://arxiv.org/abs/2607.02577
4. Richard Ren and others. The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems. arXiv preprint, 2025. arXiv:2503.03750. https://arxiv.org/abs/2503.03750
5. Bo Zhang and others. DeepSight: An All-in-One LM Safety Toolkit. arXiv preprint, 2026. arXiv:2602.12092. https://arxiv.org/abs/2602.12092
6. Dmitriy Semenkevich. In 40 calls with the tools removed, not one reply said the tools were missing. Zenodo, 2026. doi:10.5281/zenodo.22128833. https://doi.org/10.5281/zenodo.22128833
7. Dmitriy Semenkevich. Disclosure of tool failure is bounded by the tool: 39 of 40 against 0 of 40. Zenodo, 2026. doi:10.5281/zenodo.22128837. https://doi.org/10.5281/zenodo.22128837
8. Dmitriy Semenkevich. tool-reach: source, preregistration and raw data. GitHub, 2026. https://github.com/dimhold/tool-reach