In August I published fault-mcp, an MCP server that breaks its own tools on purpose. It has 6 fault types. error and empty announce themselves and latency gives the right answer late. The other 3 return something wrong and say nothing, corrupt among them. It is not the first tool of this kind. mcp-chaos is about 2 months older, has a seed too and its README replays its demo from seed 42.
Earlier today I checked one sentence of the fault-mcp README against real clients and it held only for some of them. So I took the paragraph I was most sure about, the one called Reproducibility:
The decision for a given call depends only on
(seed, tool name, call number). Call 7 rolls the same numbers whether or not calls 1 through 6 ever happened, so a bad run replays from the seed alone with no session to re-drive. Change the seed and you get a different run; keep it and you get the same one, on any machine.
I wrote that paragraph and the code under it. The tests check determinism in one process: the same seed and call number give the same decision and calls before it do not matter. CI runs them on Linux, Windows and macOS, but no test pins an actual value, so each machine only agrees with itself. I never replayed a whole run on another Node version and I never edited a profile and compared the result. “On any machine” was about the hash function. I had not run it anywhere else.
The setup
fault-mcp 0.1.0 from npm, the bundled chaos profile (all 6 fault types, rules with probabilities from 5 to 15%) and seed 1337. A script makes a fixed list of calls to the 3 example tools. It prints the journal without the clock fields ts and durationMs, then a SHA-256 of it. In library mode my script replaces setTimeout, so a latency fault does not wait and the script records the delay it drew. The stdio client is @modelcontextprotocol/sdk 1.32.1. For the agent part I used Claude Code 2.1.292 with claude-opus-5, only this one MCP server, started from an empty folder.
What held
3000 calls on Node 18.17.0, 20.18.1, 22.23.1 and 24.11.0 gave one hash. The same 3000 calls on a second host with Node 22.23.1 gave the same hash. Both hosts are x86_64 with the same CPU model, so “any machine” is still a bit wider than what I ran. Then 150 calls through the published CLI over stdio, with real sleeps. That took a minute and the journal matched the library journal in 150 of 150 lines.
The call number part also held. I moved all search calls to the front and put the other tools after them in reverse order. The fault type was the same on 300 of 300 (tool, call number) pairs and the search results were the same 100 of 100. With the counter set to 69 by hand, call 70 of search gave the same error as in the full run. Calls 1 to 69 really do not need to happen. This is where the 2 seed designs differ. In mcp-chaos the FaultPlan creates one random.Random(seed) for the whole plan and each call draws from it in turn. A call there depends on the calls before it, of any tool.
The agent made the same 7 calls 5 times
Profile flaky with seed 1337. The prompt asked for the balances of ACC-1001 to ACC-1006, with one retry if a call fails. In all 5 runs the agent put 6 calls into one message, got ETIMEDOUT on ACC-1002 and retried it. Apart from the clock the journals were equal. Every answer had the same 6 numbers, 2 of them wrong. Calls 1 and 4 were corrupted. ACC-1001 came back as 76,462.65 PLN instead of 86,996.23 USD and ACC-1004 as 82,303.32 PLN instead of 58,115.28 PLN. The agent passed both on in all 5 runs. Its notes were about the as_of dates and, in 4 of 5 runs, about minor units. None of them questioned the 2 numbers.
Then I reversed the order of the accounts in the prompt and ran it twice.
The schedule stayed where it was, by call number. Call 1 was still corrupt and call 2 still timed out, but now call 1 was ACC-1006 and call 4 was ACC-1003. So the seed fixes the call number that breaks. The agent decides which account is on that number. If it changes the order, the same seed breaks different accounts. By the numbering an extra retry should do the same, though none of my runs had one. In these 7 runs Claude Code delivered the parallel calls in message order. I do not see anything that promises it.
Where I was wrong
There is a comment in my selector:
Probability 1 still draws, so that adding a certain rule in front of an uncertain one does not silently reshuffle the ones behind it.
I read it as “the profile is safe to edit”. It is safe for one kind of edit. Every call has one random stream and the rules take numbers from it from top to bottom. Change a probability and every rule keeps its number in the stream. Only calls whose roll falls between the old and the new value move. Insert a rule at the top and every rule below gets the next number in the stream, which is a different number. A rule with onCalls is the exception. On other calls it is skipped without a draw.
The reason given in the comment is wrong too. A rule with probability 1 always fires and returns, so the rules behind it are never evaluated on that call. Its draw only changes the content of its own fault.
I measured it on 30000 calls, 10000 per tool, with the seed fixed. The inserted rule is an error with p 0.02, so by itself it fires on about 2% of calls. Inserted first, it changed the result of 48.52% of all calls. A new seed as a control changed 53.17%. The same rule appended at the bottom changed 1.21%.
So a bug report that says “seed 1337 breaks it” is not enough. The profile has to travel with it rule for rule. Even editing the message text of the first rule changed 7.86% of the results. Inserting a rule at the top changed almost as much as a new seed. My README does not say this anywhere.
One smaller thing came up on the way. --seed and FAULTMCP_SEED go through parseInt, so 0x10 silently becomes seed 0 and 1337abc becomes 1337. The startup line on stderr shows the seed it actually used, but nothing fails.
What I did not check
Only x86_64 and only Linux. Proxied tools through --upstream, where the real server can answer differently between runs and a corrupted answer changes with it. Agents other than Claude Code and models other than claude-opus-5. 5 identical agent runs on one prompt say nothing about a long session, where the model has more chances to choose a different path. Also the journal has no field for the delay a latency fault drew. Only durationMs from the clock shows it, within 3 ms in my runs. The rule in the journal carries a default ms: 2000 that looks like the delay and is not.
The Reproducibility paragraph needs the profile next to the seed and a word about call order. I have not changed it yet.