dimhold.by

Writing

59 posts Archive →

October 12, 2026

I wrote "rerun it yourself" under 4 gists. None of them ran as written

In August 4 posts on X linked a gist with the whole run and the promise that anyone can rerun it. I cloned each gist into an empty folder and ran its own command. 0 of 4 worked: the files carry number prefixes, a package.json is missing and the Java gist reported a green build with 0 tests. After 2 fixes each, the stored results reproduced and small live runs matched August. One gist was updated in place under a post that now quotes different numbers.

12 min read

reliability language models git

October 12, 2026

I quoted Haiku's $8k miss twice. The miss that matters is $684k and comes from dividing by 0.10

In August I quoted a Haiku 4.5 miss on a 36 month cash forecast twice, $5,775 short and then $8,451 off, from 1 answer per run. I asked the same question 150 times with no tools. 7 of 30 Haiku answers were $684k too high. With the working shown every such answer divides by 0.10 where the formula says 1.11 - 1. The smaller misses track the error in its own 1.11^36. Asked to flag doubt, it marked 7 close answers and none of the 4 big ones.

11 min read

language models numbers reliability

October 12, 2026

My fault injector hides outputSchema from strict clients. Claude Code 2.1.292 did not check it at all

The README of my MCP fault injector says it hides outputSchema because a strict client would reject a schema fault before the agent sees it. I ran 9 fault profiles through raw JSON-RPC, the TypeScript SDK 1.32.1, the Python SDK 2.3.0 and Claude Code 2.1.292. With the schema published, TypeScript after listTools and Python threw on 4 of 8 faults, including empty and truncate. Without listTools the TypeScript client checked nothing. Claude Code passed every schema fault to the model, which answered $40,718.77 each time. One fault never left my own server.

15 min read

agents reliability

October 12, 2026

My control read the file 4 of 4 in August. Today it said the file did not exist and it was right

In August models with the tools taken away never said the tools were missing. A control with tools read the file 4 of 4. I added the other side: the same questions about a path that does not exist. On Claude Code 2.1.292 the control failed first, because Read was reading the disk of another host. In the clean run both models were right 10 of 10 and haiku handed over the value of a neighbouring file 10 of 10. With the tools off the replies were the same whether the path existed or not.

16 min read

agents language models reliability

October 11, 2026

earshot promised no network calls and onnxruntime broke that the day it went public

The earshot README says network calls: none. With the network cut it really works, 10 of 10 transcripts identical. With the network on, every start asks Hugging Face for the model revision. onnxruntime, pulled in for voice activity filtering, sends telemetry to Microsoft since 1.29.0. Offline sessions queue the event until the machine is online again. 2 environment variables bring it to 0 connections.

9 min read

networking dependencies

October 11, 2026

On Linux recording the speakers is free until you touch the volume

The earshot README says system audio works out of the box on Linux and that macOS has no capture API at all. The macOS line is wrong since macOS 13; a virtual device on Windows and macOS is still needed because of ffmpeg. On Linux the monitor is bit for bit at 100%, but on PulseAudio the output slider and mute reach it. On PipeWire the app slider does. With the PulseAudio output slider at 60% earshot cuts no clip at all from 10 spoken sentences.

13 min read

linux debugging

October 11, 2026

My watch asks my laptop for $0 in API and every new question still costs 7 cents

The README of my watch app says one reply from the watch costs about $0.07 of subscription limit. I took it for the price of a cold start and measured it. 10 new questions cost $0.070 to $0.072 each and only a word for word repeat drops to about a cent, because every new question writes about 6,300 tokens to the one hour cache at double price. Without Claude Code's tools a new question costs $0.009.

10 min read

language models agents

October 10, 2026

Claude wrote a correct invoice program 19 times out of 19, then made up what it printed

I always say that money is counted by a deterministic engine and the model only talks, but I never measured the invoice half. On Polish VAT invoices Claude Opus 5 without tools was exact to the grosz in 18 of 20 at 5 lines and 19 of 20 at 40 lines when told to count by hand. Every wrong answer still added up. Without that instruction it wrote a program in 18 of 20 answers. All 19 programs it wrote were right and in 9 answers it invented what the program printed.

15 min read

language models numbers

October 9, 2026

On my Windows laptop whotop found who holds each port. 11 of its answers quoted a word that was not there

I took one snapshot from my Windows laptop at 01:59 and checked what whotop 0.4.1 said about each of the 33 processes holding a listening port. The port column held. 8 astro processes sat on a port their command line did not name and whotop showed the real one. But 11 rows quoted vite as evidence when the word is not in the command line. 8 inferred directories were the npx cache or a temp folder and the service names reached only the interactive screen.

14 min read

debugging networking