Tagged reliability
- My control read the file 4 of 4 in August. In October it said the file did not exist and it was right
- 2 lists of 30 test names from the same Claude model share a name in up to 98% of pairs. Faker does it in 1.29%
- The same question to opus 5 gave 21 different answers in 64 calls, so I went looking for where the difference starts
- Halfway through the upgrade, half of my corpus scored 0
- Haiku 4.5 found the key 243 times of 245 in prompts up to 96 thousand tokens. For a key that was not there, it returned a value 12 times of 20.
- colors and faker were broken by their author in January. In December both names still belong to the same account
- The storm that took 38 Starlinks was a weak one and the catalog saw only 6
- My January and December venvs gave GPT-2 the same 20,000 tokens. The logits changed in between
- Flan-T5 answered 2,000 calls with 200 OK and got 161 of 600 amounts wrong
- A model behind my bank statement regexes turned 300 loud failures into 249 silent ones
- Flan-T5 read 1,393 of 1,413 invoice lines right and added up 1 of 400 invoices
- The tenant came back as the lessor after a round trip through Russian
- I forced GPT-2 to follow my JSON schema and USD 11,113.79 came out as 11
- I added "none of the above" and the model refused as often as before, but a third of the questions switched
- A chatbot told me the capital of Ghana is Moscow and said yes when I asked if it was sure
- 93 percent of the work went to clients who had already left
- 199 requests through a limit of 100, with nobody attacking