Tagged language models
- My control read the file 4 of 4 in August. In October it said the file did not exist and it was right
- Claude 5.5 says its knowledge ends in June 2026. It got no package release from March 2026 right
- 2 lists of 30 test names from the same Claude model share a name in up to 98% of pairs. Faker does it in 1.29%
- The same question to opus 5 gave 21 different answers in 64 calls, so I went looking for where the difference starts
- Halfway through the upgrade, half of my corpus scored 0
- Haiku 4.5 found the key 243 times of 245 in prompts up to 96 thousand tokens. For a key that was not there, it returned a value 12 times of 20.
- Please costs 24 dollars per million calls, which is nothing. Just the answer cost me 32.5 points of accuracy.
- 2 comment lines turned 62 findings into 0
- The critic passes 13 of the 16 drafts my checks fail
- A model reading my essays gives back 2 of my 16 rules
- My January and December venvs gave GPT-2 the same 20,000 tokens. The logits changed in between
- Flan-T5 answered 2,000 calls with 200 OK and got 161 of 600 amounts wrong
- A model behind my bank statement regexes turned 300 loud failures into 249 silent ones
- Flan-T5 read 1,393 of 1,413 invoice lines right and added up 1 of 400 invoices
- GPT-3 reads Belarusian at 6.5 times the price of English
- I asked my notes 15 questions they cannot answer and got 13 answers
- OpenAI's biggest embedding model costs 75 times more than the smallest and loses ArguAna to a 22M one
- I forced GPT-2 to follow my JSON schema and USD 11,113.79 came out as 11
- I added "none of the above" and the model refused as often as before, but a third of the questions switched
- I asked GPT-2 to show its work and it trusted its own wrong arithmetic
- I tested prompt tips on GPT-2 and the order of the examples beat all of them
- A chatbot told me the capital of Ghana is Moscow and said yes when I asked if it was sure
- I trained a reward model on 16,000 Reddit votes and it liked length more than the voters did