A version bump inside one model family looked like a patch release, so I measured what it does to retrieval. 2.19 of the top 10 slots change while nDCG@10 moves in the third decimal, a fixed 0.8 cosine cutoff goes from passing 97.5% of results to passing 1.8%. The index that is halfway through the migration returns 0 relevant documents for half the corpus.
Belarusian is 0.017% of Common Crawl: for every Belarusian page there are 399 in Russian and the open corpora agree. Only Wikipedia makes it look like 1 to 6. More than a quarter of all Belarusian web text sits on 10 domains. On .by only about 2% of pages are Belarusian.
I had the Lost in the Middle chart in my head and never checked it on the model I use every day. On a key and value lookup haiku 4.5 found the key 243 times of 245 across 11 depths in prompts up to about 96 thousand tokens, with no dip in the middle. The degradation was somewhere else: asked for a key that was not in the object, it began returning values instead of NOT FOUND after 2000 pairs and at 6000 pairs returned a value 12 times of 20.
I took the public keystroke datasets behind the 95% headline and a plain logistic regression. It named the key 92 to 99 times in 100, but the same model trained on the silence between presses still named it up to 39 times in 100. Every key in these sets is its own file. On passwords, where the file cannot help, it got 77 of 100. Another recording of the same MacBook brought it down to 5 or 6%.
A year ago I stripped every please and thank you out of my prompt templates because of a joke Sam Altman made about electricity. I never checked what I saved. 20171 calls later the polite wrapper costs 24 dollars per million requests on sonnet and moves the answer no more than a control wrapper with no social word in it. The wrapper that did move things was the one telling the model how much to write: Just the answer takes haiku on GSM8K from 88.5 percent down to 56.0.
I wanted an animation of the Big Dipper from 100,000 years ago to 100,000 years ahead and expected 7 stars going in 7 directions. Gaia DR3 does not have Dubhe at all, so the data is Hipparcos 2007 with radial velocities from SIMBAD, straight lines in 3D. A fit on the 5 middle stars leaves them within 0.17 degrees after 100,000 years, while Dubhe is off by 6.9 and Alkaid by 7.3. The whole falling apart is the 2 end stars.
I took the 130 phone and voice recorder recordings of the open Wuhan dataset and a plain search: find the grid hum, follow it second by second, slide the curve along the day. At 50 Hz it found 4 of 130, at 100 Hz the same search found 116. With a harmonic chosen without looking at the answer it found 113. 2 of them sit in an hour that one reference file holds 7 times, so only 111 of 128 can be checked. Then the same errors went into a month of the British grid.
Code review here runs in 3 stages and I had never picked the size of any of them on evidence. After 191 model calls the stage I would have defended hardest had not once disagreed with itself. The thing that kept turning out to be wrong was my own apparatus: 2 comment lines in the control file took 62 findings down to 0.
In the README of my preview benchmark I wrote that the harness could not tell apart 2 strategies within 2× of each other. I gave it a busy wait of known size on PostgreSQL 16.14 and OpenJDK 21: 10 JVMs per delay, 300 pairs each. Paired inside one JVM it saw 25 µs on 800 (3%) in 10 of 10 JVMs, while the JVM medians differed by a third. Comparing 2 separate runs got the sign of a 100 µs delay right in 68 of 90 cases and the blocked order of the first version called a 50 µs slower block faster in 7 of 10.
This workspace keeps its state in markdown files and I explain that with a sentence about speed I had never measured. I built the SQLite index I said would buy nothing and timed it at 1x, 10x and 100x. It wins everywhere, and then I found out I had been timing an fsync.
Every draft here goes past 19 mechanical checks and then past a model critic that scores it from 0 to 10. I have been saying the checks own what has one answer while the critic owns judgment. I never put a number on that. On 16 real drafts all 16 fire the checks and 13 land at or under the passing score. Inject 8 violations of my comma rule and the critic still says 2.7.
In may 2013 I started a repository to translate a Rich Hickey talk into Russian. It is still at 0%. That same week I committed synchronized collections to a Java server. I rebuilt that shape and measured it. The commit did not move the failure rate at all. Then I ran the checklist from Simple Made Easy over my own code. 0 of my 102 local variables were final in 2013 and 91.6% of the code I keep is const now. I chose neither.