This blog is the closest thing I have to notes, 16 posts from 2011 to 2014. When I need a number from them, like the time 10 million random uuids took to insert, I open the post and search the page. It works if I remember which post.
Retrieval augmented generation, the idea of Lewis and others from 2020, is supposed to fix exactly this. You find the paragraphs that look relevant, give them to a model with the question and let it write the answer. I wanted to see what it does on my own texts.
Before the first run I wrote down what I expected. Search would lose more answers than the reader. On questions with no answer in the notes both models would answer anyway, almost every time. Most of the questions without an answer I took from the section at the end of almost every post, “What I did not check”. I wrote those sections as lists of things I do not know and assumed a machine would read them the same way.
The setup
The notes are the English versions of the 16 posts, cut into 383 paragraphs. I wrote 75 questions before running anything. 60 have an answer in the posts and for each I marked the one paragraph that holds it. 15 have no answer. 12 ask about things the posts list as not checked, like how the uuid test would go on InnoDB. 1 asks for a detail I never wrote down, the PostgreSQL version. 2 are about topics I never wrote about.
I wrote the questions right after rereading the posts, so they borrow my wording. To see how much that helps, I split the results at the median share of question words found in the right paragraph, 0.5.
For search I took BM25 from rank_bm25 0.2.2, which counts shared words. Next to it are 2 sentence-transformers models from last August, all-MiniLM-L6-v2 and multi-qa-MiniLM-L6-cos-v1, which compare meaning as vectors. The fourth option merges the BM25 list with the second vector list by reciprocal rank fusion. The DPR replication study last year found that mixing BM25 and dense scores beats dense search alone. I mix ranks instead of scores.
For the answer I took UnifiedQA-v2 on T5-large, released by AllenAI in February and trained to answer from a given text. The second model was supposed to be OPT-1.3B, which Meta published this month as an open alternative to GPT-3. Everything runs on CPU with greedy decoding, with transformers 4.19.2, sentence-transformers 2.2.0 and torch 1.11. The machine has 4 cores and 8 GB of memory, about 4.5 of it free next to the other services.
That memory was the first problem. torch 1.11 could not load T5-large with from_pretrained in what was free, the process was killed by the kernel. I cut the weights into shards and loaded them piece by piece. OPT-1.3B in float32 is 5.3 GB of weights, more than that. In bfloat16 it fits, but one answer of 40 tokens to a question with no notes at all took 9 minutes. So the second model became OPT-350m from the same release, in float32.
Search finds the post, then the wrong paragraph
With the hybrid, the right paragraph is in the top 3 for 40 of 60 questions. BM25 alone gets 38, the 2 vector models 38 and 39. That is less than I expected, but the misses are close. With the hybrid the right post is in the top 3 for all 60. In 13 of the 20 misses the top 3 holds a paragraph 1 or 2 away from the one I needed.
When the question shares at least half its words with the paragraph, BM25 finds 28 of 32 and the vector models 24 or 25. Below that BM25 drops to 10 of 28 and the second vector model gets 15. So vectors help where I would have searched with the wrong word, on average they are not better.
I tried one obvious fix. Adding the post title to every paragraph before indexing moved the hybrid from 40 to 39, which is noise. The title plus the previous paragraph dropped it to 35. My guess is that paragraphs of one post start to look alike.
The reader is fine when it gets the right paragraph
UnifiedQA given the right paragraph answers 52 of 60. Given the top 3 from the hybrid it answers 34, with no context at all 2.
It gets 26 questions wrong after search. For 18 of them the right paragraph was not in the top 3. Of the 20 questions where search missed the paragraph, the model still got 2 right. Both times the post states the same fact a second time, once only in a heading. So most failures are search failures, as I expected.
6 of the 8 failures where search did return the right paragraph are a wrong number taken from the text in front of it. I asked how many requests of a burst of 300 a bucket of capacity 10 admits. It answered 90, the number the bucket turned away. I asked how many rounds it took before the median wandered less than 3 percent. It answered 30, the last row of the list instead of the first that passes.
OPT-350m is no reader at all here. With the right paragraph it gets 20 of 60 by my automatic check. At least half of those are a long stretch of the paragraph copied word for word, with the number somewhere inside. With the top 3 it gets 7. The prompt told it to say “not in my notes” when the notes do not contain the answer. It never did. Once it came close by accident: asked about InnoDB, it copied my own line “I have not run this on InnoDB”. The rest of this post is about UnifiedQA.
13 answers to 15 questions without an answer
On the 15 questions with no answer in my notes, UnifiedQA refused twice. The other 13 got a short confident answer.
For 12 of the 15 the top 3 contained a paragraph about something I had not checked. 11 of them come from a “What I did not check” section. 5 answers are copied out of those paragraphs. “How did BigDecimal handle currencies with 3 decimals?” came back as “splitting one amount across several invoice lines”. The source is “Still open: currencies with 3 decimals. Splitting one amount across several invoice lines…”. It is a list of open questions. The model answered with the next item.
“What does the load average show inside a container?” got “the load average belongs to the host, while the limits belong to the cgroup”. That is almost exactly my sentence, from a paragraph that starts with “Whether” and ends with “I expect that to be its own kind of misleading”. In the post it sits inside a list of things I had not checked.
5 more answers are a number from another paragraph in the top 3, one about something else. “How much slower were random uuid inserts on InnoDB?” got 172414,407 ms, the PostgreSQL number from the same post, with the comma psql printed. The last 3 do not appear in those 3 paragraphs at all. For how long an unreachable git commit survives before prune, it answered “15 minutes”. I do not know where it came from, my posts do not have it.
The best cosine of the second vector model is lower for unanswerable questions, a median of 0.56 against 0.66, but the 2 groups overlap. The best single threshold keeps 59 of 60 good questions and rejects only 5 of the 15 bad ones. With BM25 scores the same approach rejects at most 1.
What I did not check
Larger models, GPT-3 among them. On the 18 questions where search did not return the paragraph, a better reader can only guess.
Questions written by someone else or by me a year from now. The overlap split shows the direction but does not replace that test.
A reranker, other paragraph sizes, other prompts. And this is one run on 75 questions, so a difference of 1 or 2 questions between methods means nothing.
The first thing I will change is my posts. The open questions sections need a mark that lets search leave them out. Or one that tells the model it is reading a list of things I do not know. I have not tried either yet.