I want search over my own notes, which are the posts of this blog. First I have to choose the vectors. In January OpenAI opened an embeddings endpoint and published the paper behind it. Table 6 there compares their search models with 8 other methods on 11 datasets from BEIR, a benchmark for search on unfamiliar data. In the average column the biggest model, cpt-text XL, is the best in the table with 52.8.
A few days later Nils Reimers, who wrote sentence-transformers, posted a long answer. He had called the API himself on 6 search datasets, with Ada and Babbage during the beta in December. His conclusion was that the search models are on par with free ones, while costing 8000 to 600000 times more to run. So I had 2 texts with numbers that said different things. I had no numbers of my own.
Before running anything I wrote my guess down. The best free model would lose to XL by 3 to 8 points on average. It would also beat the smallest OpenAI model on at least 2 of 4 datasets. The 3 to 8 points came only from the average column, I did not look at the rows.
The setup
I did not call the API. On the OpenAI side I take the numbers from their paper. By vector length the smallest size S is Ada in the API and the biggest XL is Davinci. They give 1024 and 12288 numbers per vector. On my side I run free models on the same datasets with the same metric. It is nDCG@10, which looks at the first 10 results and gives more weight to the higher ones.
The datasets are the 4 smallest of the 11, because everything runs on a CPU with 4 cores. SciFact has 5183 abstracts and 300 scientific claims to check. NFCorpus is medical articles with 323 queries written for the general public. FiQA has 57638 answers from the investment section of Stack Exchange with 648 questions. ArguAna has 8674 arguments. For each of its 1406 queries the right result is the counterargument.
The free models are 5 sentence-transformers models from June to August last year. all-MiniLM-L6-v2 has 22 million parameters and vectors of 384 numbers. all-mpnet-base-v2 and multi-qa-mpnet-base-dot-v1 have about 110 million and 768 numbers. The small multi-qa-MiniLM-L6-cos-v1 is made for questions. msmarco-distilbert-base-tas-b is TAS-B, which is in the OpenAI table too. Next to them is BM25 from rank_bm25 0.2.2, plain word matching. Search is exact, every query against every document.
TAS-B is how I checked that my numbers mean the same as theirs. On the 4 datasets I got 64.28, 31.89, 42.70 and 30.02, where the table has 64.3, 31.9, 42.9 and 30.0. My BM25 is not theirs: on SciFact it is 1.3 points lower, on ArguAna 7 points higher. BM25 in BEIR is Elasticsearch with its own settings, so I keep mine only for orientation.
The first run showed me how slow this is. TAS-B on SciFact took 55 minutes for 5183 abstracts, on sentence-transformers 2.2.0 with torch 1.11. On the way I lost track of it, started a second copy and killed it only when I saw both in the process list. Most of the grid then went to a second install that is about 3 times faster on this CPU. I ran 4 cells on both installs and nDCG@10 came out exactly the same.
Per dataset
On average my guess held, close to its lower edge. XL is at 52.7 on these 4 datasets. The best single free model, all-mpnet-base-v2, is at 48.8. That is 3.9 points lower and 2.4 above S. Against S, taking the best free model on each dataset, the free side wins on ArguAna and FiQA. On NFCorpus all-mpnet-base-v2 has 33.29 against 33.2, which I count as a draw. On SciFact the free side loses.
But the average is made of very different rows. On SciFact XL has 75.4 and all-mpnet-base-v2 has 65.6, almost 10 points less. On NFCorpus it is 40.7 against 33.3.
ArguAna goes the other way. multi-qa-mpnet-base-dot-v1 gets 50.4 and all-MiniLM-L6-v2 gets 50.2. Of the 4 OpenAI sizes M is the best with 49.2 and XL the worst with 43.5, in OpenAI’s own table. Here the query is a whole argument and the answer is the argument against it. My guess is that this is just a different task from answering a question, but I did not test it. In the same table XL is also below S on Quora and TREC-COVID, which I did not run.
FiQA surprised me the most. multi-qa-mpnet-base-dot-v1 was the first to finish there, with 48.8 against 51.2 for XL, above L. That looked too good, so I opened the list of what the model was trained on. It includes title and answer pairs from money.stackexchange.com. FiQA is built from the same site. 4 of my 5 free models have it, all except TAS-B. Their authors published those pairs, 29404 for this site, so I compared. For 532 of the 648 test questions the question is a title there, paired with an answer that FiQA marks as relevant.
I nearly left all-mpnet-base-v2 out of FiQA, because it needs 3.8 hours there even on the faster install. It finished with 50.0, the best free result there and 1.2 below XL. Its training list has the same pairs.
I expected the scores to fall on the other 116 questions. They barely moved: 48.6 against 48.8 for multi-qa-mpnet-base-dot-v1, 48.3 against 50.3 for all-mpnet-base-v2. But the 116 are not clean either. 72 of them are titles there too with another answer. 5 more have a relevant answer among the pairs. Only 39 questions appear in neither way. On them the 2 models get 44.8 and 43.4. TAS-B, which never saw this site, gets 31.7 there against 29.9 on the other 609. So the lead of all-mpnet-base-v2 over TAS-B is 12 points on the 39 and 20 on the rest. On the 39 its standard error is about 6. With so few questions I cannot tell if the overlap helps. OpenAI says their models were trained on data from the internet, so the same pages could be in theirs too.
What it costs
I counted the tokens with the GPT-2 tokenizer and cut documents at 2040 tokens. Reimers mentions this cut in an update to his post, from a script the paper’s author later provided. The prices are from the OpenAI page at the end of April: $0.008 per 1000 tokens on Ada and $0.60 on Davinci, 75 times more. The corpora and test queries of all 4 datasets are 15.0 million tokens. That is $120.29 on Ada and $9021.57 on Davinci. FiQA alone is $5960 of it.
Davinci gives 12288 numbers per document. In float32 the vectors of FiQA take 2.8 GB, against 89 MB with the 384 numbers of MiniLM.
The free side costs CPU time. all-MiniLM-L6-v2 encoded SciFact in 11 minutes on the same 2.2.0 and 4 cores. On the faster install the 3 models with 768 numbers took 6 to 16 times longer than MiniLM on SciFact.
So which one I should use depends on what my notes look like. If they look like SciFact, XL in the paper is about 10 points of nDCG@10 above the best free model. SciFact alone costs $1015.80 to encode on Davinci, so it is about $1000 per 5000 abstracts. If they look like ArguAna, all-MiniLM-L6-v2 with 22M parameters is 0.2 behind the best free model and above every OpenAI size. I will only know after I write a small set of questions with marked answers for my own notes. I have never done that for anything I search.
What I did not check
- The API itself. Reimers got different results from the paper on FiQA with the API and later traced it to how long texts are cut. The models behind the API could also have changed since the paper.
- 7 of the 11 datasets from the table, including Quora and
TREC-COVIDwhere XL loses to S. On 4 cores they would take weeks for MiniLM and months for the bigger models. - Overlap with training data on the other 3 datasets. I only looked at FiQA because its number looked suspicious.
- Rerankers and SPLADE. In the paper SPLADE v2 has 52.7 on average over the 11, 0.1 below XL.