I am from Belarus and every post on this blog goes out in Belarusian too. GPT-3 is the obvious helper for that work now and it is paid by the token. Before using it I wanted to know what Belarusian costs there.
The OpenAI help page on tokens gives a rule of thumb: 1 token is about 4 characters of English text. I took that rule and applied it to Russian in my head with a correction. A Cyrillic letter is 2 bytes in UTF-8, the tokenizer works on bytes, so Russian should cost about twice as much, maybe a bit more. The same page has an example in Spanish and a sentence saying that other languages can be more expensive. I read past it and never counted either language.
Before counting I wrote my guess down: Russian about 4 times English, Belarusian about 5. It was still too low.
The setup
The corpus is FLORES-101 devtest, published by Facebook last June. It has 1012 sentences translated by people from English into 100 other languages, so the meaning stays fixed and only the language changes. I took English, Russian, Belarusian, plus Ukrainian and Polish to compare. Every sentence is encoded alone, without special tokens. The tax is the token count of a language divided by the token count of English on the same tokenizer.
The GPT-3 paper says the model reuses the tokenization of GPT-2, a byte level BPE whose vocabulary is public since 2019. Next to it I took 4 more tokenizers. ruGPT-3 from Sber was trained on Russian. mBERT, XLM-R and mT5 are multilingual and had Belarusian in their training data. The counting is done with tokenizers 0.12.1 and sentencepiece 0.1.96.
People counted this before me for mBERT. Ács looked at its vocabulary in 2019 and measured how it splits words in 54 of its 104 languages. Rust and others compared it with monolingual tokenizers last year, Russian included. I did not find anyone who put Belarusian through the GPT-3 tokenizer or compared it in Cyrillic and in Latin letters on the same sentences.
6.47 times
On GPT-2 English is 27044 tokens. Russian is 5.68 times that and Belarusian 6.47. Ukrainian is 5.71, almost the same as Russian. Polish, which uses Latin letters with diacritics, is 2.68.
At the Davinci price, $0.06 per 1000 tokens, the English corpus costs $1.62 and the Belarusian one $10.50. The base Davinci model takes 2048 tokens for prompt and answer together. An average English sentence here is 27 tokens and a Belarusian one 173, so the window fits 76 English sentences and 11 Belarusian ones.
The multilingual tokenizers are much milder: on XLM-R Russian is 1.16 times English and Belarusian 1.45, on mBERT Belarusian is 1.99.
Then I wanted to see the vocabulary itself. My first count said that GPT-2 has no Cyrillic in it at all. That cannot be right, because Russian came out at 1.08 tokens per character and not about 1.8, its bytes per character. I had forgotten to turn the byte pieces back into text before searching. The second count found 18 entries with Cyrillic out of 50257. 17 of them are single lowercase letters: о, а, е, и, т and others that are frequent in Russian. Every other letter is 2 tokens, its 2 bytes. Every capital is 2 tokens too.
So GPT-2 reads Cyrillic letter by letter and Belarusian uses the expensive letters more. ў and і are not among the 17 and together they are 7.4% of all Belarusian letters. With з, ц, ч, ш, the capitals and the others, 29.1% of Belarusian letters cost 2 tokens, against 17.0% of Russian ones. The Belarusian translation also has about 4% more letters. A count of only the letters, 1 token or 2 each, explains about 90% of the 21598 tokens Belarusian takes over Russian.
Latin letters
Belarusian has its own Latin alphabet, łacinka, with č, š, ž, ŭ and ł. If the cost comes from the alphabet, the same words written in łacinka should be much cheaper on GPT-2. I wrote a rough converter with the classical rules for ł/l, ia/ja and the soft ś before a soft consonant. It has no exceptions for loanwords. A Belarusian reader would find mistakes in its output, but the words stay the same and only the letters change.
On GPT-2 łacinka costs 2.98 times English instead of 6.47, so the bill drops by 54%. “навукоўцы” (scientists) is 12 tokens in Cyrillic and 5 as “navukoŭcy”. This part matched my expectation, I had written down at least 30% cheaper.
Then I ran the same conversion on the other 4 tokenizers and it went the other way on every one of them. On mBERT łacinka is 6% more expensive than Cyrillic and on ruGPT-3 52%, with mT5 and XLM-R between. The 3 multilingual tokenizers saw Belarusian in Cyrillic during training. I think they learned its pieces and Latin spellings of the same words were rare in their data, ŭ especially. On XLM-R “навукоўцы” is 2 tokens, one less than Russian “учёные”. In łacinka it is 4. ruGPT-3 has only 2 vocabulary entries with ў, but probably the same works there: it learned Cyrillic pieces from Russian and łacinka throws them away.
The neighbour’s tokenizer
ruGPT-3 has the same structure as GPT-2, a byte level BPE with 50257 entries, but it learned its merges on Russian. For Russian, Sber removed most of the tax: on it Russian takes 29833 tokens, 1.10 times what English takes on GPT-2.
I expected Belarusian to profit from that, the languages are close and most of the letters are shared. On ruGPT-3 Belarusian is 2.12 times Russian. On GPT-2 the gap was only 1.14, because both languages were equally badly served there. 78% of Russian words are a single token on ruGPT-3 and 27% of Belarusian ones. “навукоўцы” is 5 tokens where “учёные” is 1.
The 2 letters again looked like the cause. I replaced ў with у and і with и in the whole Belarusian corpus. On GPT-2 that closes 42% of the gap to Russian, which fits the letter by letter reading. On ruGPT-3 only 16%. There the vocabulary has whole Russian words and Belarusian words are simply different words, so swapping letters does not make them Russian. My guess is that this is the bigger part of the 2.12, but I did not separate it.
What I did not check
- Only the cost in tokens. Whether GPT-3 answers worse in Belarusian, I did not measure.
- My łacinka converter is rough. My first version had no ji after vowels and no soft ś. Adding them moved GPT-2 from 2.91 to 2.98 and the other 4 stayed more expensive than Cyrillic. A careful human transliteration may move it a bit more.
- FLORES is text from Wikimedia projects like Wikinews, translated from English. Text written in Belarusian from the start may tokenize differently.
- OpenAI may change the tokenizer for newer models. I counted with the GPT-2 vocabulary, which the GPT-3 paper itself calls “developed for an almost entirely English training dataset”.