The mT5 paper lists 101 languages and Belarusian is one of them. In its appendix there is a table with the size of every language in mC4, the corpus mT5 was trained on. Belarusian has 2.0 billion tokens, Russian 713 billion. I read that table more than once and took from it only the ratio. It is small, but it is there. That the 2 billion tokens are Belarusian text I simply assumed, although the files are public and I never opened them.
So I opened them. mC4 is on Hugging Face in the AllenAI copy, the Belarusian part is 128 files and 3.4 GB compressed. Next to it I took CC-100, the corpus behind XLM-R, where Belarusian is 1 file of 0.72 GB. I did not save the corpora to disk. Every document went through 2 language detectors in Python and I wrote one line of features per document. The first is CLD3, the same detector mC4 used to pick the language of a page. The second is fastText 0.9.2 with its lid.176 model from 2017. mC4 did not use it. CC-100 did. I found that out only after counting and it matters below.
I am not the first to look inside. Kreutzer and others audited samples of many web corpora by hand and published it this year in TACL. For Belarusian in mC4 they rated about 100 sentences. 92% were correct, 4% another language and the rest not language at all. Before counting I wrote down what I expected: fastText says Belarusian on at least 90% of mC4 documents and at least 95% of CC-100. Below 80% I would call the label unreliable.
Size
Russian against Belarusian in mC4 is 408 to 1 by compressed bytes and 357 to 1 by the mT5 token counts. mC4 was built from the Common Crawl scrapes up to 2020. For comparison, the latest crawl I took, from January 2022, labels 0.0153% of pages as Belarusian and has 408 Russian pages for every Belarusian one. The same 408 is luck. These are pages against bytes and a crawl mC4 never saw. It only says the order is the same.
CC-100 is different. There Russian is only 68 times larger than Belarusian. English is 121 times larger, against 1174 in mC4. Both corpora start from Common Crawl, but CC-100 took 1 crawl for English and 12 for the other languages. That explains much of the English gap. On Russian the 2 corpora end 6 times apart and that I cannot explain. CC-100 also has its own filtering and I did not dig into which step makes the difference.
What the Belarusian folder contains
mC4 Belarusian has 1742030 documents. CLD3 calls all of them Belarusian, which is expected because that is how they got into the folder. fastText agrees on 81.6%. On 14.3% it says Russian, on 1.6% Ukrainian. So my 90% was wrong and the result landed just above the line I had drawn for an unreliable label.
Then I counted words instead of documents. By words fastText says Belarusian on 93.2% and Russian on only 2.7%. That is close to the 92% from the manual audit. I spent some time thinking my numbers disagreed with Kreutzer before I understood that they counted sentences and I counted documents. Words are not sentences either, so the match is partly luck. But the reason for the gap is clear: the Russian documents are tiny.
There are 249162 of them. The median length is 27 words, 3 of 4 are shorter than 50 words. 146160 of them sit on .ru domains. I read the 47 that fell into a random sample of 400 documents. 14.3% would give 57, but 400 is a small sample. 26 of them are clearly shop pages: dry pastel, mattresses 90x205 in Samara, shower cabins, a battery for a SonyEricsson K770. 5 more are lists of products on pages that are not quite shops. The rest is torrents, a forum thread, a list of doctors, a casino, a recipe and similar. 3 of the 47 are not purely Russian. I did not find one ordinary article among them. A page of product titles has almost no ordinary Russian words, so the letters that separate Russian from Belarusian barely show up.
I took one such page, a grill from a Russian online shop. I gave CLD3 its lines one by one. With only the first line it says Russian. From the second line on it says Belarusian until line 7, where the word “доставка” (delivery) brings Russian back. With all 8 lines it is Belarusian again, with 0.797 confidence. mC4 dropped pages below 0.7, so this one passed.
CC-100 looks cleaner. fastText says Belarusian on 98.8% of its 1450734 documents and on 99.96% of its words. My expectation of 95% held. But CC-100 was filtered with this same lid.176 model, so here fastText mostly confirms its own earlier decision.
What this means for mT5
mT5 trained for 1 million steps on batches of 1024 sequences of 1024 tokens, which is about 1.05 trillion tokens. Its sampling gave Belarusian 0.59% of them. That is 6.19 billion tokens from a pile of 2.0 billion, so the model went over the Belarusian folder about 3 times. These are input tokens after the span corruption, so the real number is maybe a bit higher. Russian got 3.71% of the training, but its pile is so big that the model saw only about 5.5% of it once.
So the model repeats the Belarusian folder 3 times, Russian shop pages included. Of the real Russian pile it reads about 5.5%. By words Russian is 2.7% of what gets repeated and in my sample shops are most of it. For a model that is probably not a problem. But if you sample documents from mC4 to count something, fastText calls every seventh “Belarusian” page Russian.
2 spellings in one folder
Belarusian has 2 spellings in use, the official one and the classical one called taraškievica. Taraškievica is mostly used by independent media and has its own Wikipedia. The corpora do not separate them. I looked for a count of taraškievica in mC4 or CC-100 and did not find one.
The classical spelling writes a soft sign between consonants where the official one does not: сьнег against снег, жыцьцё against жыццё. I made a list of such clusters plus the letter ґ and checked it on about 200 random articles from each Belarusian Wikipedia. On the official one it found nothing at all. On the classical one it caught 49 of 52 articles of 100 to 200 words and every longer one.
Among Belarusian documents of at least 100 words, 22.0% in mC4 and 20.9% in CC-100 have the classical spelling. I expected between 15% and 40%. So roughly 1 in 5 long Belarusian texts in both corpora is in the classical spelling, which the official rules do not accept.
What I did not check
fastText is not ground truth. Short Belarusian pages with few letters are hard for every detector. The mT5 paper says mC4 kept only pages with at least 3 lines of 200 characters. The Russian pages in the folder are much shorter than that. Maybe the long lines were removed later as duplicates, but I did not check it. The taraškievica count misses a few texts of 100 to 200 words, 3 of 52 in the Wikipedia check, so 22.0% is a bit low. Short texts I did not count at all. The count also takes a document as classical even if it only quotes the spelling once. I did not train anything, so what the Russian shops do to mT5 Belarusian is a guess.