There is an old check for machine translation. You translate the text, then translate the result back. If the original comes back, you trust the middle. I believed in this check and I never counted what it misses. To count that, I need a pair where I can read the middle step myself and say if it is right. For me this pair is English to Russian and back to English.
The doubt is not new. Harold Somers wrote in 2005 that round trip translation is a weak way to judge a translation system. A bad middle step can come back looking fine. I wanted to see it on my own sentences, with several kinds of meaning at once and with the middle step checked separately.
The setup
The models are opus-mt-en-ru and opus-mt-ru-en from Helsinki-NLP, MarianMT, the revisions from September 2022 with their own config, which means beam search with 4 beams. Python 3.9, torch 1.13, transformers 4.25.1, CPU only, 4 cores of an AMD EPYC virtual machine. One round of 600 sentences there and 600 back takes about 4 minutes.
A script with seed 2022 wrote 600 short sentences, 100 of each of 6 kinds. Every sentence carries something a regex can check after the round trip:
- a number with a currency or a date, like
The invoice for 1,028,211 zlotys is due on 26 December.; - a negation word like
notorneverin 50 sentences and the same 50 without it; - a modal verb,
must,mayorshould; - a bound,
at least,at most,more thanorless thanwith a number; - a role and a pronoun,
The doctor called because she needs the invoice.; - names of people and companies, mostly Polish and Belarusian, some with diacritics.
Then I ran 5 rounds. Each round translates the English that came back from the previous one.
The first round
The numbers came back 100 of 100. The negation word came back 50 of 50 and no negation appeared in the 50 sentences without it. Pronouns came back 100 of 100. Bounds came back 92 times. 7 times at least or at most just disappeared. The office is at most 80 kilometers from the station returned as The office is 80 kilometres from the station. Once 3631 dollars became 631.
The kind of modality came back 67 times. should became должен in Russian and must or has to in English 32 of 33 times. Russian has a softer word for it, следует, but the model used it once.
Names were the worst kind. Of 300 names 113 came back exactly. Of the 65 with Polish letters only 7 did. Piotr became Peter and Paweł Wiśniewski became Pavel Vignevsky. The shop chain Żabka became Абка in Russian and Abka in English, the company Mleczna Dolina became Melena Valley.
Up to here it was half my expectation. I expected the diacritics to go, but I did not expect Piotr to become Peter. And the pronouns I was most worried about came back every time.
Where my check was wrong
My script said that negation was perfect. Then I read all 600 Russian sentences of the first round by eye and found 3 things the script did not see.
All 21 tenants became landlords. The tenant has not paid the rent for 30 months went to Russian as Арендодатель не платил арендную плату. That is the landlord who did not pay. It came back as The lessor did not pay rent for 30 months. The not is still there, so the negation check passed, but now the other person owes the money.
The second one goes in the other direction. The fee is not refundable after 3 days became не подлежит возмещению, which is correct Russian for not refundable. It came back as This fee is not payable. In 9 of 10 such sentences the Russian was right and the English that came back was wrong. So in these 9 the error is on the way back.
The third is the invoice. Sometimes the model translates it as счёт-фактура, the formal invoice. Other times it is plain счёт. The formal word came back as invoice 46 of 46 times. Plain счёт came back as invoice 0 of 20 times, mostly as account. Plain счёт also means a bank account in Russian.
The other direction
The pronouns, where the check said 100 of 100, had a problem of their own.
Russian puts gender on the verb in the past tense. In 16 of 60 sentences with such a verb, the verb follows the usual gender of the profession and not the pronoun. The accountant called because she needs the invoice became Бухгалтер позвонил, потому что ей нужен счет. A Russian reader is likely to see 2 people here, a man who called and some woman who needs the invoice. A masculine verb after бухгалтер is possible for a woman too, so it does not prove 2 people, but it pushes the reader there. In 13 of the 16 it is a male verb next to she. The nurse went the other way 3 times.
The English that came back was The accountant called because she needs an account and the pronoun check passed all 16. English verbs carry no gender, so the error in the middle has no place in the English that comes back. The English is a little ambiguous by itself, she could be someone else, but I read it as 1 person.
More rounds
I expected that the text would stop changing after 2 or 3 rounds. It almost did. Compared with the previous round, 540 sentences changed in round 1, then 172, 63, 31 and 25. Only 60 of 600 came back word for word the same as the original.
138 sentences had lost something after the first round and 142 after 5 rounds. I was ready to write that later rounds add 4 new losses. Then I looked at them. 1 is my own regex: at most 1817 came back as does not require more than 1,817, which means the same. My pattern only knew not more than written together. 1 is only a mix of separators in 1,028 211. 2 change the meaning. Our rent went up to 54,466 zlotys became 54 466 человек in round 2, which is 54,466 people. In round 5 the rent was $54,466. In round 2 the developer who was she became Застройщик, a property developer. The Russian dropped the pronoun and the English came back with he.
Of the 25 sentences still changing in round 5, 17 swing between 2 versions, mostly she needs the bill and she needs an account. Only 8 still change in some other way.
What I did not check
It is one pair of models. I planned NLLB-200 with 600M parameters, released by Meta this July, as the second one. It did not fit on the disk of the machine. The checks are regexes on the words that carry the meaning and they do not understand meaning, so refundable to payable passed the negation check. I read the first round by eye. Rounds 2 to 5 I did not read, so there may be more there. The sentences are short, about 9 words, from templates. Negation has only 36 different ones, so 100 sentences there is less than it sounds. Russian was only the middle language. I did not try Polish or Belarusian in the middle.
Back translation was fine for numbers. It missed the accountant in all 16 sentences and it blamed a correct Russian fee 9 times of 10. Next time I need such a check I will read the middle myself or ask someone who reads it.