Reading UTF-8 as 1251: "Привет"

I wanted to see this in bytes instead of guessing at it again. So I printed Привет as UTF-8 and got 12 bytes for 6 letters. Then I handed them to a decoder that believes one byte is one character:

$ printf 'Привет' | xxd
00000000: d09f d180 d0b8 d0b2 d0b5 d182            ............

$ printf 'Привет' | iconv -f CP1251 -t UTF-8
Привет

My fix for this has always been clicking through an encoding dropdown until the text comes back. It works. I have never once looked at what was underneath.

66 letters, 2 leading bytes

Russian Cyrillic sits at U+0410 through U+044F, with Ё and ё off on their own at U+0401 and U+0451. All of them are 2 bytes in UTF-8. I collected the leading byte of every letter, upper and lower case:

$ perl -CO -e 'print map { chr } (0x401, 0x451, 0x410..0x44f)' \
    | perl -ne 'for my $b (unpack "C*", $_) { $seen{$b} = 1 if $b >= 0xC0 }
        END { print join " ", map { sprintf "%02x", $_ } sort keys %seen }'
d0 d1

2 values for 66 letters. So a single byte code page reading that stream prints its own idea of d0 and d1 in front of every second character. That makes the garbage readable in reverse:

A wall of Р and С means the file was read as 1251. That is the case I hit almost every time. The other 4 are rarer and just as readable off the first column.

CP1252 and ISO-8859-1 agree on d0 and d1, so the fingerprint is the same. They disagree from 80 to 9f. CP1252 has printable characters there. It prints Привет, 12 characters for 12 bytes. In ISO-8859-1 that band is C1 controls. Bytes 9f, 80 and 82 print as nothing at all, so the same string comes out as ÐÑивеÑ, 9 visible characters. I lost half an hour comparing character counts before I worked that out.

The trick is narrower than it looks. Belarusian also fits in d0 and d1, ў included, but Ukrainian breaks it. Ґ is d2 90, which 1251 draws as Т.

Backwards it is not recoverable

Reading 1251 bytes as UTF-8 is the same accident in reverse. It behaves nothing like the first one:

$ printf 'Привет' | iconv -f UTF-8 -t CP1251 | xxd
00000000: cff0 e8e2 e5f2                           ......

$ printf 'Привет' | iconv -f UTF-8 -t CP1251 | iconv -f UTF-8 -t UTF-8
iconv: (stdin):1:0: cannot convert

None of those 6 bytes start a valid UTF-8 sequence, so iconv stops on the first one. The editor I had open on the same file was less honest about it. It put U+FFFD in every position, which looks like a text you can still work with.

The Р/С version keeps every original byte. Feeding the mojibake back through 1251 returns the source exactly:

$ printf 'Привет' | iconv -f CP1251 -t UTF-8 | iconv -f UTF-8 -t CP1251 | xxd
00000000: d09f d180 d0b8 d0b2 d0b5 d182            ............

Saving the mess instead of converting it back is how a file gets encoded twice. 12 bytes become 25.

$ printf 'Привет' | iconv -f CP1251 -t UTF-8 | wc -c
25

KOI8-R put the alphabet in Latin order

KOI8-R looks like somebody shuffled the alphabet. ю comes before а and ц sits between б and д. Here is the upper half of the table with the ASCII character each byte turns into once the top bit is gone:

c0 ю @  c1 а A  c2 б B  c3 ц C  c4 д D  c5 е E  c6 ф F  c7 г G
c8 х H  c9 и I  ca й J  cb к K  cc л L  cd м M  ce н N  cf о O
d0 п P  d1 я Q  d2 р R  d3 с S  d4 т T  d5 у U  d6 ж V  d7 в W
d8 ь X  d9 ы Y  da з Z  db ш [  dc э \  dd щ ]  de ч ^  df ъ _

а is at A, б at B, ц at C, ф at F. Most letters sit on top of their Latin transliteration. The ones with no Latin counterpart took whatever was left, which is how ю ended up on @ and ъ on _. Clear the eighth bit of a KOI8-R string and the transliteration falls out:

$ printf 'Кодировка' | iconv -f UTF-8 -t KOI8-R | perl -pe 's/(.)/chr(ord($1) & 0x7f)/ge'
kODIROWKA

Здравствуйте   fa c4 d2 c1 d7 d3 d4 d7 d5 ca d4 c5   ->   zDRAWSTWUJTE
Кодировка      eb cf c4 c9 d2 cf d7 cb c1            ->   kODIROWKA
Спасибо        f3 d0 c1 d3 c9 c2 cf                  ->   sPASIBO

Case comes out inverted because lower case Cyrillic occupies the upper case ASCII range. Strip the high bit somewhere in transit and the text is still legible. At this hit rate that cannot be an accident. The same operation on the 1251 bytes gives Ophber.

All 3 words I tried happen to use letters from the mapped side of the table. ъ or щ in there would come out as punctuation.

What did not work

I tried to run the longer phrase Здравствуйте, коллеги through every decoder in one pass. CP1252 refused:

CP1252   ЗдравÑ
iconv: (stdin):1:11: cannot convert

It stopped at byte eleven, which is 81. CP1252 leaves 5 byte values undefined. 81 is one of them, so a converter following the table has nothing to return. The C1 band that makes CP1252 and ISO-8859-1 print differently is the same band that makes one of them fail outright.

That band is busy. 34 of the 66 letters have their second byte between 80 and 9f. So a little over half of any Cyrillic text lands where those 2 tables stop agreeing. I have not checked yet what CP1250 or CP1257 do with the same bytes.