Since September I run Whisper on recordings after a call is over. That works, but the text arrives when I no longer need it. I wanted it while people are still talking, so I wrote the simplest live version I could think of: read 5 seconds of audio, give them to transcribe(), print the text, repeat.
It is about 40 lines of Python. ffmpeg writes raw 16 kHz PCM to stdout, the script reads it and a line of text appears every few seconds. For a while I was happy with it. Then I ran the same cutting over an audiobook chapter, read a full page of the output and saw this:
... a golden fortune, and a-
a happy life. He was like unto my father in a way, and yet was not my f-.
father. Also, there was a stripling page who turned into a maid.Every line is one 5 second piece. At the end of a piece Whisper sees half a word and writes f-.. In the next piece it sees the other half and writes father again. Elsewhere the end of a piece came out as thank waitags heißa, which is not a language.
I had assumed that cutting would cost a word here and there and I never measured it, so I did.
The setup
I needed speech with a known correct text, so I did not use my own calls. LibriSpeech test-clean has read books with exact transcripts. I took 4 readers, 2 women and 2 men. For each of them I glued the first chapter into one stream, cut at 5 minutes where the chapter was longer. Together that is 19 minutes and about 3200 words.
Whisper is the repository as of 9 December, model base, on CPU in fp32 with the language set to English. The machine has 4 virtual cores of an AMD EPYC 7713 and torch 1.12.1. I count word error rate after Whisper’s own English text normalizer, so commas and “Mr” against “mister” do not count.
The baseline is the whole 5 minute file given to transcribe() at once. That is how I used Whisper since September and it gave 5.79% of words wrong.
What the cutting costs
The version with 5 second pieces gave 14.89%, 2.6 times more errors on the same audio. I split the errors by where they sit, counting the first 2 and the last 2 words of every piece as next to a cut. That zone is about 30% of the words and it holds 252 errors. The other 70% hold 224, while the whole file had 185 errors on all of its words. So the middle of a piece is also read worse, just less badly than the edges. Extra words went from 18 to 148, about half of them right at a cut.
My first suspicion was luck. Maybe this grid of cuts happened to hit many words, so I moved it by 2.5 seconds and ran it again. It gave 14.11%, so the position of the grid changes less than a point.
Then I tried the 3 fixes that seemed obvious.
- Longer pieces. 10 seconds gave 11.76% and 30 seconds gave 8.20%. Better, but at 30 seconds the text arrives half a minute late.
- Give Whisper the text of the previous piece as
initial_prompt. 14.02%, practically the same. I expected a bit more from it and I do not know why it did so little. - Cut in a quiet place. The script looks between 4 and 6 seconds for the quietest 100 milliseconds. It cuts in the middle of them. This gave 10.01%, the biggest gain for the same delay.
The quiet cut also showed me where I was wrong. In the same sentence it produced a ha- and be life, because the quietest 100 milliseconds in that window were inside the word “happy”. My guess is the short silence before the “p”, but I did not check it on the waveform.
Why even 30 seconds is worse than the whole file
This one surprised me. Whisper looks at 30 seconds of audio at a time, so I expected 30 second pieces to give the same result as the whole file. They gave 8.20% against 5.79%.
Part of the answer is in transcribe(). When it reads a long file, it moves the next window to the last timestamp where a phrase finished. A word cut by the window edge is simply read again, together with what comes after it. My script cuts the audio before Whisper sees it, so Whisper cannot go back. This does not explain everything: of the 77 errors above the whole file, 33 are in the middle of pieces. Maybe it is the lost text of the previous piece, which transcribe() keeps as context inside a file, but I did not separate that.
The same code explains the noise at the ends of pieces. If a piece ends in the middle of a phrase, transcribe() decodes the remaining tail once more as a separate window. When the decoder output looks bad, Whisper retries at a higher temperature. In 20 of the 235 pieces of 5 seconds some later part was decoded at temperature 0.6 or above. The piece with heißa went up to 1.0.
What it costs in compute
Before the encoder, transcribe() pads the spectrogram of every piece to 30 seconds. On this machine the base encoder takes about 1.2 seconds for a padded 5 second piece and the same for a real 30 second one. The whole file costs 0.17 seconds of work per second of audio, the 5 second pieces cost 0.67, which is 4 times more.
I also ran the main variants with small, the next model up. On the whole file it gave 3.60%, with 5 second pieces 13.80% and with the quiet cut 7.57%. On this machine a 5 second piece takes about 12 seconds with small, so live it would fall behind the speech.
What I took for the live version
I kept the quiet cut between 4 and 6 seconds and the base model. I played one chapter through it at real speed. On 51 of the 52 pieces Whisper took between 1.6 and 4.4 seconds. Once it took 11.4 seconds and that line ended with “Who couldn’t believe for safety”, which the reader never said. With a piece of up to 6 seconds on top, a word usually reached the screen within about 10 seconds. Around that stall it took up to about 17.
On this reader the live text had 3.83% wrong words, against 2.63% for the whole file. Over all 4 readers the quiet cut is 1.7 times the whole file, so the file after the call stays the version I keep. I use the live text only to follow the call.
The proper fix is probably overlapping windows. Keep part of the previous audio and transcribe it again, like transcribe() does inside a file. whisper.cpp already does something like this in its stream example. Every 3 seconds it transcribes again a window of up to 10 seconds. When it starts a new line, it keeps 200 milliseconds of the old audio. whisper_mic cuts the microphone on a pause of 0.8 seconds. I have not measured either of them.
What I did not check
- Real conversation. These are books read by one person in a quiet room. 2 people talking over each other will change all the numbers, probably not in the same proportion.
- A real microphone. Only the live run went through ffmpeg at real speed and I did not record anything. All other numbers come from cutting the files directly.
- More than one run. With temperature fallback some pieces can decode differently on a repeat, so small differences like 14.11 against 14.02 mean nothing.
- A GPU. I did not try one. There the padding should cost much less time.