earshot, my local transcriber, went public in August. Its README has a table I wrote about audio capture. Microphone works on all 3 systems. For system audio, meaning what the speakers are playing, the table says “works out of the box” on Linux. Windows needs Stereo Mix or VB-CABLE and “macOS 12 and later” needs BlackHole. Under the table I wrote that macOS gives an ordinary process no system audio capture API at all.
I wrote that line from memory. The memory is from the years when everybody installed Soundflower to record a Mac. I never checked if it was still true. The Linux row I never questioned either, because on my machine it just worked. This week I went through both rows.
What the operating systems actually offer
I don’t have a Windows or a Mac machine for this, so this part is from the documentation and from source code.
Windows has had loopback capture in WASAPI since Vista. You open a capture stream on a playback device with the AUDCLNT_STREAMFLAGS_LOOPBACK flag. There is also process loopback, which captures one application and its child processes. Microsoft lists Windows 10 build 20348 as its minimum. No Windows 10 client release reached that build, it is the build of Windows Server 2022, so in practice it means Windows 11 or that server. macOS got audio in ScreenCaptureKit with macOS 13. Since macOS 14.2 Core Audio also has AudioHardwareCreateProcessTap, a tap on the output of one process or of all of them. So my sentence about macOS is wrong for every version since 13.
earshot still needs a virtual device because of ffmpeg. earshot never opens an audio device itself, it starts ffmpeg, reads 16 kHz mono PCM from its stdout and leaves the platform to ffmpeg. Today, on 11 October, I listed libavdevice on the ffmpeg master branch. For Windows audio it has dshow and no WASAPI input. On macOS the avfoundation input has no ScreenCaptureKit code, so for audio it sees capture devices and nothing else.
Linux, measured
The Linux row I could measure. The node I work on is a server without a sound card. So I unpacked PulseAudio 16.1 and PipeWire 1.0.5 with WirePlumber 0.4.17 from the Ubuntu 24.04 packages into a local folder and created virtual outputs. Both servers were driven by the same pacat and pactl, so the client side is identical. The test sound is 3 seconds of seeded noise plus a 1 kHz tone in 48 kHz stereo. Every capture is aligned to the original and compared sample by sample.
At 100% volume the monitor gives back the sound bit for bit. That is 144,000 of 144,000 frames, on both servers and in every run of the final harness.
Then I moved the slider on PulseAudio. With the output at 50% the capture is the original multiplied by 0.125, up to rounding of the last bit. That is 0.5 cubed, the volume curve of PulseAudio. Muting the output gives pure zeros. Setting the volume of a single application to 50% while it plays also gives 0.125. So on PulseAudio the monitor reads the sound after all of these controls.
It also remembers. After I turned one application down, its next stream started at 0.125 again. The server keeps that volume on disk, keyed by the role of the stream or, without a role, by the application id or name. For earshot that means the old setting of an app also applies to its next call.
I almost wrote that PipeWire behaves the same. On my first outputs it did. Then a check against the source showed why. I created those outputs through the PulseAudio compatibility layer of PipeWire. That layer turns on monitor.channel-volumes for them. PipeWire’s own default is off. WirePlumber 0.4.17 leaves it off for real sound cards. So I made one more output directly in PipeWire with the default. On it the output at 50% and the muted output both gave the sound back bit for bit. The volume of the application still gave 0.125 and was still remembered.
Switching the output is fine on both. I started recording from @DEFAULT_MONITOR@, switched the default output to another one and played there. Both servers moved the recording to the new monitor and the capture was bit for bit again.
Where it starts to hurt
earshot does not send every sound to the model. It cuts the stream into 20 ms frames. A frame counts as speech when its RMS is at least 600 on the 16 bit scale. A clip is kept only if it has at least 300 ms of such frames. I picked these defaults and never thought about them together with the volume slider.
So I played a speech recording at different volumes. I captured it with ffmpeg 6.1.1 and the same arguments earshot builds on Linux, into a file instead of stdout. Then I fed the result to the segmenter of earshot at commit 15da817 with its default settings. The recording is 10 Harvard sentences from the Open Speech Repository, converted to stereo for playback. At 100% it peaks at about -12 dBFS and its loudest 20 ms frame has RMS about 3,500 to 3,650, depending on the run.
On PulseAudio, with the output slider, earshot cuts 10 clips from the 10 sentences down to 80%. At 70% it cuts 8. From 60% down it cuts none. At 60% the loudest frames still pass 600, since 3,650 times 0.216 is about 790, but only 18 frames in the whole recording do. No clip collects 300 ms of them. On PipeWire the output slider changes nothing and earshot cuts all 10 clips even at 30%. The application slider brings the same cliff back there: 9 clips at 70%, none from 60% down. The 8 against 9 at 70% I put down to rounding and frame boundaries and did not check further. Nothing fails in any of these runs, ffmpeg and the segmenter keep running and the transcript stays empty.
My own harness also fooled me for a while. In the first runs the recorder started 1 second before playback and stopped 1 second after. Some captures lost up to a second at the start or a piece at the end. With 3 seconds on each side none of it came back. The lost ends were pipewire-pulse handing the recorder data in large chunks. Why the start went missing I don’t know and I did not dig.
What changes for earshot
The README row for Linux should not say only “works out of the box”. It works, but what it records depends on the volume. On PulseAudio the output slider and mute decide if earshot hears speech at all. On PipeWire only the volume of the application does. In both cases an empty transcript comes with no error. The macOS line is simply wrong. It needs to go from the README and from docs/audio-capture.md too.
For Windows and macOS the honest fix is a native capture path next to ffmpeg. That means WASAPI loopback on one side and a process tap on the other. It is more code than earshot has today. I have not written it.
What I did not check
Nothing here ran on Windows or macOS. I don’t know if WASAPI loopback or a process tap sees the sound before or after the system volume.
All outputs were virtual. PulseAudio can apply part of the volume in the sound card hardware. In its source that part comes after the monitor, while mute is always done in software. So on a laptop with PulseAudio the cliff may sit lower on the slider and mute should still give silence. I read that in the code and could not run it. My PipeWire output only copies the default setting of a real card. It is not a real card either.
The cliff itself belongs to this one recording. A quieter call moves it up the slider and a louder one moves it down. The threshold of 600 can also be changed with --threshold.