The pitch for Benford’s law is that you point it at a column of numbers and the faked ones stand out. Real quantities start with 1 about 30 percent of the time and with 9 about 4.6 percent of the time, invented ones spread out flat, so the histogram does the accusing for you. I have seen that histogram in slides for 2 years and I never once ran it on numbers I knew were clean.
So I ran it on 4 columns where nobody has a motive. All of them come from the git history of express, cut off at 2013-06-12, which is 4108 commits and a tree of 197 files at aec34284. Nobody inflates how many lines a commit adds. The columns are the lines added by each commit, the size of every file in bytes, the length of every file in lines and the seconds between one commit and the next.
3 of the 4 fail.
The numbers behind the picture. Mean absolute deviation is the average gap between the 9 observed shares and the 9 predicted ones. Nigrini’s thresholds put close conformity under 0.006, acceptable under 0.012 and nonconformity above 0.015. Lines added by a commit give 0.0259. File sizes give 0.0242. File lengths give 0.0287. Seconds between commits give 0.0065, which is the only column that passes.
I expected the span of the data to explain that. The intervals run from 1 second to 6209407 seconds, which is 6.79 orders of magnitude. A column that covers that much room has to spread its first digits the way Benford says. The other 3 columns cover 3.06, 3.36 and 4.61 orders. That is the standard explanation and it is half right: the file sizes cover 4.61 orders, half again as much room as the file lengths. They still come out at 0.0242 while the file lengths come out at 0.0287.
The 2 screens disagree
Then I ran the other standard test, chi square, on the same 4 columns. With 8 degrees of freedom the table says 15.51 at 5 percent and 20.09 at 1 percent.
The file sizes score 13.3 and chi square does not reject them, while MAD calls them nonconformant. The intervals score 20.1 against a threshold of 20.09, which is as close to the line as a verdict gets. Chi square rejects them at 1 percent while MAD says they are fine. So on the same repository, in the same hour, 2 published screens hand back opposite verdicts. Which one you get depends on which test the auditor happens to run.
The mechanism is not subtle once you write the formula down. Chi square multiplies every squared deviation by the sample size, so a fixed shape of error crosses the threshold as soon as you have enough rows. The file sizes have 197 values and the intervals have 4043. That ratio of 20 is doing more work in the verdict than any property of the numbers. MAD has the opposite blindness: it divides the deviation by 9 and never asks how many rows produced it, so it treats 158 file lengths and 4043 intervals as equally trustworthy.
Where I tripped
The commit column has 4108 rows in git and 3587 in my histogram. 521 commits add no lines at all: merges, deletions, a permission change. A leading digit does not exist for 0, so those rows drop out silently. It took me a while to notice that my sample was quietly 13 percent smaller than the history I said I was measuring. Dropping them is not neutral either. They are exactly the commits at the small end. The small end is where the excess of 1 comes from.
The excess is real and it has an ordinary cause. A repository at this age is mostly small changes. Once a great many commits touch between 1 and 9 lines, the digit 1 collects them all. Nothing is being hidden. The screen sees the honest shape of software work and calls it suspicious.
What I did not check
Whether a planted fraud would even move these columns. If honest data sits at 0.026 and the threshold is 0.015, the interesting question is how much faking you would have to add before the number changes in a way an auditor could tell from this baseline. I have not measured that. I also ran only the first digit test. The second digit test and the last 2 digits test are the ones the accounting literature leans on. They may behave better or worse here. And this is 1 repository of 197 files, chosen because I had it on disk.
The claim I would keep is narrow. Benford’s law is a statement about numbers that span many orders of magnitude. A screen built on it inherits that condition, but the condition is not stated on the slide with the histogram. Point it at a column that lives inside 3 orders and it will fire, every time, at nobody.