Somebody presents a model at 93 percent accuracy and the room nods. I have been in that room and I nodded. This is what I should have asked for instead.
The task is real and small enough to check by hand. Take a repository, stand at the first of a month, then predict for every file whether anybody touches it in the next 7 days. The features come only from the past: how many times the file changed in the previous 180 days, how long since the last change, how big it is. Training uses the months before July 2013 while testing uses everything after, so the model never sees its own future. The classifier is logistic regression written by hand, 3 features, gradient descent, because the point is not the model.
6.8 percent of the test cases are positive. Answering never to everything therefore scores 93.2 percent accuracy, which is the number to beat.
The best accuracy in the sweep is 93.3 percent. It belongs to the threshold at 0.7, which finds 3 of the 163 files that actually changed.
Every threshold in the sweep sits inside the same band, from 92.5 to 93.3, while recall goes from 19.6 percent down to 0.6. The single number cannot separate a model that finds 1 change in 5 from a model that finds 1 in 163. It also cannot separate either of them from answering never at all.
The confusion matrix separates them in a glance. 3 against 160 is the thing itself rather than a summary of it.
Where the metric works
I want to be fair to accuracy, so here is the same experiment with the horizon stretched from 7 days to 90. Now 34.6 percent of cases are positive, answering never scores 65.4, while the best model reaches 70.7 with recall at 31 percent. The number moves, ranks the thresholds and behaves like a metric.
That is the shape of the problem. Accuracy is a weighted average of 2 numbers whose weights are the class sizes. When 1 class is 93 percent of the data the average is that class. The other one is rounding.
Where I tripped
I ran the 90 day version first, got the mild result above, then had to go back and shorten the horizon to make the imbalance sharp. That deserves saying plainly: I chose the task until the effect appeared. The honest form of my claim is narrower than “accuracy is useless”: the same experiment shows the failure at 6.8 percent positives and hides it at 34.6. That is the sentence I could not have written from the first run.
What I did not check
The area under the curve, which is the metric people reach for exactly here. By construction it ignores the threshold I swept. Whether the model is calibrated, meaning whether a score of 0.7 corresponds to anything happening 70 percent of the time. And the model is 3 features on 1 repository, so nothing here says anything about how well file changes can be predicted. It says something about what the reported number does or does not tell you.
The narrow claim is a question to ask in that meeting. Not what the accuracy is, but how many of the real cases the model found and how often it was wrong when it spoke.