In August I opened codecard. It renders a code or log snippet as a card for a feed. Before drawing, it reads the text and refuses if the text names a codebase. That means a package line, an import of your own code, a build path with a username, an internal host or something shaped like a credential. It fails closed, with exit code 2 and no file. I made it this way because a warning printed at 1am gets scrolled past and the image stays on somebody’s timeline.
The guard has 11 rules and 171 tests and they all pass. But every case in that suite was written for the suite. The example in the README is a Java file I made up so that it has exactly 5 leaks. The guard never saw code that I did not write, so I ran it on code I did not write.
I took codecard from GitHub at its last commit, f10a09e, on Python 3.12. Next to it I put 9 public repositories at their last commit before 1 October. They are in 8 languages from Java to Rust, among them spring-petclinic and vite. From each repository I took up to 300 files. From each file I took 2 windows of 15 lines, the top of the file and one random window from the rest. That second one is roughly what I copy when I want to show a function.
How often it refuses public code
Everything in this corpus is already published, so every refusal here is a refusal of code that is not a secret. On the random windows from the middle of a file it refused 17 of 790, which is 2.2%. On the top of the file it refused 138 of 1200. 132 of those are C# files in MediatR, where the namespace sits in the first 15 lines. By its own definition the guard is right there, because a namespace does name the codebase. This one is public and --guard off exists for that case.
Some of the 17 are noise. In express it refused a test with the URL /users/0-2. Among other folders, the path rule looks for /Users/ on macOS and /home/ on Linux. But codecard compiles all its default rules with re.I, so a lowercase route matches too. Over all files of the corpus the path rule matches 107 lines and 23 of them match only because of that flag. In okhttp it refused imports of org.robolectric and org.bouncycastle. The private import rule has an allowlist of 26 group names and treats every other name as your own code.
2.2% is a price I accepted in August without knowing the number and now I know it. A refusal costs me one --guard off. The case flag is different, that one I have to fix.
The line it promised to catch
Then I ran the guard on the declaration line of every file without sampling, one line at a time. Java gave 106 of 106 caught and C# gave 133 of 133. Kotlin gave 0 of 569.
The rule wants a semicolon at the end of the package line and Kotlin does not put one there. When the rule fires, it prints the reason “a Java or Kotlin package name is your company and product, spelled out”. I wrote that sentence and the regex while looking at a Java file. The tests for it are Java too.
It is worse than one missed line. I took the leaky example from the README and rewrote it as Kotlin. The guard still refuses it, with 4 findings: the import, the key, the host and the path. The package is not in the list. When I fix the 4 things it told me about, the card renders with package com.acmebank.ledger.settlement on line 1 and exit code 0. The same file with a semicolon gets exit code 2. At 1am the list looks complete, so I would fix the 4 and post.
Next to a secret scanner
Then I wanted to know what the guard misses. Real leaks I can not count, so I wrote a list of 31 shapes that come out of an editor paste or a log. I ran them through codecard and through gitleaks 8.30.1, the scanner I would use on a repository. The secrets in the list are random strings of the right format, generated at run time.
The 2 tools barely overlap. Of the 18 shapes that name a codebase, codecard caught 10 and gitleaks none, which is fair, because gitleaks is built for secrets. Of the 13 secrets gitleaks caught 12 and codecard 6. Its credential rule only sees a value in quotes after a name like api_key or password. A line from a .env file renders. So do a password in YAML without quotes, an Authorization: Bearer header in a log and a Slack webhook URL. This one hurts a bit. I made the ember theme for failing logs and a log is exactly where a bearer token sits without quotes.
Together they caught 23 of 31. All 8 that neither of them caught name a codebase. Among them are the Kotlin package and an internal host without https:// in front. The hardest is a host on the company’s own domain and I do not know how to catch it without knowing the company. The README already says that the guard does not know what your company is called.
The first line of the README
The README starts the install with pip install codecard. I ran it in a clean virtualenv and pip found no distribution. The name is not registered on PyPI at all. So the README of a tool that exists to stop leaks tells people to install a name that anyone can register. It has been like this since the middle of August and I did not notice.
What I did not check
The 31 shapes are my own list, so they say which shapes pass and not how often people paste them. The corpus is 9 repositories and 1 random seed. A different window size could move the 2.2%. I did not run trufflehog. I also did not fix anything yet. The semicolon, the case flag, the unquoted secrets and the PyPI name are still open in codecard at f10a09e.