In August I put a small benchmark on GitHub, preview-cost. It asks what a live preview costs if you build it on a transaction rollback. The preview writes the slider’s overrides, reads the model back, computes and rolls back. The other strategy is a copy in memory that never touches the database. On Postgres 16 in Docker the rollback lost all 4,500 pairs, by 6.16 ms at 40 plans.
The run section of the README makes 2 promises. H2 in Postgres mode is the default, so it promises a run in 10 seconds “with nothing installed”. The numbers in the table come from real Postgres, “because H2 in memory does not reproduce what a write and discard costs a real database.”
I wrote the second sentence without ever looking at an H2 run closely. It is what everybody says about H2 and I say it too when somebody wants to test against it. Use the real database, the embedded one lies. So the default mode of my repository was a mode the same README says does not reproduce the numbers. This time I ran it.
10 seconds
I cloned the repository at commit 26dd519 into an empty directory with an empty Gradle cache, on a VM with 4 vCPU and Gradle 9.8.0. There is no Gradle wrapper in the repository, so “nothing installed” already means JDK 25 plus Gradle. To be fair, the README says so in the sentence right before it.
With only JDK 21 on the machine the build stops after 16 s. The toolchain asks for version 25 and Gradle has no repository to download it from. With Temurin 25.0.4.1 the first gradle run on the empty cache took 24 s. The next 3 runs took 6 to 7 s each. So 10 seconds is true from the second run. Somebody who clones the repository for the first time waits 24 s.
What H2 says
The default run is 5 fresh JVMs with 300 paired previews each. I ran the unchanged Bench against H2 2.3.232 in memory and against PostgreSQL 16.14 installed locally on the same VM with pgjdbc 42.7.4. Every size ran twice. The table shows the median of rollback minus copy.
| plans | H2 in memory | PostgreSQL 16.14, local | README, Postgres in Docker |
|---|---|---|---|
| 10 | 0.50 / 0.54 ms | 0.78 / 0.76 ms | 6.54 ms |
| 40 | 0.50 / 0.46 ms | 0.91 / 0.83 ms | 6.16 ms |
| 120 | 0.49 / 0.47 ms | 1.22 / 1.18 ms | 7.02 ms |
On every backend and size the rollback was slower in 99.4% to 100% of pairs. H2 gives the same verdict as the README, with a number 12 to 15 times smaller. Local Postgres is 1.4 to 2.6 times away from H2 while the README Postgres is 6 to 9 times away from local Postgres.
I expected H2 to make the rollback nearly free, because nothing leaves the process. It does not. I split the rollback preview into its steps in one JVM. On H2 in memory the update of 4 rows alone took 238 to 291 µs. At 40 plans that is about 4 times the whole calculation, which took 59 µs.
Where the difference sits
H2 can also run as a server. It is the same engine with the same storage in memory. But the driver talks to it over a TCP socket on localhost, the way pgjdbc talks to Postgres. I added it to the grid next to the others, together with H2 on a file on disk. The file landed on H2 in memory, 0.49 to 0.59 ms, so it is not in the chart.
H2 over TCP gave 0.84 / 0.83, 0.82 / 0.82 and 1.22 / 1.25 ms. That is within 0.1 ms of local Postgres at every size, including the rise at 120 plans. So on this harness most of the gap between H2 and Postgres goes with the socket and the server behind it. My sentence did not mention either. I checked it again on the same VM while another job was loading it. Then H2 over TCP sat 0.04 to 0.23 ms above Postgres at 40 plans, so the 0.1 ms is from a quiet machine.
The write and the discard differ less than I thought. On H2 in memory the update took 238 to 291 µs, on Postgres 312 to 389 µs. For the rollback step it was 78 to 90 µs against 188 to 205 µs, where Postgres also pays a round trip.
Most of the rise with plans comes from the read back. Each preview reads every field of the model, 42 rows at 10 plans and 482 at 120. On Postgres that read took 387 µs and 1,006 µs. H2 in memory hands the same rows over in 128 and 174 µs, I think because no bytes cross a socket. My README called the overhead “a constant” and made it the main finding. On local Postgres it grew by about half, from 0.76 to 0.78 ms at 10 plans to 1.18 to 1.22 ms at 120.
Round trips
A rollback preview on Postgres is 3 round trips: the batch of 4 updates, the select and the rollback. I counted them with a small proxy between the driver and the database. 100 previews added 300 writes from the client to the 12 writes that the setup with its schema and seed rows took without any preview.
The same proxy can hold every client write for a fixed time before passing it on. That is a crude model of a slower path to the database. I ran the unchanged Bench through it at 40 plans with 5 different delays.
Without a delay the proxy itself added about 0.2 to 0.3 ms in that run, 1.12 ms against 0.83 to 0.91 directly. The first 250 µs of delay added 1 ms, more than 3 times 250. After that the line goes up about 3 µs for every microsecond of delay, which is the 3 round trips. The extra 0.25 ms is probably the proxy’s own wait overshooting each time it sleeps, but I did not measure that separately. At 2 ms per round trip the penalty was 7.52 ms, a bit above the README numbers.
Between the measured points at 1000 and 2000 µs, 6.16 ms falls at about 1.6 ms per round trip. This is only arithmetic. I don’t have the August setup here. This harness reaches the README numbers at a round trip of that size, but I can’t say Docker on that machine had one.
What I would change in the README
The 10 seconds can stay, with a note that the first run takes about 25. The H2 sentence is wrong in its reason. H2 in memory gives the same verdict as Postgres with a number within 3 times of a local Postgres. It has no socket and H2 in server mode adds one back. The results table should say which Postgres was measured and how far away it was. On my numbers the distance is most of the 6 ms.
The conclusion of the original README does not move. 1 ms or 6 ms behind a 400 ms debounce is invisible to the user. The cost that matters is the rows written and locked for every preview.
What I did not check
I did not run Docker at all, on any machine. Concurrency is also missing, which the README already names as the next thing to build and which is where the locks start to cost. H2 was only 2.3.232 and the driver only pgjdbc 42.7.4. The grid ran twice and the proxy series once. For the step split I used one JVM per cell, so its microseconds are rough. On this VM the calculation is also up to 2.6 times faster than on the August machine. The copy in memory takes 0.17 ms at 120 plans against 0.45 in the README. The calculation cancels in the difference, because both sides of a pair run it. But the driver and the database run on the same CPU, so on a slower machine my 0.9 ms would probably grow too.