Benchmarks
My performance testing of Holster so far has come from just working on the applications I want, and then seeing where things feel slow. This has the double pay-off of improving the usability of the apps and improving Holster overall.
I decided to add some benchmark testing though, to get a better sense of where things are at. This was a task Claude was able to set up very quickly, and when I first had some results I wasn't sure what to compare them to. I first looked for existing GunDB performance numbers but knew that benchmarking is notoriously subjective, so decided to add side by side testing in the benchmark itself. Everything is in the repo, so others can try it or at least see the assumptions made to produce the results below. Both Holster and GunDB needed some config tweaks to produce comparable results.
Adding benchmarking showed where a number of performance improvements could be made, so these have also been added to the latest Holster release and are also reflected in the results. The main fix was updating the batch size used when writing to disk back to the same value as GunDB, which was causing disk churn under load. Also a callback timeout was used very often but wasn't required, which is a more noticeable improvement in real applications.
The aim here is to get both Holster and GunDB writing to disk and then checking whether the data can be read back afterwards. It tries to make a fair comparison by reducing GunDB's write delay to 1ms to match Holster, down from its default 250ms delay. I think this delay might have been set to reduce failed writes, but my testing at the default, and at in-between values only reduced the failures, it didn't account for them completely.
Setup
Holster 3.0.0 and Gun 0.2020.1241.
Node v22.22.1 on an Intel Core Ultra X7 358H, writing to an ext4 NVMe drive.
Both libraries wait 1ms before writing a batch (
--gun-until 1), and both batch up to 10,000 writes into files of up to 1 MB.Every process ran on the four fast cores, CPUs 0 to 3 (
--cpu 0-3), so results don't depend on which cores a run is given.Each figure is the median of 3 runs. Verified counts how many of a sample of the records were read back correctly afterwards, in a fresh process.
Results
Scenario | Holster (writes/s) | Gun (writes/s) | Holster median latency (ms) | Gun median latency (ms) | Holster verified (records) | Gun verified (records) |
|---|---|---|---|---|---|---|
small-sequential | 96 | 111 | 10.2 | 9.2 | 1001/1001 | 675/1001 |
small-concurrent | 3,939 | 6,237 | 21.8 | 14.6 | 1001/1001 | 984/1001 |
small-burst | 7,932 | 12,259 | 122.1 | 74.4 | 1001/1001 | 922/1001 |
large-1kb-concurrent | 6,897 | 7,135 | 12.5 | 12.3 | 101/101 | 101/101 |
many-properties | 276 | 171 | 3.5 | 5.6 | 1/1 | 0/1 |
large-1kb | 285 | 237 | 3.5 | 3.9 | 101/101 | 101/101 |
large-10kb | 247 | 195 | 3.8 | 4.8 | 51/51 | 51/51 |
large-100kb | 179 | 103 | 4.7 | 8.0 | 11/11 | 11/11 |
overwrite | 309 | 265 | 3.2 | 3.5 | 101/101 | 101/101 |
user-small-sequential | 210 | 128 | 4.7 | 7.5 | 101/101 | 42/101 |
user-small-concurrent | 1,124 | 848 | 81.9 | 106.8 | 201/201 | 194/201 |
user-overwrite | 248 | 262 | 4.0 | 3.6 | 21/21 | 21/21 |
radisk-direct | 19,415 | 9,868 | 4.9 | 9.6 | 3003/3003 | 2967/3003 |
Findings
The two are quite similar in many ways, as you would expect since Holster is a fork of an early version of the GunDB code. The reason Holster exists was the missing writes, so they're not unexpected here. The radisk numbers look huge for both Holster and GunDB, but that's because they're writing straight to disk and get to skip all the API overheads of the other tests. The reason why Holster is twice as fast on this benchmark is that it keeps files it has already parsed in memory, whereas GunDB reads and parses each file again before every write.
GunDB is faster for small public records because it uses keys as the ids it stores data under, so records with sequential keys land in the same file. Holster uses random ids, which spread a batch across files. This is possibly something Holster could improve on later.
The most important result for me in the numbers was being able to find areas of Holster to improve, which are now in the 3.0.0 release. Most of the improvements don't make a difference at an application level, but being able to run a server on fewer resources is also an important goal and a big win here.
