Benchmarks
My performance testing of Holster so far has come from just working on the applications I want, and then seeing where things feel slow. This has the double pay-off of improving the usability of the apps and improving Holster overall.
I decided to add some benchmark testing though, to get a better sense of where things are at. This was a task Claude was able to set up very quickly, and when I first had some results I wasn't sure what to compare them to. I first looked for existing GunDB performance numbers but knew that benchmarking is notoriously subjective, so decided to add side by side testing in the benchmark itself. Everything is in the repo, so others can try it or at least see the assumptions made to produce the results below. Both Holster and GunDB needed some config tweaks to produce comparable results.
Adding benchmarking showed where a number of performance improvements could be made, so these have also been added to the latest Holster release and are also reflected in the results. The main fix was updating the batch size used when writing to disk back to the same value as GunDB, which was causing disk churn under load. Also a callback timeout was used very often but wasn't required, which is a more noticeable improvement in real applications.
The aim here is to get both Holster and GunDB writing to disk and then checking whether the data can be read back afterwards. It tries to make a fair comparison by reducing GunDB's write delay to 1ms to match Holster, down from its default 250ms delay. Update: This post previously showed verification stats where only Holster was reading back every record. The great news for GunDB is that the latest commit on GitHub also reads back every record, so those columns have been removed! 🎉️ Note that this version hasn't yet been published to npm, I will update the results again when it has.
Setup
Holster 3.0.0 and Gun GitHub commit 6b72bd1.
Node v22.22.1 on an Intel Core Ultra X7 358H, writing to an ext4 NVMe drive.
Both libraries wait 1ms before writing a batch (--gun-until 1), and both batch up to 10,000 writes into files of up to 1 MB.
Every process ran on the four fast cores, CPUs 0 to 3 (--cpu 0-3), so results don't depend on which cores a run is given.
Each figure is the median of 3 runs.
Results
Scenario | Holster (writes/s) | Gun (writes/s) | Holster median latency (ms) | Gun median latency (ms) |
|---|
small-sequential | 94 | 135 | 10.5 | 7.3 |
small-concurrent | 3,969 | 7,293 | 21.2 | 11.3 |
small-burst | 7,984 | 12,140 | 122.0 | 74.8 |
large-1kb-concurrent | 6,919 | 7,979 | 12.8 | 11.0 |
many-properties | 271 | 225 | 3.5 | 4.3 |
large-1kb | 278 | 211 | 3.5 | 4.6 |
large-10kb | 238 | 184 | 4.0 | 5.0 |
large-100kb | 173 | 107 | 4.8 | 7.4 |
overwrite | 305 | 297 | 3.2 | 3.3 |
user-small-sequential | 215 | 176 | 4.5 | 5.3 |
user-small-concurrent | 1,126 | 548 | 81.8 | 181.9 |
user-overwrite | 245 | 281 | 4.1 | 3.5 |
radisk-direct | 19,605 | 12,637 | 4.9 | 7.3 |
Findings
The two are quite similar in many ways, as you would expect since Holster is a fork of an early version of the GunDB code. The radisk numbers look huge for both Holster and GunDB, but that's because they're writing straight to disk and get to skip all the API overheads of the other tests. The reason why Holster is faster on this benchmark is that it keeps files it has already parsed in memory, whereas GunDB reads and parses each file again before every write.
GunDB is faster for small public records because it uses keys as the ids it stores data under, so records with sequential keys land in the same file. Holster uses random ids, which spread a batch across files. This is possibly something Holster could improve on later. Holster is faster for large values, many properties on a node and writes to user space.
The most important result for me in the numbers was being able to find areas of Holster to improve, which are now in the 3.0.0 release. Most of the improvements don't make a difference at an application level, but being able to run a server on fewer resources is also an important goal and a big win here.