Holster 1.0.24

I've just published Holster 1.0.24 which is quite a major change, as signatures are now added per property rather than per node. Before making this change I was trying to debug a signature validation issue on a node, but there was too much data being signed to know what could've gone wrong.

This led to simplifying signatures to each property, which actually fixed a whole bunch of other problems in Holster. Signing a node means you always have to fetch the full node to update it even though you might only be updating one property. There were also timing issues around node updates that aren't a problem any more.

There were a few other recent releases too, including adding Docker support. See the updated README for how to start the server with either Docker or Node.

Migrating from Gun to Holster

The last few weeks have seen a lot of development activity, first in Holster itself then in applications using it. This has been driven by the Free Association project, which is now capable of using Holster instead of Gun. The start of the migration process required some further updates to Holster, which is now at version 1.0.21 and feeling pretty stable!

The rest of the work was done in switching API's in the application itself. Holster's API is not only slightly different to Gun, but also conceptually different. The biggest difference is that Gun streams data whereas Holster resolves all references before returning. This doesn't mean much from a performance perspective, but it does significantly change your application code. I've done a few of these migrations now so I already had the notes from the first one, which I've used again successfully since. I'm sharing them here as hopefully they're general enough for other migrations to benefit from.

On the server I've been developing with both Node and Bun, the first step in Node is:

npm uninstall gun
npm i @mblaney/holster

My previous Gun initialization code was:

import Gun from "gun"
import "gun/lib/then.js"
import "gun/sea.js"

const gun = Gun({
  web: express().listen(8765),
  multicast: false,
  axe: false,
})

This has been replaced by:

import Holster from "@mblaney/holster/src/holster.js"

const holster = Holster()

Holster is a module so the path to the file is required. If you're not using a build tool you can switch src to build in the path above. The only option I left out of both examples above was secure: true, which means only data under user accounts will be saved (no public data).

Here is the browser code I previously used for Gun (also not including secure here):

import Gun from "gun"
require("gun/lib/radix.js")
require("gun/lib/radisk.js")
require("gun/lib/store.js")
require("gun/lib/rindexed.js")
require("gun/lib/then.js")
require("gun/sea")

const gun = Gun({
  // Assume Gun is directly available on localhost, otherwise reverse proxy.
  peers: [
    window.location.hostname === "localhost"
      ? "http://localhost:8765/gun"
      : window.location.origin + "/gun",
  ],
  axe: false,
  localStorage: false,
  store: window.RindexedDB(),
})

And here is the browser code for Holster. Note that it doesn't rewrite the scheme for WebSockets like Gun, so you need to specify it or use the default:

import Holster from "@mblaney/holster/src/holster.js"

// If on localhost assume Holster is directly available and use the default
// settings, otherwise assume a secure connection is required.
let peers
if (window.location.hostname !== "localhost") {
  peers = ["wss://" + window.location.hostname]
}

const holster = Holster({peers: peers, indexedDB: true})

The rest of these notes are for the API's in general, so apply to both browser and server.  First is error handling, where Gun returns an object with an err property Holster returns just an error. This means a lot of code changes from ack.err to just err.

The SEA functions work pretty much the same, except Gun uses encryption keys under user._.sea, Holster simplifies this to rely on the user.is object. They can also be used on their own with import SEA from "@mblaney/holster/src/sea.js"

I used user.get(key).once(callback) a lot when working with Gun. In Holster, that pattern is now just user.get(key, callback). I would also previously use user.get(key), and pass that around as a reference to use later. Holster removes the context that gets created once a callback is provided, so that isn't possible and you instead need to write the full query each time.

As mentioned above, the big difference is between streaming data and resolving nodes. So when using Gun you could do: get("items").map().once((item, key) => callback) to get a stream of updates. In Holster it's just get("items", items => callback). So to get the equivalent of Gun's map() you would do: Object.entries(items).forEach(([key, item]) => {callback(item, key)}) on the returned data if you wanted to use the same callback.

Also get(key).map().once(callback) subscribes to updates in Gun, you would need to use get(key).on(callback) instead in Holster for updates.

Gun has get(key).then() which is nice for awaiting data, but it's just a convenience function which can be done in Holster with:  await new Promise(res => {get(key, res)})

The biggest change in terms of lines of code has generally been modifying get().get() chains from Gun, to get().next() in Holster. One more thing to watch out for is calling put() in a loop, it's better to use an actual for loop rather than forEach and awaiting each put() so they don't make updates at the same time, especially to the same node.

Holster 1.0.19

This is another bug fix and performance update, message queuing felt a little slow so I've bumped up queue processing from 4 messages per second to 10, we will see how that goes.

The other fixes are more data sync related than timing, the first is that listeners can subscribe to nodes, but the underlying data can change from a property to a reference. A bit of an edge case but this led to listeners not resolving full objects, which is now fixed.

Next was that updates that didn't change the in-memory graph weren't going out on the wire, so other devices on the network weren't alerted. There's also some better checks for when a websocket isn't available messages won't be dequeued until they are actually sent.

Post-performance improvement updates

The last update was all about improvements that made Holster more efficient on the server. I'm glad to say those changes, plus a couple of recent updates, mean server processing is looking really good now. There are no more CPU spikes for the same processing load, which means Holster is happy running on small severs when given a reasonable amount of work to do.

The big change that allows this to work efficiently is that Holster now stores all radix trees in memory. This allows for much faster gets and puts less pressure on the radisk implementation by no longer requiring parsing from disk as often. This does mean you need to configure your application to provide Holster with enough memory. There is a new memoryLimit option which has a default of 500MB, which is fine for small amounts of data. If you plan to store more data you will need to increase this, but Holster also logs how much memory it's currently using so you will know when it's an issue. When it hits the limit it resets the cache and goes back to files on disk, so it does continue working but this process will probably repeat itself.

The other change required to keep servers running well was that clients needed to throttle their message sending. This also helps browsers not spike a user's CPU and memory because it's making requests that the server couldn't keep up with anyway. This led to some changes that resulted in orderly message queuing and processing, but introduced some timing issues now that have now been fixed in 1.0.17.

To check for these timing issues there's now a whole set of new system tests. I've continued using Claude to write these tests and help with debugging the timing issues which is working really well.

Performance Improvements

My project to test Holster started hitting 100% CPU the other day during peak processing times, so have been doing some investigation. It's not a big server, but Holster shouldn't need the resources it was using and this was a great opportunity to identify what needed improving.

This has led to a few performance improvement releases this week. The first was that the caching radisk was doing was only for the most recent file. This doesn't do very much when keys come in randomly for different files, and each key caches a new file. Holster is currently not designed to hold heaps of data, so it now just caches all files, which was a big improvement as reading from disk is expensive.

Next was improving the client throttling to use an actual processing queue. It previously tried to throttle with timeouts but this just caused messages to build up later. The advantage of a processing queue is that you can also return an error if the queue gets too long, because there is probably something wrong with your Holster queries if this happens (ie requesting too much data.)

Lastly the Holster website got an update! It now has a nice little visualization of connected clients.