mal.haza.website

The Holster website has been gone since rebuilding i.haza.website, it's now back as a replacement for the github wiki docs with updated information for all API's and better examples 🎉

I Haza Website also got some updates as I was working on this, so it now does nice preview navigation while you're editing pages. You can start from one page in preview mode, navigate to another page you're working on and continue editing from there.

There's also a lot of code examples on the site so I added a Copy Code plugin that allows you to copy code blocks if you want to run them interactively in Node. There's also a theme switcher plugin for light and dark mode:
 

A new Holster release is available that provides await support.

Using Holster for building i.haza.website was very different than working on RSStream as it needed to be much more responsive to Holster data changes. Looking back at the code up until this point, it was full of await new Promise(resolve => { }) wrappers so that state could update in the correct order.

The latest Holster release removes all of this from application code by providing a Thenables interface onto terminal API calls. That sounds weird but it does exactly what you need, which is when you would otherwise provide a callback to the Holster API you can just await the call instead. Internally Holster knows when a callback isn't provided and can extend the response to include a then() function, if you were chaining the request then the extra function is ignored, otherwise it provides your data.

This works for get and next in the API, put requests can also be awaited but they now internally return a promise as they only ever returned an error or null anyway. It allows for simplifying get requests like:
 

// From wrapped calls:
const existing = await new Promise(resolve =>
  user.get("mediaRootId", resolve),
)

// To just:
const existing = await user.get("mediaRootId")

And previously put requests where you waited for the result were:

async function bumpProjectsVersion(user) {
  const now = Date.now()
  return new Promise((resolve, reject) =>
    user
      .get("projectsVersion")
      .put(now, err => (err ? reject(new Error(err)) : resolve(now))),
  )
}

Are now:

async function bumpProjectsVersion(user) {
  const now = Date.now()
  await user.get("projectsVersion").put(now)
  return now
}

It also makes it easier to work with try and catch blocks, but it's also nice that the old callback style isn't effected by these changes when you want to use it.

Happy to share that i.haza.website is now available again as a new local-first app!

It was previously a PHP app with a similar goal to what the new version is doing, which is to make it easy to publish web pages. The new version is a web app backed by Holster, and besides publishing websites is great for taking notes and transferring media between devices. Holster provides automatic syncing between devices that you're logged in to, as well as encryption and offline support.

This has been another app where the complimentary goal has been Holster improvements alongside having useful tools. This has been true again because I Haza Website is much more interactive with the Holster API then other apps, so it's caught a few things that have needed fixing.

This blog hasn't been replaced yet because I Haza Website doesn't have some of the blogging features of the old app. Expect that to change soon though, I'm looking forward to moving to the new interface it's much nicer to write in. 😊

Another Holster release to add a new nested listener feature that I've needed for multiple projects now. In RSStream I added it outside of Holster itself, since I wasn't sure how often I would need it.

The idea is that you add a listener to a query in Holster, but that query always refers to a particular node. That node can reference other nodes, but those references don't change. So even though the API chains nodes together you need to explicitly list which ones you want to listen to changes on. The external way to do this is to create on() callbacks for each of the child nodes in the parent's on() callback. This works fine, but the code is really the same no matter what the project. So it's now included with Holster and just needs the extra parameter to the parent query to set up all listeners below that point. They're also automatically removed when the parent off() is called.

I asked Claude for a simple example and it of course gave a stand alone example that works in Node 🤖:
 

import Holster from "@mblaney/holster/src/holster.js"

const holster = Holster()
const user = holster.user()
  
user.create("alice", "password", () => {
    user.auth("alice", "password", () => {
      // Without {nested: true}, this listener only fires when a NEW key is
      // added to "team" - it won't fire again for a later, value-only
      // change to an EXISTING member, because that's a write to the
      // member's own soul (every object-valued child gets its own), not to
      // "team" itself.
      user.get("team").on(members => {
        console.log("team changed:", members)
      }, true, {nested: true})

      // A brand new key - the listener above always sees this.
      user.get("team").next("alice").put({role: "engineer"})

      // A value-only change to an ALREADY-KNOWN key - only reaches the
      // listener because of {nested: true}. Without it, this write would
      // be silently missed.
      setTimeout(() => {
        user.get("team").next("alice").put({role: "lead engineer"})
      }, 100)
    })
})

New Holster release focused mainly on concurrency fixes. I've been working on a new app that runs on Holster which is very front end heavy and running all sorts of async requests, and creating lots of tests for it at the same time. The tests started getting pretty complicated, matching the types of interactions involved, and these all run on top of real holster test instances. The tests started getting a bit flaky as they exhausted their memory limit and swapped out cache files.

I realized Holster has never had any specific concurrency tests, so it was interesting adding them and looking at what sort of race conditions were causing problems. I have to say that Claude is getting really good at methodically working through scenarios without getting lost in the details. I'm impressed with the way it found the issues that have been fixed in this release and the relatively simple changes that were made.

 

I've created another npm package that is complimentary to holster-router, this one is called holster-browser. It's another extraction of core utilities and components from RSStream so I can use them in other projects.

So now if you want to use Holster with Express and React, it's super easy to put them all together and have a working authentication system. Holster is still completely independent, but the other two side projects assume the same endpoints and account structures. I'll probably add to both as I need extra features and not worry so much about publishing the applications that use them.

Lots of updates this week, I gave a presentation on Holster at the June LoFi meetup and published a new npm package. The meetup was at 1am for me, so not the best time to be trying to explain how Holster works! 😅 but it was great to be invited to speak and hear from others doing similar work. I've also been going back and watching previous LoFi episodes which are really interesting and great to see how much work is going on in the local first application space.

The npm package I've released is called holster-router and is an extraction of some of the work that went into RSStream. I've done that so I can re-use the same routes and user creation process in a new Holster based project that I'm about to start. RSStream uses the concept of invite codes to control accounts, but the new project will be more open than this, so the holster-router functions that are shared between these projects will allow for more than one way to create an account. The recent work that went into user storage limits should mean that sign-ups can be open without worrying about server storage. All the other routes for managing password updates and resets will be shared and the new project will add new routes to holster-router as the sign-up model for that is worked out.

A new version of Holster is available that provides a new feature I've wanted to add for a while. This is per-user storage limits. It is off by default, but starting Holster with a new userLimit option set to true will check how much data is stored under a public key before writing updates to disk. This is a server only feature and allows controlling your disk usage if you have limited resources.

One of the things I often thought about when learning about GunDB was how you were meant to run a relay when all data is distributed. Holster is careful to only store data that browsers request, but if you host a public relay I think you should be able to decide how those resources are used. Holster already had options for memory limits, so now it has options for disk usage too. The default is only 1MB, but can also be set in the options when starting. If you set it to zero then unlisted users are effectively blocked from storing data on your server.

Listed users should be added to a simple .user_limit.json file in your application directory, it's just an object containing public keys and their allowed storage in megabytes. Your application can update this file for Holster to read from, which is what RSStream is doing now. Holster then checks it periodically while also tracking user storage in a second file called .user_storage.json. For more information and settings to update when running a relay see the Holster API docs.

Future work for this feature can be done at the application level, for instance you can provide feedback to users about how much storage they're using and provide an endpoint to increase it. Note that this is only backup storage on your server, all their data is still stored locally if you use IndexedDB.

I just updated RSStream to build using Vite instead of the old CRA tools. I didn't realize it would be such a small change otherwise I would've switched over earlier! 🎉 I've also removed the build folder from the github repo, which was suggested I do for Holster to avoid pr's being merged where the build files don't match the source change. The Holster github workflow runs the build before publishing to npm, for RSStream this means an extra step, but it's worth it to make sure the build matches your files.

RSStream has replaced my previous (also homegrown) reader and I'm not finding I want to add any other features at this stage. That means it's time to move on to another project... the next one involves adding new Holster features that I'm looking forward to being able to use, so stay tuned for updates.
 

RSStream is working well now, check out the walk through video below!

It starts with signing into the app, then demonstrates bookmarking, loading older items on scroll-back and search. The crazy thing that I really wanted to show however is that the whole video is offline! 🤯

This is thanks to Holster storing everything in indexedDB and the service worker cache storing images. I'm now also working on storing audio and video for offline playback and allowing cache management too.

 

The last few weeks have felt like Holster is capable enough to now focus on some application development. Previously trying to work on an app that used Holster meant constantly jumping back into the Holster code trying to fix edge cases or find performance improvements.

But I've recently been able to focus on improving RSStream and haven't had to context switch too much. I've just made a small Holster 2.0.1 bug fix release, but besides that it's feeling solid.

The real improvement I've found is not in any Holster change, but in the data model I'm using. A feed reader is possibly the worst starting application to test Holster with, it's too much data arriving all the time, and it's always new! What I realised is that I was treating everything as graph data, starting from feeds all the way down to properties on an item. But unless you're using the ability to query data on the graph, it's a massive performance hit to create references for every piece of graph data. So at some point it's good to stop and decide what can just be stored as strings.

When I was looking into this Claude reported stringifying items would reduce reference look ups from O(N) to O(1)... I didn't believe this, so cleared the session and started again, but same result!  So I looked further and realised that as well as saving all the property references on an item, the whole item itself didn't need a reference created for it as it became a simple property on it's parent node. This is a big deal for a feed reader since the N mentioned here is the number of items per day per feed, so with this simple change the app feels usable now.

A 2.0 release sounds like it should be a big update, and that actually has turned out to be true here, but more by coincidence. Holster uses semver for version numbers, so this 2.0 update is really just for a small breaking API change: a server now needs to be started with either an explicit port number or a websocket server in their config, there's no longer a default port as a fallback.

This was required because some browser testing frameworks were breaking on the const isNode = typeof document === "undefined" check as they also don't define document. The updated check const isNode = typeof process !== "undefined" && process.versions?.node != null, is a better test for server environments but again is also true for testing libraries like jsdom. So the fix isn't to rely on isNode in the tests, but to explicitly require a port when running a server. The real benefit of this isNode change is that Web Workers should now be supported, which is enough of a reason to make the change. But the reason this has become a big update is because I made some radisk changes, and then didn't want to publish those until I was confident it was stable. This meant I just kept adding other patches while watching how the radisk change was performing.

One other API change is that there is no need for a wait parameter on requests, so there's no reason for an options object at all now. This was previously helpful if you wanted to wait longer for certain queries, but this update has dealt with timeout issues internally so there's no longer a reason to specify it. So requests are either fast if you've requested a key previously, or given much more time to check the network if it hasn't been seen before.

My last update mentioned using private browsing mode for testing, but even better than that is going offline and seeing how Holster handles fetching data. The honest answer was not very well, requests would fall through to the network and get stuck in a queue that couldn't be processed. But this also made it quite easy to find what needed improving. When a query is made via the API, the first place to check is in-memory, ie "the graph". There were a few places where it wasn't capturing everything available, so requests would fall through to disk.

Next there were the radisk improvements I mentioned. The change was quite simple, radisk shouldn't ever really time out unless there's an actual problem reading from disk. So timeouts have been mostly removed, file lookup has got faster and file splitting has improved too. It would previously split into the configured size and a smaller file for the overflow, now it aims for half each which is what GunDB tries to do too.

So now if the browser has previously synchronised all available data, no network requests are required. Lastly if something does fall through to a network request when offline then there's much better recovery there too.

Hello! This is one of those "I can't believe I haven't written anything in so long" posts where I try and explain why my last update was in November... well it's enough to say that life can get busy and I'm excited to be able to get back to some Holster development.

My focus at the moment is actually on application development on top of Holster, but that has just found all sorts of things that have needed fixing. They are mostly to do with timing, either timeouts from slow networks or stale cache data arriving before other updates on a callback. The easiest way to test these issues is with a private browsing window that forces a network fetch, so that has become part of my testing routine.

The latest release fixes a lot of bugs, some of which could be considered new features... for instance it was always possible to listen to a path on the graph that doesn't exist, but the listener now waits for that path to be converted to a node and attaches the original listener to that. That simplifies listeners from an application point of view as they now don't need to check that condition. There's also more retrying of underlying wire requests which mean applications don't need to check if Holster is returning something useful.

Besides that there's now some nice data consistency fixes as in-memory data was available but not being returned before a lookup was being done, and store was missing some error callbacks that mean better data checking is now happening.

These last few fixes were found by Claude, which I was using for development last year, but as a lot of people are noticing AI is getting better at code changes. It's only been a few months but it previously wouldn't find these sorts of problems so it's definitely getting more useful.

So long per property signatures, we hardly knew ye! ...I ran the per property signature update on my test server and it blew up my CPU usage with all the extra processing it was doing.

We couldn't go back to node based signatures because they're too hard to verify, so this meant coming up with a new solution. I realised we didn't need to sign content at all, we're just trying to verify ownership of the update, so signing the timestamp associated with the the update is enough.

This is great because generally you're putting a bunch of properties onto a node at once, so they all have the same timestamp, but they only need one signature. If you do update individual properties on a node then it will get it's own timestamp and an updated signature with it.

This has meant my test server is back to running normally, update is available in Holster 1.0.25.

I've just published Holster 1.0.24 which is quite a major change, as signatures are now added per property rather than per node. Before making this change I was trying to debug a signature validation issue on a node, but there was too much data being signed to know what could've gone wrong.

This led to simplifying signatures to each property, which actually fixed a whole bunch of other problems in Holster. Signing a node means you always have to fetch the full node to update it even though you might only be updating one property. There were also timing issues around node updates that aren't a problem any more.

There were a few other recent releases too, including adding Docker support. See the updated README for how to start the server with either Docker or Node.

The last few weeks have seen a lot of development activity, first in Holster itself then in applications using it. This has been driven by the Free Association project, which is now capable of using Holster instead of Gun. The start of the migration process required some further updates to Holster, which is now at version 1.0.21 and feeling pretty stable!

The rest of the work was done in switching API's in the application itself. Holster's API is not only slightly different to Gun, but also conceptually different. The biggest difference is that Gun streams data whereas Holster resolves all references before returning. This doesn't mean much from a performance perspective, but it does significantly change your application code. I've done a few of these migrations now so I already had the notes from the first one, which I've used again successfully since. I'm sharing them here as hopefully they're general enough for other migrations to benefit from.

On the server I've been developing with both Node and Bun, the first step in Node is:

npm uninstall gun
npm i @mblaney/holster

My previous Gun initialization code was:

import Gun from "gun"
import "gun/lib/then.js"
import "gun/sea.js"

const gun = Gun({
  web: express().listen(8765),
  multicast: false,
  axe: false,
})

This has been replaced by:

import Holster from "@mblaney/holster/src/holster.js"

const holster = Holster()

Holster is a module so the path to the file is required. If you're not using a build tool you can switch src to build in the path above. The only option I left out of both examples above was secure: true, which means only data under user accounts will be saved (no public data).

Here is the browser code I previously used for Gun (also not including secure here):

import Gun from "gun"
require("gun/lib/radix.js")
require("gun/lib/radisk.js")
require("gun/lib/store.js")
require("gun/lib/rindexed.js")
require("gun/lib/then.js")
require("gun/sea")

const gun = Gun({
  // Assume Gun is directly available on localhost, otherwise reverse proxy.
  peers: [
    window.location.hostname === "localhost"
      ? "http://localhost:8765/gun"
      : window.location.origin + "/gun",
  ],
  axe: false,
  localStorage: false,
  store: window.RindexedDB(),
})

And here is the browser code for Holster. Note that it doesn't rewrite the scheme for WebSockets like Gun, so you need to specify it or use the default:

import Holster from "@mblaney/holster/src/holster.js"

// If on localhost assume Holster is directly available and use the default
// settings, otherwise assume a secure connection is required.
let peers
if (window.location.hostname !== "localhost") {
  peers = ["wss://" + window.location.hostname]
}

const holster = Holster({peers: peers, indexedDB: true})

The rest of these notes are for the API's in general, so apply to both browser and server.  First is error handling, where Gun returns an object with an err property Holster returns just an error. This means a lot of code changes from ack.err to just err.

The SEA functions work pretty much the same, except Gun uses encryption keys under user._.sea, Holster simplifies this to rely on the user.is object. They can also be used on their own with import SEA from "@mblaney/holster/src/sea.js"

I used user.get(key).once(callback) a lot when working with Gun. In Holster, that pattern is now just user.get(key, callback). I would also previously use user.get(key), and pass that around as a reference to use later. Holster removes the context that gets created once a callback is provided, so that isn't possible and you instead need to write the full query each time.

As mentioned above, the big difference is between streaming data and resolving nodes. So when using Gun you could do: get("items").map().once((item, key) => callback) to get a stream of updates. In Holster it's just get("items", items => callback). So to get the equivalent of Gun's map() you would do: Object.entries(items).forEach(([key, item]) => {callback(item, key)}) on the returned data if you wanted to use the same callback.

Also get(key).map().once(callback) subscribes to updates in Gun, you would need to use get(key).on(callback) instead in Holster for updates.

Gun has get(key).then() which is nice for awaiting data, but it's just a convenience function which can be done in Holster with:  await new Promise(res => {get(key, res)})

The biggest change in terms of lines of code has generally been modifying get().get() chains from Gun, to get().next() in Holster. One more thing to watch out for is calling put() in a loop, it's better to use an actual for loop rather than forEach and awaiting each put() so they don't make updates at the same time, especially to the same node.

This is another bug fix and performance update, message queuing felt a little slow so I've bumped up queue processing from 4 messages per second to 10, we will see how that goes.

The other fixes are more data sync related than timing, the first is that listeners can subscribe to nodes, but the underlying data can change from a property to a reference. A bit of an edge case but this led to listeners not resolving full objects, which is now fixed.

Next was that updates that didn't change the in-memory graph weren't going out on the wire, so other devices on the network weren't alerted. There's also some better checks for when a websocket isn't available messages won't be dequeued until they are actually sent.

The last update was all about improvements that made Holster more efficient on the server. I'm glad to say those changes, plus a couple of recent updates, mean server processing is looking really good now. There are no more CPU spikes for the same processing load, which means Holster is happy running on small severs when given a reasonable amount of work to do.

The big change that allows this to work efficiently is that Holster now stores all radix trees in memory. This allows for much faster gets and puts less pressure on the radisk implementation by no longer requiring parsing from disk as often. This does mean you need to configure your application to provide Holster with enough memory. There is a new memoryLimit option which has a default of 500MB, which is fine for small amounts of data. If you plan to store more data you will need to increase this, but Holster also logs how much memory it's currently using so you will know when it's an issue. When it hits the limit it resets the cache and goes back to files on disk, so it does continue working but this process will probably repeat itself.

The other change required to keep servers running well was that clients needed to throttle their message sending. This also helps browsers not spike a user's CPU and memory because it's making requests that the server couldn't keep up with anyway. This led to some changes that resulted in orderly message queuing and processing, but introduced some timing issues now that have now been fixed in 1.0.17.

To check for these timing issues there's now a whole set of new system tests. I've continued using Claude to write these tests and help with debugging the timing issues which is working really well.

My project to test Holster started hitting 100% CPU the other day during peak processing times, so have been doing some investigation. It's not a big server, but Holster shouldn't need the resources it was using and this was a great opportunity to identify what needed improving.

This has led to a few performance improvement releases this week. The first was that the caching radisk was doing was only for the most recent file. This doesn't do very much when keys come in randomly for different files, and each key caches a new file. Holster is currently not designed to hold heaps of data, so it now just caches all files, which was a big improvement as reading from disk is expensive.

Next was improving the client throttling to use an actual processing queue. It previously tried to throttle with timeouts but this just caused messages to build up later. The advantage of a processing queue is that you can also return an error if the queue gets too long, because there is probably something wrong with your Holster queries if this happens (ie requesting too much data.)

Lastly the Holster website got an update! It now has a nice little visualization of connected clients.

 

I'm honestly surprised to be saying this but I've been working on Holster recently using claude.ai. I've thrown away a bunch of attempts, but have recently found a way to develop using small incremental improvements.

This means I've managed to get through the TODO list I had written up. The first task was limiting graph size which was done in 1.0.8. The other tasks were to do with wire performance, which I thought would take much longer. Browsers now store data based on what they've requested previously, either via get or put. Messages are still always passed through the network so that synchronisation is possible.

The other wire performance required is that I noticed when a browser hadn't connected for a while that it would pull data as quickly as possible. If it has a lot to catch up on then this overloads both the browser and server. There is now some nice throttling added in 1.0.9 that means browsers will sync eventually but the server is not overloaded.

I haven't seen any problems with writing data to disk, but I noticed that Holster is creating only single character file names. Looking into this I found that it's due to the default file size being 1 megabyte, so by the time the initial file reaches that size it has generally already created a radix tree with a single character at the top. That doesn't happen in the unit tests because there's a lot less data so the radix trees don't split out as much.

The reason why the radix tree is important is that Radisk only splits files at the start of a node. It does this so it doesn't have to read multiple files to return the node, which would be quite complicated. And because all the radix trees start with a single character, Radisk will only find single character file names to write to.

This works fine but I realised this limited the number of files it will create. The names depend on the available characters for node ids, which are alphanumeric characters plus ~ for user data and ! for the initial file. That means only 64 files in total! So after 64 megabytes of data it will just evenly distribute data amongst the existing files.

I did find a small issue in the fact that there are so few file names to choose between, it's possible that the new file selected to write to could match the current file name, which would overwrite data. I haven't seen this happen but pushed a fix to avoid it in 1.0.7.

Going forward there's a few ways to deal with this. Firstly "64 megabytes is enough for everyone" sounds ok. And if it's not enough the file size will grow but still work. Second is that writing data requires listing files, so there is a cost to having more files anyway. That means keeping the current implementation and adding a process to remove "least recently used" data which will keep things fast. This is my preference and would be interesting to see what moving data out to "offline" or "cold" storage looks like. Coincidentally this came up in the gun chat today with some interest in making it work.

The third way of dealing with this is to look at the Radisk code again. Holster added the top level node check whereas GunDB reads from multiple files. There is probably a middle way that will work that just needs some brave soul to investigate. 

Holster has been working well this week, my test application has so far stored over 4MB on disk and file splitting looks good now.

I was asked in gun chat about a public relay for Holster, so decided to set one up at holster.haza.website. I have also been playing with github.com/1j01/simple-console and wanted to use it to allow people try out the API, so the website provides both!

I don't want to provide permanent storage for client applications, as I don't believe it's a good model to rely on free services. So at this stage data is removed from the server each day. Saying that it will probably sync back while browsers are open, since that's how it's meant to work 🙂. Hopefully in the future we will add better support for distributed storage and relays coming and going won't be a problem.

My last update finished with "time to throw lots of data at it again and see how it holds up!", sadly it didn't hold up for more than a day. 😣

It's pretty easy to see that data was still not being written to disk properly, so first step was to write some unit tests to replicate the issue. That wasn't too hard either, and just required having a slightly complicated data structure already on disk when the limit is hit. That way the radix tree has enough nodes and properties to make it complicated for the radisk code to try and split across files. Holster now has a test file, split.test.js just for this problem.

So back to the radisk code to work out what was going on. The issue I first noticed was just an edge case, but I then realised I could keep adding edge cases to test and break and fix... so it pretty quickly felt like the existing file splitting code was too brittle.

As I was testing I also noticed that radisk was doing all the work to encode the data for writing to disk, but if it hit the file size limit it would halve the size and start again, throwing away the existing data. I decided to try keeping the existing encoded data and write that to disk first, and deal with the extra data after that. This means that the first file stays at the file size limit and smaller files are created after, but the final code change was so much simpler I decided to stick with it. Plus it could handle every test case I could think of 🎉. Yes I am going to jinx myself for writing this.

I didn't want to write anything until I saw this working in a real application, and am glad to see that it is. In the process of testing it I found some unrelated timing issues so release is now up to 1.0.6.

This week has seen a couple of bug fix releases to Holster, the first was just dealing with connection issues to websockets and also indexedDB in the browser. Holster will now wait for both to be ready and will retry any requests. I previously had some waits in application code to deal with this, but it feels better for Holster to handle it. Now you can just make requests without worrying if the browser is ready.

The second fix was more serious because I noticed I had some missing data in my application using Holster. I tracked it down to how files are split on disk once they reach the configured size limit. I had issues with this in the beginning but didn't look at the existing radisk code too closely. This time I went through the whole flow and found better ways to handle splits, so time to throw lots of data at it again and see how it holds up! 

Another small bug fix release, this one was more of a problem in application code using Holster, but investigating what went wrong there led to some improvements.

The app I'm working on requires a server account and if one doesn't exist when it starts it would create one in Holster. The problem is the user lookup timeout was too short, so it created a second account! This was not helpful since it provides a public key to the browser to look up account information... and now it's split across multiple accounts which isn't going to work.

The answer was to remove the account creation code from the app entirely. This meant I needed a new way to set up the server account in the first place, which led to a nice use of the Node REPL to configure it manually:
 

// Or switch to "@mblaney/holster/src/holster.js" in your own project.
const {default: Holster} = await import("./src/holster.js")
const holster = Holster()
const user = holster.user()

I've left out the actual commands, but from there you can use holster and user to query the API. Besides fixing my problem it also provides a nice way to interact with your data from the command line.

I had one annoying bug that I hadn't got to the bottom of before yesterday's 1.0 release. I even added a bunch of unit tests to make sure Holster was working as I expected. The previous versions of the unit tests looked like this:
 

test("get unknown keys in for loop callbacks null", (t, done) => {
  for (let i = 0; i < 5; i++) {
    holster.get("unknown" + i, data => {
      assert.equal(data, null)
      if (i === 4) done()
    })
  }
})

The idea is that you should be able to make as many requests as you like and Holster will manage the context of each request and map it to the right callback. Can you see the problem with the test though? The loop will finish before the callbacks are called, so the condition will match and done() will be called regardless of if the callbacks were each called correctly. In fact only one was being called but the tests still passed.

They now look like this:
 

test("get unknown keys in for loop callbacks null", (t, done) => {
  let count = 0
  for (let i = 0; i < 5; i++) {
    holster.get("unknown" + i, data => {
      assert.equal(data, null)
      count++
    })
  }
  setTimeout(() => {
    if (count === 5) done()
  }, 200)
})

Which makes sure each was called correctly. Holster wasn't able to pass this version of the test because the context id was being dropped. The id is passed to the API as a parameter that I thought would form part of the closure, but it seems the callback function itself is created before that can happen. The answer was to pass the id to a function that then returns a new callback, which then appears to be called correctly.

This was enough of a problem in the app I'm working on for a 1.0.1 bug fix release.

Wow I can't believe it has been over a month since I first tried switching to Holster in an existing application. It's just what it needed though, as it brought up all sorts of edge cases and timing issues that I hadn't thought of. The end result is that Holster now feels usable! This is a 1.0 release because multiple breaking API changes were required.

The biggest change is to the user API, which I mentioned needed fixing in the last update. It actually needed a complete redesign so that calling holster.user() returns a usable API and can also be used to fetch user data on other accounts. It now does this by taking an optional array of two values when making a query, so if you log in and try user.get("key") you will get your own data, and user.get(["other-account", "key"]) will get data on another account and not interfere with your current log in status.

The other API change is to holster.on(), which now takes four parameters. Previously it was only given a lex object and a callback, but I found a lot of situations where I wanted to add a listener and get the current copy of the data at the same time. So the third parameter is a boolean to get the data, and the fourth parameter is the same options object passed to get() to update the wait time if required.

Another API change is that user.recall() previously required a flag to specify if credentials are stored in local storage or session storage, now it just tries both so no flag is required.

As well as this there are a lot of updates that make it feel like a solid 1.0 release.

Verification is now done properly on user nodes, previously it would sign an update which didn't always contain all properties on a node. Now it will fetch the node first if required to make sure it signs everything. Signing and verification also need to be performed on the exact same object, so it sorts the keys to guarantee the same output.

When I started the port I noticed GunDB was doing a lot more wire sends than Holster, I now see how that helps to keep data in sync. It also means the in-memory graph is always up to date, so get requests now don't need to go to disk unless there's been a restart.

This felt like a lot of debugging to get this release out, but looking at the changes being committed it's far from a complete overhaul to how it was originally written! Hopefully this means it's close to being reliable. Next goal is to push the update to rsstream.app and see how it performs over the next few weeks.

I'm still in the process of converting an app from GunDB to Holster, and haven't hit too many hurdles so far. Enough changes are required to Holster for another release though, so I'm publishing version 0.3.0 as there's a small API change.

The main change required at this point is fixing user queries, I previously added holster.user(public-key) as a way to query other accounts, but it overwrites the context of a logged in user so I think I've fixed that.

I also updated the per-query options object that allows you to wait longer for data to be returned if required. It was available in the wire spec API but wasn't exposed to the Holster API, so get() and next() now both have an optional fourth parameter after the callback.

The last change is to support projects using webpack, which didn't like some Node imports in Holster. They should be ignored anyway so adding /*webpackIgnore: true*/ to the imports seems to work for that.

The Holster options config now support some more options from GunDB, including providing a peer as a string, or as an array, or as {peers: ["url", "etc"]} along with other options in an object. This also meant updating Holster's wire.js to support multiple WebSocket clients. I also added the server and port options from GunDB, all of which are required for real applications.

As I started converting existing code over to Holster I realised that set() would be nice to have. I decided to add it via put() rather than adding a new function, by using an extra parameter that converts the data you want to put to {"random-key": data}. That makes it a pretty simple change, since all updates are added to the set.

There is no map() in Holster to fetch the values in the set, but I'm finding that to be ok. More on how to convert code from GunDB to Holster later.

I've also added publishing new releases from github to npm, which required releasing the first version of Holster so v0.2.0 is now available!

I've just added an initial version of Lex to Holster. It can only query properties on nodes at the moment, while I was working on it I had hopes that queries could be chained to do more interesting things, but that's going to be a lot of work! 😅

So for now lex needs to be added along with a callback for the data, not in a context chain. That allows queries like:

const nested = {
  key: "nested value",
  other: "other value"
}

holster.get("nested").put(nested, err => {
  holster.get("nested", {".": {"*": "k"}}, console.log)
  // logs: {key: "nested value"}
})

I think I'm also just ready to try using Holster from a real application and continue working on it as required from there. With that goal in mind I've also published Holster to npm! https://www.npmjs.com/package/@mblaney/holster

Time to switch gears and try building something with it.

Very excited about this update, Holster now has a working User API which means it's getting very close to being usable for real applications! Some obligatory code:

const user = holster.user()

user.create("alice", "password", console.log)
user.auth("alice", "password", console.log)
console.log(user.is.username) // "alice"

user.get("hello").put("world!", console.log)
user.get("hello", console.log) // "world!"

This has been a few weeks in the making, even with the GunDB code looking in pretty good shape to be used in Holster. I followed the same sign and verify model, but while looking at how to implement it I could really only see that GunDB was verifying data that started with a public key. There are some comments in the code about improving this, with the issue being that nodes often don't start with a public key but instead have their own id which is linked to a public key via a rel.

To get around this problem I decided just to add the user's public key to all nodes. Then if an update is made to existing data, the current public key needs to match what is provided in the update. If there's no existing node then the soul of the data needs a rel on the parent node, but that would need to check the current public key too, so it's not possible to associate random nodes with another user.

Other than that the GunDB code was a great starting point for adding users to Holster. I've kept the same model for storing system data under ~@username and ~publickey. I've also added secure mode as an option so that only user space can be written to. I think I got lucky that the existing context chain in Holster worked well with adding a user API, it really only required changing the root node in the graph.

The wire API changes were a bit more work, as it now fetches all nodes before updating them to check their public keys. This isn't so bad if the graph is already in memory. This might also just be the price of distributed security, if there really is such a thing!

I would really like to switch RSStream over to using Holster, but when I was working on that application I would've liked to have Lex working with user data, but had some trouble with queries in GunDB. So I think I will take a look at adding that next.

Another big update to Holster, I've just added the SEA implementation from GunDB with relatively few modifications. A lot of the environment checks were no longer necessary as Node now supports the Web Crypto API.

Also thanks to Joey, Holster now runs natively in the browser and SEA works exactly the same. There's still a fair way to go to introduce a Holster.user API, but the SEA primitives all work and have lots of tests to show it.

The tests pretty much copy and paste into the browser console to show that they're working there too, all it requires is a JavaScript file with:

import SEA from "../src/sea.js"
globalThis.SEA = SEA

And then you should be able to check the basic functionality. Not quite sure yet how this will integrate with user data in Holster, but very happy with it as a first step.

Lexi in the GunDB chat suggested renaming then() to next() to avoid confusion with the async API's. That makes sense to me so I've pushed the change and updated the API docs.

To update examples from previous posts, that means Holster queries now look like:

holster.get("hello").next("world!").on(console.log)

// Which will get called by this update.
holster.get("hello").next("world!").put("update", console.log)

Now all I need to use Holster in a real application is to add SEA from GunDB. No idea how that's going to happen... wish me luck!

Another item on the TODO list ticked off, IndexedDB is now working in the browser by using Holster({indexedDB: true}). Thankfully the GunDB code was pretty easy to re-use for this! I've kept it opt-in to use browser storage, as Holster currently works fine with server only storage and I'd like to keep both working.

I also missed an update to the examples folder I could've made earlier. It previously didn't try to run both Holster API updates at the same time due to the shared context, but now that's working it's fine to make the two updates occur at random times similar to wire.js. The main difference now being that index.js doesn't set state values on properties, so the value that is fetched at the end is whichever update happened last.

I finally finished adding event listeners to the Holster API, and decided to stick with GunDB's idea of on and off functions.

It took me longer than I expected because on callbacks are long-lived, so the context chain I was using needed to stick around somehow. I also realised that I couldn't do much else once there was an event listener because there was only one context. Or if there was another request the on callback would lose what it was referring to when called.

This meant that the API need to store each context while it was being used, and only remove them when a callback had finished. This took a few attempts, as holster.get().get() previously made sense, but having another request start a new holster.get() meant the two requests couldn't be distinguished from each other and would overwrite the single context available. The solution ended up being that each holster.get() starts a new context, and to differentiate new requests I've added then() to provide the chaining API.

Now Holster can do queries such as:

holster.get("hello").then("world!").on(console.log)

// Which will get called by this update.
holster.get("hello").then("world!").put("update", console.log)

I figured that was enough API changes to add a new page to the Github Wiki, so I've also added a new API page with more details.

Another step on the roadmap has been completed and removed! Holster will now resolve all data in and out. It previously just returned an object containing rels, now it resolves the ids and replaces the rel in the returned data with the full object. Luckily the code to do this was quite similar to the way ids are resolved when putting nodes, so both holster.get() and holster.put() should work with any arbitrarily nested data.

This change makes the underlying data completely opaque when using the Holster API, so data out is the same as data in:

holster.get("hello", console.log)

{
  "world!": {
    "things": "I want to add",
    "which": {
      "can": "also have nested objects"
    }
  }
}

Another interesting thing you can now do with Holster is see all your data. This is possible because of the above change, but you also need to know the top level keys to get. You can do that using the wire API to get the root node:

holster.wire.get({"#": "root"}, msg => console.log(Object.keys(msg.put.root)))

// Or use the above to display everything on disk
holster.wire.get({"#": "root"}, async msg => {
  // wire API returns meta data.
  delete msg.put.root._
  for (const key of Object.keys(msg.put.root)) {
    const value = await new Promise(resolve => holster.get(key, resolve))
    console.log(key, value)
  }
})

A few caveats, the wire API can be used to create top level nodes too, so this method will only show everything created with the Holster API. The Promise is required above because holster.get() controls the context until the callback is called, so this forces the loop to wait until the previous get has finished.

I've added a roadmap to the Github Wiki for Holster, which includes a list of TODO's, one of which I was already able to remove. 🎉

Holster is able to set keys to null, but this previously just set rels to null and left nodes untouched. Removing nodes is quite difficult as the wire spec just wants to update existing values. Holster will now fetch the node and also set all keys to null, so it will at least scrub existing data when required.

Removing the nodes themselves would be great, but needs some more thought for how to add that to the API.

Yesterday's example was meant to be:
 

holster.get("hello").get("world!").put({
  things: "I want to add",
  which: {
    can: "also have nested objects",
  },
}, console.log)

Which I decided to test before adding to the post, and of course it didn't work! I then realised I hadn't tested chained gets at all, so I modified the example to something that worked and then looked at the chained get problem today.

I've just pushed a fix for it, which thankfully also cleared up some edge cases in the code around resolving keys, so it now works and the code is nicer too.

I've just pushed a pretty big update to Holster, it now has an API that sits on top of the wire spec. This allows you to run nice queries similar to the original Gun API, such as:
 

holster.get("hello").put({
  things: "I want to add",
  which: {
    can: "also have nested objects",
  },
}, console.log)

It then resolves the query and puts objects on the graph, resolving nested objects as it goes, making extra calls to the wire API as it needs to. The webpack build exposes Holster as a single library, so the previous wire API is now available via holster.wire. I've also added a root node to the graph, so it's also possible to put non-object properties at the root level:

holster.get("key").put("value", console.log)

Check out the tests for more examples! There's also a lot more tests including using mock-socket to add more test coverage to what is now wire.js.

test("write and read a soul relation with state is ok", (t, done) => {
  radisk("key", [{"#": "soul"}, 1234])
  setTimeout(() => {
    assert.deepEqual(puts, {
      "!": '\x1F+0\x1F#\x1F"key\x1F=\x1F#soul\x031234\x1F\n',
    })
    done()
  }, 10)
})

I noticed some typos after my first commit to Holster and wondered how they didn't get picked up. A lot of the current code is copied over from Porting Gun, so I'm still working out how it all works and was genuinely curious why it hadn't caused errors.

The typo was in a utility function that checks if an object is a soul relation (fixed version):

// Check if an object is a soul relation, ie {'#': 'UUID'}
const rel = {
  is: value => {
    if (value && value["#"] && !value._ && obj.is(value)) {
      let o = {}
      obj.map(value, map_soul, o)
      if (o.id) return o.id
    }

    return false
  },
  // Convert a soul into a relation and return it.
  ify: soul => obj.put({}, "#", soul),
}

But why hadn't it caused any errors? Turns out it was called in Radisk.encode(), but only if it was passed an object, and it wasn't. Radisk is called by store, but it was using JSON.stringify to store a value and it's state as an array, so Radisk was only receiving strings! So now besides the original typo, I'm wondering why I can't just store the array and not use JSON in store.js. That led to updating the file format to include state, which explains the test at the start of this post!

What needs work

Announcing Holster

What does it mean for an application using a distributed database to be federated? Gun already knows how to distribute data efficiently, so federation here is not about data, but about identity.
 
If you've read this far you know that account codes are the main reference to an account in RSStream, and they need to be stored on the host account. In a federated system there is more than one host account, because there is more than one application or at least different providers of the same application. Each host will treat the others as authoritative for their relevant application data under their host accounts. This means hosts need to share the same data structure for their user data, and account codes need to be unique across the federation.

How do you create unique invites codes amongst multiple hosts? This part turns out to be relatively easy. When creating new codes you just need to query the other hosts for any duplicates. You provide the new codes you've generated and if all hosts return that no duplicates were found, it's fine to add the new codes as available for use in your application. They just need to do the same when they're generating new codes.
 

async function checkCodes(newCodes) {
  const codes = Object.keys(await user.get("accounts").then())
  for (let i = 0; i < newCodes.length; i++) {
    if (codes.includes(newCodes[i])) return false
    if (inviteCodes.has(newCodes[i])) return false
  }
  return true
}

In the future this should make it possible to log into other applications that share federated hosts. To make this happen each application just needs to listen for account changes in Gun under each of the host accounts and store the account details alongside their own user data in their host account.

It will also make it possible to switch hosts, and there's an added benefit that only the new host needs to know about moving accounts. That means there's no problems for users if their existing host disappears from the federation.

If an account switches hosts and a password reset is later required, the new host can't decrypt the user's email address to help with account recovery. This means there would need to be a process of re-verifying the account. The current host domain is stored on the user's account so that other hosts can check if this is required, or they could point them back to the original host if they haven't been verified yet.

Public data in Gun can be read by anyone. They may not have a full copy of the database, but if someone knows your public key they can request everything under your account. This is by design and makes account migration much easier if someone needs to reset their password.

Private data in Gun can also be read by anyone, but the difference is that it's encrypted. It should only be possible for users who you've shared data with to decrypt your messages. RSStream doesn't create much encrypted user data yet, but it does support a mechanism for encrypted data recovery.

The recovery method built into the application is to listen for public key changes on accounts you already follow. This is possible because account codes don't change. If someone you know needs a password reset and creates a new public key, you know that they've lost effectively all the encrypted data on their account. What you're able to do as a contact is re-share your copy of any data you previously encrypted for them. This is done automatically because RSStream knows what data was previously shared and now has their new public key with which to encrypt the same data. If all users do this they will effectively restore all previous conversations.

Here's what the code looks like for that:

    const updateAccount = (account, accountCode) => {
      if (!account) return

      let check = {
        pub: account.pub,
        alias: account.alias,
        name: account.name,
        ref: account.ref,
        host: account.host,
      }
      // Check this account against the users list of contacts.
      const contacts = user.get("public").get("contacts")
      contacts.once(async c => {
        if (c) delete c._
        let found = false
        for (const contactCode of Object.keys(c ?? {})) {
          if (contactCode !== accountCode) continue

          found = true
          const contact = await contacts.get(contactCode).then()
          if (contact.pub === check.pub) break

          // If the public key has changed for this contact then store their
          // new account details and re-share encrypted data with them to
          // help restore their account.
          contacts.get(contactCode).put(check, ack => {
            if (ack.err) console.error(ack.err)
          })
          gun
            .user(contact.pub)
            .get("epub")
            .once(oldEPub => {
              if (!oldEPub) {
                console.error("User not found for old public key")
                return
              }

              gun
                .user(check.pub)
                .get("epub")
                .once(epub => {
                  if (!epub) {
                    console.error("User not found for new public key")
                    return
                  }

                  const shared = user.get("shared").get(contactCode)
                  shared.once(async s => {
                    if (!s) return

                    delete s._
                    const oldSecret = await Gun.SEA.secret(oldEPub, user._.sea)
                    const secret = await Gun.SEA.secret(epub, user._.sea)
                    for (const key of Object.keys(s)) {
                      if (!key) continue

                      const oldEnc = await shared.get(key).then()
                      let data = await Gun.SEA.decrypt(oldEnc, oldSecret)
                      let enc = await Gun.SEA.encrypt(data, secret)
                      shared.get(key).put(enc, ack => {
                        if (ack.err) console.error(ack.err)
                      })
                    }
                  })
                })
            })
          break
        }
        // Add the new contact if we referred them.
        if (!found && account.ref === code) {
          contacts.get(accountCode).put(check, ack => {
            if (ack.err) console.error(ack.err)
          })
        }
      })
    }
    // Listen for account changes to add new contacts, update existing contacts.
    gun.user(pub).get("accounts").map().on(updateAccount)

If you've locked yourself out of an account, there's no way to fix the state of a distributed database that would allow you to log in again. Luckily data recovery is still possible and starts by providing your email address when you create an account.

Email provides a way to verify ownership of your account code. Any verification method that is external to Gun would be fine, but email is the easiest to set up. Your email address was associated with your account code when your account was created. I shared some server code in an earlier post that stores your account data, but the actual code looks more like this:
 

app.post("/claim-invite-code", async (req, res) => {
  const encEmail = await Gun.SEA.encrypt(req.body.email, user._.sea)
  const data = {
    pub: req.body.pub,
    alias: req.body.alias,
    name: req.body.alias,
    email: encEmail,
    validate: encValidate,
  }
  user.get("accounts").get(code).put(data)
  validateEmail(req.body.alias, req.body.email, code, validate)
})

The first line in the function shows that your email address is encrypted when stored in Gun, it can only be accessed via the host account on the server. The validation code also needs to be encrypted, otherwise we couldn't prove that the validation request came from an email that was sent to you, since anyone would be able to create the link from data available in Gun.

Once your email is validated, if you ever need to reset your password then RSStream can send you a reset code. The reset code is stored under your account code in the host's account data, which also has your previous public key. Now when you follow the password reset link in an email, a new account will be created in Gun but the application has enough information to associate the old and new accounts. There is public data associated with your account that is not encrypted, that can also be copied over just by knowing your old and new public keys. What about private data though, is that lost along with your old account? Next we need to look at encrypted user data.

Anyone who's worked on websites that allow accounts to be created has received this request for help. Many times. Unfortunately there's just no way to fix this problem in a distributed database. Accounts are literally public/private key pairs and the only way you can prove you have ownership over them is to provide a password that matches the correct decryption algorithm. In a traditional, server based web application you can hash the password provided by a user, compare it to a value stored in your database and provide them with a session token to use your application. In a distributed database, if you don't know the password that matches your key pair then you're locked out of your account.

I wasn't interested in working on a project where this was a blocker though, as it's too much of a requirement to put on user's that they can never reset a password. The answer was to come up with a workaround that looks like a traditional password reset to the user. What I'm using in RSStream works surprisingly well and I plan to use the same pattern in other projects that use Gun.

If a user forgets their password, you can't just replace the password hash in your database to fix the problem. You actually need to create a new account for them in Gun and migrate their data across. However now you have more problems. How do you maintain an identity when switching accounts? And how do you associate the old account with the new one?

The user can pick a username and password, but they will need to change both if a user has locked themselves out of their account. The process RSStream uses is to store their chosen username in the host's account data and then add a numerical suffix to it automatically if a password reset is required. The login screen hides this process by looping through the range of possible usernames and suffix, along with the new password until a match is found. (This is similar to what Gun does anyway, as usernames aren't guaranteed to be unique.)

As I mentioned previously, your account code is the main reference to your account in RSStream. Your username can change, but it's stored against this code which also has a reference to your previous public key. This allows data to be migrated when you next log in. Unfortunately, this really isn't enough for secure account migration, which is why we need to discuss email next time.

No I'm not talking about the UI toolkit, I mean how do you start an application using a distributed database that runs in the browser? In the last post I mentioned relying on a host account, which can be queried in Gun via it's public key. One of my first additions to RSStream was a /host-public-key endpoint. I wanted to be able to start from just the static build files and have the startup process take care of itself, so the first thing the browser does on load is check if it has the host account's public key in local storage, otherwise it will fetch it.
 

if (!pub) {
  fetch(`${window.location.origin}/host-public-key`)
    .then(res => res.text())
    .then(key => setPub(key))
  return
}

setHost(gun.user(pub))
localStorage.setItem("pub", pub)

Once it has this, it knows how to find all the data associated with the account. The next step to a useful piece of software is user accounts. Since Gun is distributed, it's possible for anyone to create an account. This is because it's all done in the browser, there's no server required to verify your credentials. But this is not overly useful if users can't discover each other, or if you want some sort of control over the sign up process. RSStream deals with this by using invite codes. The sign up process requires a code, which is stored alongside your public key in the host account's data after your user account is created in Gun.

// The server logs into the host user account on start up:
const user = gun.user()
user.auth(alias, pass, auth)

app.post("/claim-invite-code", async (req, res) => {
  const data = {
    pub: req.body.pub,
    alias: req.body.alias,
    name: req.body.alias,
  }
  user.get("accounts").get(code).put(data)
})

So the invite code starts as a sign up requirement and becomes a reference to your account in the host's data. The sign up form allows you to choose a username and password like it's a normal process, but as you will see, the code you're given becomes the main reference point for your account. From here on, once an account has been created from an invite code I will refer to this value as the account code.

Hello and welcome! This blog will be documenting some new projects built around a distributed database called GunDB. The idea with these projects will be that they're local and offline first, and they keep a copy of everything they need to function in the browser. Gun ticks a lot of the boxes required to enable this type of functionality. There's other projects that offer similar features, but I'm going to skip the pros and cons of what else is available and jump into working with Gun.

The first project I've been working on is called RSStream. It's a feed reader that stores everything in Gun, but does the feed processing using SimplePie and Dobrado, which are other projects I've worked on. Those services run on the same server and push data to a Node app that has access to Gun.

Before looking at how to store data in Gun, there's a bit of a mindset change required when working with a distributed database. Your first question should be, if everything is distributed how can I protect my data? Gun solves this problem by providing user accounts that can encrypt, sign and verify data. This makes it possible to store data under an account that can only be written when the account owner is logged in.

RSStream uses this feature to create a host account on the server, which provides a central location for application data. Wow first post and we're already talking about centralising data in a distributed database? Well we actually want two things. Data should be distributed as widely as required, to support users. But control should also be available to the administrators of an application when running a hosted service.

With this in mind, creating a host account means we now have a reliable place to store data that can't be modified, even though that data is distributed to browsers and other servers. We've reached the first step to having a usable piece of software! Stayed tuned for more updates.