Claude

I'm honestly surprised to be saying this but I've been working on Holster recently using claude.ai. I've thrown away a bunch of attempts, but have recently found a way to develop using small incremental improvements.

This means I've managed to get through the TODO list I had written up. The first task was limiting graph size which was done in 1.0.8. The other tasks were to do with wire performance, which I thought would take much longer. Browsers now store data based on what they've requested previously, either via get or put. Messages are still always passed through the network so that synchronisation is possible.

The other wire performance required is that I noticed when a browser hadn't connected for a while that it would pull data as quickly as possible. If it has a lot to catch up on then this overloads both the browser and server. There is now some nice throttling added in 1.0.9 that means browsers will sync eventually but the server is not overloaded.

Still thinking about Radisk

I haven't seen any problems with writing data to disk, but I noticed that Holster is creating only single character file names. Looking into this I found that it's due to the default file size being 1 megabyte, so by the time the initial file reaches that size it has generally already created a radix tree with a single character at the top. That doesn't happen in the unit tests because there's a lot less data so the radix trees don't split out as much.

The reason why the radix tree is important is that Radisk only splits files at the start of a node. It does this so it doesn't have to read multiple files to return the node, which would be quite complicated. And because all the radix trees start with a single character, Radisk will only find single character file names to write to.

This works fine but I realised this limited the number of files it will create. The names depend on the available characters for node ids, which are alphanumeric characters plus ~ for user data and ! for the initial file. That means only 64 files in total! So after 64 megabytes of data it will just evenly distribute data amongst the existing files.

I did find a small issue in the fact that there are so few file names to choose between, it's possible that the new file selected to write to could match the current file name, which would overwrite data. I haven't seen this happen but pushed a fix to avoid it in 1.0.7.

Going forward there's a few ways to deal with this. Firstly "64 megabytes is enough for everyone" sounds ok. And if it's not enough the file size will grow but still work. Second is that writing data requires listing files, so there is a cost to having more files anyway. That means keeping the current implementation and adding a process to remove "least recently used" data which will keep things fast. This is my preference and would be interesting to see what moving data out to "offline" or "cold" storage looks like. Coincidentally this came up in the gun chat today with some interest in making it work.

The third way of dealing with this is to look at the Radisk code again. Holster added the top level node check whereas GunDB reads from multiple files. There is probably a middle way that will work that just needs some brave soul to investigate.

Holster Haza Website

Holster has been working well this week, my test application has so far stored over 4MB on disk and file splitting looks good now.

I was asked in gun chat about a public relay for Holster, so decided to set one up at holster.haza.website. I have also been playing with github.com/1j01/simple-console and wanted to use it to allow people try out the API, so the website provides both!

I don't want to provide permanent storage for client applications, as I don't believe it's a good model to rely on free services. So at this stage data is removed from the server each day. Saying that it will probably sync back while browsers are open, since that's how it's meant to work 🙂. Hopefully in the future we will add better support for distributed storage and relays coming and going won't be a problem.

Radisk

My last update finished with "time to throw lots of data at it again and see how it holds up!", sadly it didn't hold up for more than a day. 😣

It's pretty easy to see that data was still not being written to disk properly, so first step was to write some unit tests to replicate the issue. That wasn't too hard either, and just required having a slightly complicated data structure already on disk when the limit is hit. That way the radix tree has enough nodes and properties to make it complicated for the radisk code to try and split across files. Holster now has a test file, split.test.js just for this problem.

So back to the radisk code to work out what was going on. The issue I first noticed was just an edge case, but I then realised I could keep adding edge cases to test and break and fix... so it pretty quickly felt like the existing file splitting code was too brittle.

As I was testing I also noticed that radisk was doing all the work to encode the data for writing to disk, but if it hit the file size limit it would halve the size and start again, throwing away the existing data. I decided to try keeping the existing encoded data and write that to disk first, and deal with the extra data after that. This means that the first file stays at the file size limit and smaller files are created after, but the final code change was so much simpler I decided to stick with it. Plus it could handle every test case I could think of 🎉. Yes I am going to jinx myself for writing this.

I didn't want to write anything until I saw this working in a real application, and am glad to see that it is. In the process of testing it I found some unrelated timing issues so release is now up to 1.0.6.

Holster updates

This week has seen a couple of bug fix releases to Holster, the first was just dealing with connection issues to websockets and also indexedDB in the browser. Holster will now wait for both to be ready and will retry any requests. I previously had some waits in application code to deal with this, but it feels better for Holster to handle it. Now you can just make requests without worrying if the browser is ready.

The second fix was more serious because I noticed I had some missing data in my application using Holster. I tracked it down to how files are split on disk once they reach the configured size limit. I had issues with this in the beginning but didn't look at the existing radisk code too closely. This time I went through the whole flow and found better ways to handle splits, so time to throw lots of data at it again and see how it holds up!