My last update finished with "time to throw lots of data at it again and see how it holds up!", sadly it didn't hold up for more than a day. 😣
It's pretty easy to see that data was still not being written to disk properly, so first step was to write some unit tests to replicate the issue. That wasn't too hard either, and just required having a slightly complicated data structure already on disk when the limit is hit. That way the radix tree has enough nodes and properties to make it complicated for the radisk code to try and split across files. Holster now has a test file, split.test.js just for this problem.
So back to the radisk code to work out what was going on. The issue I first noticed was just an edge case, but I then realised I could keep adding edge cases to test and break and fix... so it pretty quickly felt like the existing file splitting code was too brittle.
As I was testing I also noticed that radisk was doing all the work to encode the data for writing to disk, but if it hit the file size limit it would halve the size and start again, throwing away the existing data. I decided to try keeping the existing encoded data and write that to disk first, and deal with the extra data after that. This means that the first file stays at the file size limit and smaller files are created after, but the final code change was so much simpler I decided to stick with it. Plus it could handle every test case I could think of 🎉. Yes I am going to jinx myself for writing this.
I didn't want to write anything until I saw this working in a real application, and am glad to see that it is. In the process of testing it I found some unrelated timing issues so release is now up to 1.0.6.
Holster updates
This week has seen a couple of bug fix releases to Holster, the first was just dealing with connection issues to websockets and also indexedDB in the browser. Holster will now wait for both to be ready and will retry any requests. I previously had some waits in application code to deal with this, but it feels better for Holster to handle it. Now you can just make requests without worrying if the browser is ready.
The second fix was more serious because I noticed I had some missing data in my application using Holster. I tracked it down to how files are split on disk once they reach the configured size limit. I had issues with this in the beginning but didn't look at the existing radisk code too closely. This time I went through the whole flow and found better ways to handle splits, so time to throw lots of data at it again and see how it holds up!
Radisk
It's pretty easy to see that data was still not being written to disk properly, so first step was to write some unit tests to replicate the issue. That wasn't too hard either, and just required having a slightly complicated data structure already on disk when the limit is hit. That way the radix tree has enough nodes and properties to make it complicated for the radisk code to try and split across files. Holster now has a test file, split.test.js just for this problem.
So back to the radisk code to work out what was going on. The issue I first noticed was just an edge case, but I then realised I could keep adding edge cases to test and break and fix... so it pretty quickly felt like the existing file splitting code was too brittle.
As I was testing I also noticed that radisk was doing all the work to encode the data for writing to disk, but if it hit the file size limit it would halve the size and start again, throwing away the existing data. I decided to try keeping the existing encoded data and write that to disk first, and deal with the extra data after that. This means that the first file stays at the file size limit and smaller files are created after, but the final code change was so much simpler I decided to stick with it. Plus it could handle every test case I could think of 🎉. Yes I am going to jinx myself for writing this.
I didn't want to write anything until I saw this working in a real application, and am glad to see that it is. In the process of testing it I found some unrelated timing issues so release is now up to 1.0.6.
Holster updates
The second fix was more serious because I noticed I had some missing data in my application using Holster. I tracked it down to how files are split on disk once they reach the configured size limit. I had issues with this in the beginning but didn't look at the existing radisk code too closely. This time I went through the whole flow and found better ways to handle splits, so time to throw lots of data at it again and see how it holds up!