I hope that what I've written previously shows I'm excited and positive about applications written using Gun. Unfortunately though, I think we need to take a couple of steps back before we can create production ready distributed applications.
What needs work
My feed processing software is adding new feed items to Gun every hour. Unfortunately there's an intermittent problem where data isn't written to disk and there are no acknowledgements from Gun.
I've avoided using Lex because it doesn't work with userspace. Lex is a great idea, but it's only useful to me if it also works with user data.
Yson parsing doesn't work. The parser cannot accurately create a JavaScript object from a given string, which is required by radisk to split files. It currently corrupts the database and so I haven't used it for rsstream.
I've already fixed other bugs in Gun and know that the current state of the code base is very difficult to work on. This is not to take anything away from the idea of Gun and the fantastic work Mark and the team have done to bring it to reality.
Announcing Holster
Gun feels too dangerous to use right now, so I need something more contained... I'm starting with something very simple, based on github.com/gundb/port which Mark helpfully walks through in gun.eco/docs/Porting-GUN.
Holster is a fork of that code base along with updates from github.com/amark/gun that were required to fill some gaps and get some initial tests passing.
I've avoided problems with the unbuild tool by targeting just the node.js version and using webpack to build the browser version. I don't think Holster will end up looking much like Gun, but I'm hoping to keep pulling in features that make sense. As Mark has mentioned, the chaining API makes the current code base very complicated so will probably leave that out. The main priority will be that the current automated tests keep passing, and are added to, as development continues.
I've tried to make the code base easier to read and follow so that development is easier to pick up. If you're interested in working on this with me please reach out! The initial version is available at github.com/mblaney/holster
Federation
What does it mean for an application using a distributed database to be federated? Gun already knows how to distribute data efficiently, so federation here is not about data, but about identity.
If you've read this far you know that account codes are the main reference to an account in RSStream, and they need to be stored on the host account. In a federated system there is more than one host account, because there is more than one application or at least different providers of the same application. Each host will treat the others as authoritative for their relevant application data under their host accounts. This means hosts need to share the same data structure for their user data, and account codes need to be unique across the federation.
How do you create unique invites codes amongst multiple hosts? This part turns out to be relatively easy. When creating new codes you just need to query the other hosts for any duplicates. You provide the new codes you've generated and if all hosts return that no duplicates were found, it's fine to add the new codes as available for use in your application. They just need to do the same when they're generating new codes.
async function checkCodes(newCodes) {
const codes = Object.keys(await user.get("accounts").then())
for (let i = 0; i < newCodes.length; i++) {
if (codes.includes(newCodes[i])) return false
if (inviteCodes.has(newCodes[i])) return false
}
return true
}
In the future this should make it possible to log into other applications that share federated hosts. To make this happen each application just needs to listen for account changes in Gun under each of the host accounts and store the account details alongside their own user data in their host account.
It will also make it possible to switch hosts, and there's an added benefit that only the new host needs to know about moving accounts. That means there's no problems for users if their existing host disappears from the federation.
If an account switches hosts and a password reset is later required, the new host can't decrypt the user's email address to help with account recovery. This means there would need to be a process of re-verifying the account. The current host domain is stored on the user's account so that other hosts can check if this is required, or they could point them back to the original host if they haven't been verified yet.
Announcing Holster
What needs work
Announcing Holster
Holster is a fork of that code base along with updates from github.com/amark/gun that were required to fill some gaps and get some initial tests passing.
Federation
If you've read this far you know that account codes are the main reference to an account in RSStream, and they need to be stored on the host account. In a federated system there is more than one host account, because there is more than one application or at least different providers of the same application. Each host will treat the others as authoritative for their relevant application data under their host accounts. This means hosts need to share the same data structure for their user data, and account codes need to be unique across the federation.
How do you create unique invites codes amongst multiple hosts? This part turns out to be relatively easy. When creating new codes you just need to query the other hosts for any duplicates. You provide the new codes you've generated and if all hosts return that no duplicates were found, it's fine to add the new codes as available for use in your application. They just need to do the same when they're generating new codes.
async function checkCodes(newCodes) { const codes = Object.keys(await user.get("accounts").then()) for (let i = 0; i < newCodes.length; i++) { if (codes.includes(newCodes[i])) return false if (inviteCodes.has(newCodes[i])) return false } return true }In the future this should make it possible to log into other applications that share federated hosts. To make this happen each application just needs to listen for account changes in Gun under each of the host accounts and store the account details alongside their own user data in their host account.It will also make it possible to switch hosts, and there's an added benefit that only the new host needs to know about moving accounts. That means there's no problems for users if their existing host disappears from the federation.
If an account switches hosts and a password reset is later required, the new host can't decrypt the user's email address to help with account recovery. This means there would need to be a process of re-verifying the account. The current host domain is stored on the user's account so that other hosts can check if this is required, or they could point them back to the original host if they haven't been verified yet.