Test driven development

test("write and read a soul relation with state is ok", (t, done) => {
  radisk("key", [{"#": "soul"}, 1234])
  setTimeout(() => {
    assert.deepEqual(puts, {
      "!": '\x1F+0\x1F#\x1F"key\x1F=\x1F#soul\x031234\x1F\n',
    })
    done()
  }, 10)
})

I noticed some typos after my first commit to Holster and wondered how they didn't get picked up. A lot of the current code is copied over from Porting Gun, so I'm still working out how it all works and was genuinely curious why it hadn't caused errors.

The typo was in a utility function that checks if an object is a soul relation (fixed version):

// Check if an object is a soul relation, ie {'#': 'UUID'}
const rel = {
  is: value => {
    if (value && value["#"] && !value._ && obj.is(value)) {
      let o = {}
      obj.map(value, map_soul, o)
      if (o.id) return o.id
    }

    return false
  },
  // Convert a soul into a relation and return it.
  ify: soul => obj.put({}, "#", soul),
}

But why hadn't it caused any errors? Turns out it was called in Radisk.encode(), but only if it was passed an object, and it wasn't. Radisk is called by store, but it was using JSON.stringify to store a value and it's state as an array, so Radisk was only receiving strings! So now besides the original typo, I'm wondering why I can't just store the array and not use JSON in store.js. That led to updating the file format to include state, which explains the test at the start of this post!

Announcing Holster

I hope that what I've written previously shows I'm excited and positive about applications written using Gun. Unfortunately though, I think we need to take a couple of steps back before we can create production ready distributed applications.

What needs work

My feed processing software is adding new feed items to Gun every hour. Unfortunately there's an intermittent problem where data isn't written to disk and there are no acknowledgements from Gun.

I've avoided using Lex because it doesn't work with userspace. Lex is a great idea, but it's only useful to me if it also works with user data.

Yson parsing doesn't work. The parser cannot accurately create a JavaScript object from a given string, which is required by radisk to split files. It currently corrupts the database and so I haven't used it for rsstream.

I've already fixed other bugs in Gun and know that the current state of the code base is very difficult to work on. This is not to take anything away from the idea of Gun and the fantastic work Mark and the team have done to bring it to reality.

Announcing Holster

Gun feels too dangerous to use right now, so I need something more contained... I'm starting with something very simple, based on github.com/gundb/port which Mark helpfully walks through in gun.eco/docs/Porting-GUN.

Holster is a fork of that code base along with updates from github.com/amark/gun that were required to fill some gaps and get some initial tests passing.

I've avoided problems with the unbuild tool by targeting just the node.js version and using webpack to build the browser version. I don't think Holster will end up looking much like Gun, but I'm hoping to keep pulling in features that make sense. As Mark has mentioned, the chaining API makes the current code base very complicated so will probably leave that out. The main priority will be that the current automated tests keep passing, and are added to, as development continues.

I've tried to make the code base easier to read and follow so that development is easier to pick up. If you're interested in working on this with me please reach out! The initial version is available at github.com/mblaney/holster

Federation

What does it mean for an application using a distributed database to be federated? Gun already knows how to distribute data efficiently, so federation here is not about data, but about identity.

If you've read this far you know that account codes are the main reference to an account in RSStream, and they need to be stored on the host account. In a federated system there is more than one host account, because there is more than one application or at least different providers of the same application. Each host will treat the others as authoritative for their relevant application data under their host accounts. This means hosts need to share the same data structure for their user data, and account codes need to be unique across the federation.

How do you create unique invites codes amongst multiple hosts? This part turns out to be relatively easy. When creating new codes you just need to query the other hosts for any duplicates. You provide the new codes you've generated and if all hosts return that no duplicates were found, it's fine to add the new codes as available for use in your application. They just need to do the same when they're generating new codes.

async function checkCodes(newCodes) {
  const codes = Object.keys(await user.get("accounts").then())
  for (let i = 0; i < newCodes.length; i++) {
    if (codes.includes(newCodes[i])) return false
    if (inviteCodes.has(newCodes[i])) return false
  }
  return true
}

In the future this should make it possible to log into other applications that share federated hosts. To make this happen each application just needs to listen for account changes in Gun under each of the host accounts and store the account details alongside their own user data in their host account.

It will also make it possible to switch hosts, and there's an added benefit that only the new host needs to know about moving accounts. That means there's no problems for users if their existing host disappears from the federation.

If an account switches hosts and a password reset is later required, the new host can't decrypt the user's email address to help with account recovery. This means there would need to be a process of re-verifying the account. The current host domain is stored on the user's account so that other hosts can check if this is required, or they could point them back to the original host if they haven't been verified yet.

Encrypted user data

Public data in Gun can be read by anyone. They may not have a full copy of the database, but if someone knows your public key they can request everything under your account. This is by design and makes account migration much easier if someone needs to reset their password.

Private data in Gun can also be read by anyone, but the difference is that it's encrypted. It should only be possible for users who you've shared data with to decrypt your messages. RSStream doesn't create much encrypted user data yet, but it does support a mechanism for encrypted data recovery.

The recovery method built into the application is to listen for public key changes on accounts you already follow. This is possible because account codes don't change. If someone you know needs a password reset and creates a new public key, you know that they've lost effectively all the encrypted data on their account. What you're able to do as a contact is re-share your copy of any data you previously encrypted for them. This is done automatically because RSStream knows what data was previously shared and now has their new public key with which to encrypt the same data. If all users do this they will effectively restore all previous conversations.

Here's what the code looks like for that:

    const updateAccount = (account, accountCode) => {
      if (!account) return

      let check = {
        pub: account.pub,
        alias: account.alias,
        name: account.name,
        ref: account.ref,
        host: account.host,
      }
      // Check this account against the users list of contacts.
      const contacts = user.get("public").get("contacts")
      contacts.once(async c => {
        if (c) delete c._
        let found = false
        for (const contactCode of Object.keys(c ?? {})) {
          if (contactCode !== accountCode) continue

          found = true
          const contact = await contacts.get(contactCode).then()
          if (contact.pub === check.pub) break

          // If the public key has changed for this contact then store their
          // new account details and re-share encrypted data with them to
          // help restore their account.
          contacts.get(contactCode).put(check, ack => {
            if (ack.err) console.error(ack.err)
          })
          gun
            .user(contact.pub)
            .get("epub")
            .once(oldEPub => {
              if (!oldEPub) {
                console.error("User not found for old public key")
                return
              }

              gun
                .user(check.pub)
                .get("epub")
                .once(epub => {
                  if (!epub) {
                    console.error("User not found for new public key")
                    return
                  }

                  const shared = user.get("shared").get(contactCode)
                  shared.once(async s => {
                    if (!s) return

                    delete s._
                    const oldSecret = await Gun.SEA.secret(oldEPub, user._.sea)
                    const secret = await Gun.SEA.secret(epub, user._.sea)
                    for (const key of Object.keys(s)) {
                      if (!key) continue

                      const oldEnc = await shared.get(key).then()
                      let data = await Gun.SEA.decrypt(oldEnc, oldSecret)
                      let enc = await Gun.SEA.encrypt(data, secret)
                      shared.get(key).put(enc, ack => {
                        if (ack.err) console.error(ack.err)
                      })
                    }
                  })
                })
            })
          break
        }
        // Add the new contact if we referred them.
        if (!found && account.ref === code) {
          contacts.get(accountCode).put(check, ack => {
            if (ack.err) console.error(ack.err)
          })
        }
      })
    }
    // Listen for account changes to add new contacts, update existing contacts.
    gun.user(pub).get("accounts").map().on(updateAccount)

Why email?

If you've locked yourself out of an account, there's no way to fix the state of a distributed database that would allow you to log in again. Luckily data recovery is still possible and starts by providing your email address when you create an account.

Email provides a way to verify ownership of your account code. Any verification method that is external to Gun would be fine, but email is the easiest to set up. Your email address was associated with your account code when your account was created. I shared some server code in an earlier post that stores your account data, but the actual code looks more like this:

app.post("/claim-invite-code", async (req, res) => {
  const encEmail = await Gun.SEA.encrypt(req.body.email, user._.sea)
  const data = {
    pub: req.body.pub,
    alias: req.body.alias,
    name: req.body.alias,
    email: encEmail,
    validate: encValidate,
  }
  user.get("accounts").get(code).put(data)
  validateEmail(req.body.alias, req.body.email, code, validate)
})

The first line in the function shows that your email address is encrypted when stored in Gun, it can only be accessed via the host account on the server. The validation code also needs to be encrypted, otherwise we couldn't prove that the validation request came from an email that was sent to you, since anyone would be able to create the link from data available in Gun.

Once your email is validated, if you ever need to reset your password then RSStream can send you a reset code. The reset code is stored under your account code in the host's account data, which also has your previous public key. Now when you follow the password reset link in an email, a new account will be created in Gun but the application has enough information to associate the old and new accounts. There is public data associated with your account that is not encrypted, that can also be copied over just by knowing your old and new public keys. What about private data though, is that lost along with your old account? Next we need to look at encrypted user data.