Replace JOSE with our own `tokens` domain and specialise tokens for Sigchain, Notifications, Identities and Sessions

[CMCDragonkai]

Specification

With the recent crypto update in #446, we have found that the JOSE library is not longer compatible, and not even with the webcrypto polyfill.

Since JOSE is easier to replace than x509, we have decided to remove JOSE entirely.

This affects several domains:

1. sigchain - claims here are general JWS structures, with multiple signatures
2. identities - claims are being put on identity providers, but this was developed before we understood JOSE and JWS, these can now use specialised tokens
3. notifications - using JWS to send "signed" messages, the encryption of these notification messages are reliant on P2P E2E encryption, but we don't store the messages as encrypted, they are stored unencrypted, but subsequently disk-encrypted, however we retain the signature of the messages as they are useful
4. sessions - using JWS MAC authenticated tokens as a short-lived session token for authentication (remember you always exchange long-lived credentials for short-lived credentials)
5. Our tokens should be based on the Ed25519 RFC for JWS: https://www.rfc-editor.org/rfc/rfc8037#appendix-A.4

This is the relationship of JOSE to what we are doing:

Additional context

• Updating Crypto to use WebCrypto API and to replace RSA with ECC #446 (comment) - discussing the incompatibility
• Updating Crypto to use WebCrypto API and to replace RSA with ECC #446 (comment) - notes on JWS
• Updating Crypto to use WebCrypto API and to replace RSA with ECC #446 (comment) - notes on sigchain API renovation
• https://datatracker.ietf.org/doc/html/rfc7515 - JWS spec
• https://www.rfc-editor.org/rfc/rfc8037 - JWS with ed25519 spec
• https://sodium-friends.github.io/docs/docs/sha - in case we need to replace the SHA algo

Tasks

1. Renaming claims to tokens, we will have a generic "Token" structure
2. Specialised tokens like TokenClaim or TokenSession can then be used by various domains like sigchain, identities and notifications and sessions
3. Claims are now just a specific variant of the more general token.
4. We store our domain types in the DB, but we can choose arbitrary "representations" of our tokens based on the JWS spec as compact, flattened or general representations. Although if we have multiple signatures, compact and flattened representations may not really make any sense.
5. The sigchain, notifications, sessions and identities will be going through some renovation of their API in accordance with the latest ways of doing things.
6. At the same time, record all the "reactive" points where property subscription could be useful, so this can be done in a subsequent issue.

[CMCDragonkai]

The hashing of the claims should be using multihash and not just a raw SHA256 hash. We can combine this with our new hashing utilities provided by the keys domain.

[CMCDragonkai]

Ok so we now have:

import * as keysUtils from '../keys/utils';

keysUtils.sha256();
keysUtils.sha512();

These use libsodium to do the hashing. I also have sha256G and sha256I variants for special usecases. Note that sha256G is a "consumer" generator that is pre-primed. That means you can start with next(Buffer.from('hello')) immediately. See https://stackoverflow.com/questions/74095105/explicit-generator-return-type-in-typescript-demands-that-generator-prototype-r/74095446?noredirect=1#comment130829319_74095446 On a side note that I figured that one could potentially derive a Consumer type from Iterator type that focuses purely on consumption and returning a result. I tried this, and while it works, I think some more iterations on it are required for actual generic consumption since there could many kinds of iterators.

Now the next issue is that multiformats now is only used in 2 places:

claims/utils.ts
client/utils/utils.ts

In client/utils/utils.ts, I believe it should instead by using Buffer directly since it's just doing base64 decoding, and we can just use Buffer polyfill even in non-node envrionments.

And so it's left to claims utils that uses it and js-id library which uses it for base encoding. Base encoding doesn't require C for speed, it's plenty fast by itself in JS, and we can continue using multiformats in that respect.

The main thing now is multihashing. Which if we want to re-use our multiformats structure, we can actually re-use our keys functions for that purpose. https://github.com/multiformats/js-multiformats#multihash-hashers

import * as hasher from 'multiformats/hashes/hasher'

const sha256 = hasher.from({
  // As per multiformats table
  // https://github.com/multiformats/multicodec/blob/master/table.csv#L9
  name: 'sha2-256',
  code: 0x12,

  encode: (input) => new Uint8Array(crypto.createHash('sha256').update(input).digest())
})

So a question is should these derivation sit inside the claims or token utilities or should they be put into a more general utilities somewhere? That way it's possible to use them in potentially other places where we may want to do multihashing.

[CMCDragonkai]

I've replaced multiformats usage in src/client/utils/utils.ts with just Buffer. It just used base64 padding encoding which is what Buffer normally does. This reduces some dependency on third party libraries here.

[CMCDragonkai]

For now I'll put the multihashing derivations into src/tokens/utils.ts. This will be the only place where this is used for now other than js-id.

[CMCDragonkai]

The current hashClaim uses:

1. Canonicalized JSON encoding - https://www.rfc-editor.org/rfc/rfc8785
2. Then uses SHA256
3. Then uses hex encoding

Basically something like:

    hashClaim = hex(sha256(canonicalize(jsonClaim)))

However I want to digress about the JWS spec.

JWS already has some form of algorithm agility through the alg header property. For example we could use HS256 or ES256 or EdDSA. These headers dictate the algorithms used for generating and verifying signatures or MAC codes.

In the general format, it may look like this:

{
  payload: base64url(payload),
  signatures: [
    {
      protected: base64url({
        alg: 'EdDSA',
      }),
      header: {
        kid: '<NodeId>'
      },
      signature: base64url(signature)
    },
    {
      protected: base64url({
        alg: 'HS256',
      }),
      header: {
        kid: '<NodeId>'
      },
      signature: base64url(signature)
    }
  ]
}

Unlike JWE there is no shared header, so headers have to be specific to each signature.

This means our hPrev makes the most sense to be in the payload like it is currently.

Now a question is whether we can use alg to also determine agility of the hashing algo for hPrev.

Now that we cannot put it into a shared header, and the fact that we are using multibase encoding for the <NodeId>, then I believe we both multibase and multihash for our hPrev.

So anything that is locked down in the JWS spec will be multibase and/or multihash.

So I propose we use the following:

    hashClaim = multibase(multihash(canonicalize(jsonClaim)))

The selected multibase will just be base64url similar to the rest of the JWS spec. While the multihash can be SHA256 for now.

Another thing I want to touch on is that JWS has both typ and cty which can be used to disambiguate the JWS by providing media types.

The typ is for the complete JWS. The cty is for just the payload. But are optional. Both are meant to be processed by the application.

Both are case insensitive. It is recommended to omit application/ prefix when no other / appears in the value. So JWT should be understood as application/jwt. Or example is application/example. But if it is application/example;part="1/2", it should be understood as application/example;part="1/2".

This is something we can use later, like application/claim vs application/somethingelse to differntiate from sigchain tokens from identity tokens from notification tokens from session tokens. Note we would use something like application/vnd.polykey.sigchain-token+json.

There are some recommendations to use JOSE+JSON when using general or flattened format.

Finally for the alg, in this case we won't need to use a custom alg like we are doing in our JWE usage. But in the future, custom names should be collision resistant:

Collision-Resistant Name
A name in a namespace that enables names to be allocated in a
manner such that they are highly unlikely to collide with other
names. Examples of collision-resistant namespaces include: Domain
Names, Object Identifiers (OIDs) as defined in the ITU-T X.660 and
X.670 Recommendation series, and Universally Unique IDentifiers
(UUIDs) [RFC4122]. When using an administratively delegated
namespace, the definer of a name needs to take reasonable
precautions to ensure they are in control of the portion of the
namespace they use to define the name.

Examples possible algs:

• io.polykey.Polykey.thisalgo - reverse domain name notation https://en.wikipedia.org/wiki/Reverse_domain_name_notation
• 1.2.3.4.5 - OID - we have registered OIDs already
• 123e4567-e89b-12d3-a456-426614174000 - UUID
• ECDH-ES-NaCl - Custom names

[CMCDragonkai]

Ok so I've come up with realisations here.

Consistency of JWS/JWT (token)

All the claims and JWS usage right now are all over the place, and we need to make it more consistent.

Here's an idea.

Let's start with a generic Token type:

/**
 * Token based on JWT specification.
 * All properties are "claims" and they are all optional.
 * The entire POJO is put into the payload for signing.
 */
type Token = {
  iss?: string;
  sub?: string;
  aud?: string | Array<string>;
  exp?: number;
  nbf?: number;
  iat?: number;
  jti?: string;
  [key: string]: any;
};

The above is based on the JWT specification. And specifically it just the payload. This payload is what gets signed. Everything else is just metadata. Although a portion of the metadata is also signed.

Then we can have specialise "tokens" for various usage. I suspect the tokens domain will specifically handle just generic tokens and produce TokenSigned structures.

/**
 * Signed token based on General JWS specification.
 */
type TokenSigned = {
  payload: string;
  signatures: Array<{
    signature: string;
    protected: string;
  }>;
};

In particular we know that the protected header which is base64url encoded of the JSON string will contain something like this:

{
  alg: 'EdDSA' | 'BLAKE2b' | 'HS512256'
}

Atm, sodium-native only exposes the blake2b algorithm and the HMAC-SHA512-256 (truncated 512). Even though libsodium has HS256 and HS512, so for now, we just focus on what sodium-native has.

The BLAKE2b is provided by crypto_generichash. It supports hashing without a key, and also hashing with a key. In the case of signing, I believe it would be useful for hashing with a key, and the key is just a symmetric key that is 32 bytes which is exactly the symmetric key sizes we use.

HS512256 is just what is exposed by crypto_auth. In the future we can get sodium-native to expose the HS256 and HS512 and then our tokens would actually be compatible.

I think since we're going to do this for now, we let's just use BLAKE2b.

In that case... we our multidigest work needs a bit of a tweak. Atm, we don't have blake2b, and I'm not sure which of these https://github.com/multiformats/multicodec/blob/master/table.csv are actually specific to libsodium. The default output size is 32 bytes or 256 bits. Therefore I think it is this one: blake2b-256 and 0xb220.

Subsequently we then need to create "specialised" tokens for all the different usecases we have for them.

Here are some draft tokens I've synthesised from the current situation in the codebase:

type TokenClaimNode = {
  jti: ClaimIdEncoded;
  iat: number;
  iss: NodeIdEncoded;
  sub: NodeIdEncoded;
  nbf: number;
  prev: string | null;
  seq: number;
};

type TokenClaimIdentity {
  jti: ClaimIdEncoded;
  iat: number;
  iss: NodeIdEncoded;
  sub: ProviderIdentityId;
  nbf: number;
  prev: string | null;
  seq: number;
};

type TokenNotification<T> = {
  jti: NotificationIdEncoded;
  iat: number;
  iss: NodeIdEncoded;
  sub: NodeIdEncoded;
  data: T;
};

type TokenSession = {
  iss: NodeIdEncoded;
  sub: NodeIdEncoded;
  iat: number;
  nbf: number;
  exp: number;
};

The old types were originally defined in the claims domain. So TokenClaimNode and TokenClaimIdentity are JWT synthesised versions of Claim and ClaimData. Compare the above to these old (current) types:

type ClaimLinkNode = {
  type: 'node';
  node1: NodeIdEncoded;
  node2: NodeIdEncoded;
};

type ClaimLinkIdentity = {
  type: 'identity';
  node: NodeIdEncoded;
  provider: ProviderId;
  identity: IdentityId;
};

type ClaimData = ClaimLinkNode | ClaimLinkIdentity;

type Claim = {
  payload: {
    hPrev: string | null; // Hash of the previous claim (null if first claim)
    seq: number; // Sequence number of the claim
    data: ClaimData; // Our custom payload data
    iat: number; // Timestamp (initialised at JWS field)
  };
  signatures: Record<NodeIdEncoded, SignatureData>; // Signee node ID -> claim signature
};

I believe the new iteration is far clearer and uses standardised terminology.

Note that ProviderIdentityId is meant to be a new JSON.stringify([ProviderId, IdentityId]) type cause it would make sense to do this for node to digital identity claims.

The jti can then be useful to keep track of an ID for these tokens. In the sigchain, this is ClaimIdEncoded. But in notifications, it would be NotificationIdEncoded.

One important thing about the NotificationIdEncoded, is that currently it takes additional data. However I will flatten this when I get to the notifications types. Instead I'll use an intersection type to take the common notification payload properties and combine it with additional properties specific to particular kinds of notifications. We currently have 3 kinds GestaltInvite, VaultShare and General.

The session token doesn't have a jti at the moment because the session token is not saved in the DB anywhere. It's just handed to the client for persistence.

This is why the generic token interface does not have any required properties.

The tokens domain will expose functionality for signing tokens, encoding them, and decoding them. However each of the specialise tokens will be declared by their respective domains.

The claim tokens are used by both identities and sigchain domain. This is because we're going to double-post the claims from sigchain to identity, and between sigchain to sigchain. This means claims domain may still exist to share this functionality. Some documentation should be written into the claims domain to indicate what this represents. Furthermore claims keyword is pretty overloaded. It might be good to disambiguate this "claims" usage with something else. We used to call these "links". It may be better to change it back to calling them "links". And thus IdentityLink and NodeLink can be clearer for both identities and sigchain.

The session token type will still be in sessions. The notifications token type will still be in notifications.

The tokens/utils should have:

• signWithPrivateKey - uses private key and EdDSA
• signWithKey - uses symmetric key and BLAKE2b
• verifyWithPublicKey
• verifyWithKey

Both functions will produce a TokenSigned structure.

Note that doubly signed claims will be made alot simpler. They will be stored on both sigchains. The iss is always the initiator of the node link.

[CMCDragonkai]

The tokens domain seems to have cross overs with the KeyManager repurposing in #472. Both would have implications for secure computation #385. I imagine further innovations coming there to include not just JWT, but taking ideas from PASETO, biscuits... etc.

[CMCDragonkai]

Ok I've replaced JSON.stringify in the keys utils with canonicalize since that ensures deterministic serialisation. Which is useful when we want to later re-generate hashes.

So now keys utils exposes hashWithKey and its variants. We've chosen BLAKE2b as the symmetric hash to use here. Furthermore, the token utilities ends up with signWithKey and signWithPrivateKey. Which takes a key, token payload, any additional header data and then produces a TokenSignature.

We don't go straight to producing a signed token structure because it can be mutated. Instead one may combine these things together to produce a TokenSigned, but one can also mutate and add additional signatures to the TokenSigned.

It seems that a safe way of doing this is to construct an rich object representing a token, and the ability to add and remove signatures at will, then the ability to produce the serialised representation. That way, one would not just be able to mutate the token payload, or just add random signatures in. It would encapsulate the mechanics of tokens internally.

[CMCDragonkai]

So I've renamed them to be more clear:

  macWithKey,
  macWithKeyG,
  macWithKeyI,
  authWithKey,
  authWithKeyG,
  authWithKeyI,

MAC is clearer than hash. Since we are not just producing a digest, but a MAC which ensures integrity and authenticity.

[CMCDragonkai]

We now have a Token class. This sits in src/Token.ts.

It exposes several methods:

• signWithKey
• signWithPrivateKey
• verifyWithKey
• verifyWithPublicKey
• serialize

During verification, the default behaviour is to iterate over the signatures to verify the token. This means it actually algorithm or signature doesn't match, it just goes to the next one. This may not be efficient if there are lots of signatures on a token. However this is not likely to be the case, so at most O(2) for all our existing usecases. We may need to apply some operational limits when receiving token data either for notifications, sigchain and more.

This Token represents a single token. Now we can imagine, that the "TokenManager" does not exist at this point in time. This is because instead we have many token managers. The Sigchain is one particular token manager, managing specifically identity link and node link tokens. While the NotificationManager manages notification tokens... etc.

So for now there is no "general" token manager. Just like how there is no "general" key manager. Such things would only come into play during secure computation. Both of which would have to be built on top of the vault system to take advantage of high-level secret exchange like git sync #385. There are of course limitations here... especially if the FS paradigm is too limiting for such secrets. But generally high level UIs tend to fit into the FS metaphor. But we could imagine a KeyManager and TokenManager sitting on top of the vault system.

[CMCDragonkai]

So therefore hashClaim is really the domain of Sigchain as it is specifically about hashing the previous link token. We are not using the "claim" keyword here to avoid confusion with "claim" properties that go into the token.

[CMCDragonkai]

JWKs are just a particular variant of a Token payload.

The token payload has some "registered claims", but these are all optional. So a payload is ultimately just any structured data that can be represented by JSON.

The token payload can be signed, which turns it into a JWS.

A token payload can be encrypted, which turns it into a JWE.

A token payload can be signed then encrypted, which means it becomes a JWS, then becomes a JWE. This allows arbitrary nesting of signing and encryption.

The expected "payload" when signing a JWT was meant to be a base64url of the JSON stringification.

When it's a JWE though, we also have different representations for the JWS if we want to use it as a payload like compact, general and flattened. So I'm not sure if it matters which representation you are encrypting. Because JWE only desires that the plaintext is some octet-sequence string.

This could mean that it depends on the cty which tells the end user what the content type of the plain text.

• application/jwt - this can be a compact JWS or compact JWE
• application/jose - could be JWS or JWE in compact representation
• application/jose+json - could be JWS or JWE in JSON representation (general or flattened) - UTF-8 json serialisation

The spec seems to recommend that when nesting a JWT, the cty should be application/jwt or just JWT. I find it strange that application/jose appears to be conflict with application/jwt.