Too many Code duplicates

[fluency03]

Such as:

raw 0x55 - base64urlpad 'U', which is 0x55
bencode 0x63 - base32pad 'c', which is 0x63
dbl-sha2-256 0x56 - base32hex-upper 'V', which is 0x56

And,

multihash 0x31 - base1 '1', which is 0x31
multicodec 0x30 - base2 '0', which is 0x30
dns6 0x37 - base8 '7', which is 0x37

[Stebalien]

See: #59 and the followup #68.

Basically, 'U' happens to encode to 0x55 in ASCII/UTF-8 but 'U', itself, is a symbol. Multibase only really makes sense in a text context where we have a string of character symbols.

Note: after some followup discussions, we realized that these really don't belong in the same table. Technically, bytes are also symbols but I'm not aware of any text encoding that allows for both character symbols and byte symbols. The current setup causes more confusion than it's worth.

[fluency03]

@Stebalien

The terminology "symbol" is really confusing, because it does not belong to any data type in, as far as I know, any programming language. At least we have data type Byte in most of languages.

From the implementation point of view, I still don't understand how to represent a symbol, where the other Codes are having Byte field at the same place.

[fluency03]

According to this implementation https://github.com/multiformats/js-multicodec/blob/master/src/base-table.js, there are only the following bases implemented:

// bases encodings
exports['base1'] = Buffer.from('01', 'hex')
exports['base2'] = Buffer.from('00', 'hex')
exports['base8'] = Buffer.from('07', 'hex')
exports['base10'] = Buffer.from('09', 'hex')

And here, the so called symbols are actually treated as hex in Byte.

Can I do this if I want to implement it in another language?

[fluency03]

Update from #76:

Both js-multicodec and py-multicodec are wrong.

[Stebalien]

The terminology "symbol" is really confusing, because it does not belong to any data type in, as far as I know, any programming language.

Copied from #76 (comment) to keep everything in this thread:

For example, binary is composed of two symbols 0 and 1 (or true and false). Bytes are defined to each be a string of 8 binary symbols but are also, themselves, symbols (there are 256 of them).

Every character is also a symbol. On a computer, these symbols may be encoded into bits/bytes but there are often several ways to encode a single symbol into bits/bytes and the symbol exists apart from these encodings (an '1' on paper is a '1', not 0x31).

When I say symbol, I'm talking about these: https://en.wikipedia.org/wiki/Turing_machine

(I agree this is confusing. It's "technically" correct but I can't think of a better explanation that's still correct.)

[fluency03]

So that means, in an actual implementation, a symbol has to be implemented as a special data structure, maybe called Symbol. Some properties of this data type Symbol could be something like this:

class Symbol (
  val isByte: Bool = ~
  val value: Bytes = ~
)

[Stebalien]

It's probably best to just have two tables:

1. Multicodecs: these use varint byte sequences.
2. Multibases: use text symbols.

Combining them under a single abstraction probably isn't worth it.

For multibase, you'd just use whatever encoding your language supports. For example, the symbol 👍 has one encoding in UTF-8, another in UTF-16, and another in UTF-32. At the end of the day, that doesn't really matter. The important part is whether or not some string starts with the symbol 👍 (regardless of encoding).

[fluency03]

However, a new question is: if we treat so many different things (such as protobuf, md4, murmur3, even ip4, udp and http) as different type of codec, why should we exclude BaseN form the Codec?

[Stebalien]

Those all occur in a binary context. That is, they all answer the question "what does this series of bytes mean". However, mulitbase occurs in a text context. It answers the question "how do I convert this sequence of characters to a sequence of bytes".