Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

murmur3 support #150

Merged
merged 5 commits into from
Oct 28, 2021
Merged

murmur3 support #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
971a35b
murmur3 support
willscott Oct 13, 2021
b2309aa
update to naming consistent with go-multicodec
willscott Oct 20, 2021
728649e
wrap with correct size
willscott Oct 20, 2021
6e4d134
update spec
willscott Oct 20, 2021
397761d
comments added to multicodec
willscott Oct 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 22 additions & 21 deletions core/magic.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,26 @@ var ErrSumNotSupported = errors.New("no such hash registered")

// constants
const (
IDENTITY = 0x00
SHA1 = 0x11
SHA2_224 = 0x1013
SHA2_256 = 0x12
SHA2_384 = 0x20
SHA2_512 = 0x13
SHA2_512_224 = 0x1014
SHA2_512_256 = 0x1015
SHA3_224 = 0x17
SHA3_256 = 0x16
SHA3_384 = 0x15
SHA3_512 = 0x14
KECCAK_224 = 0x1A
KECCAK_256 = 0x1B
KECCAK_384 = 0x1C
KECCAK_512 = 0x1D
BLAKE3 = 0x1E
SHAKE_128 = 0x18
SHAKE_256 = 0x19
MD5 = 0xd5
DBL_SHA2_256 = 0x56
IDENTITY = 0x00
SHA1 = 0x11
SHA2_224 = 0x1013
SHA2_256 = 0x12
SHA2_384 = 0x20
SHA2_512 = 0x13
SHA2_512_224 = 0x1014
SHA2_512_256 = 0x1015
SHA3_224 = 0x17
SHA3_256 = 0x16
SHA3_384 = 0x15
SHA3_512 = 0x14
KECCAK_224 = 0x1A
KECCAK_256 = 0x1B
KECCAK_384 = 0x1C
KECCAK_512 = 0x1D
BLAKE3 = 0x1E
SHAKE_128 = 0x18
SHAKE_256 = 0x19
MURMUR3X64_64 = 0x22
MD5 = 0xd5
DBL_SHA2_256 = 0x56
)
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ require (
github.com/minio/sha256-simd v1.0.0
github.com/mr-tron/base58 v1.2.0
github.com/multiformats/go-varint v0.0.6
github.com/spaolacci/murmur3 v1.1.0
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/sys v0.0.0-20210309074719-68d13333faf2 // indirect
lukechampine.com/blake3 v1.1.6
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
github.com/multiformats/go-varint v0.0.6 h1:gk85QWKxh3TazbLxED/NlDVv8+q+ReFJk7Y2W/KhfNY=
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
Expand Down
10 changes: 5 additions & 5 deletions multihash.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ const (

DBL_SHA2_256 = 0x56

MURMUR3_128 = 0x22
// Deprecated: use MURMUR3_128
MURMUR3 = MURMUR3_128
MURMUR3X64_64 = 0x22
// Deprecated: use MURMUR3X64_64
MURMUR3 = MURMUR3X64_64

SHA2_256_TRUNC254_PADDED = 0x1012
X11 = 0x1100
Expand Down Expand Up @@ -104,7 +104,7 @@ var Names = map[string]uint64{
"sha3-384": SHA3_384,
"sha3-512": SHA3_512,
"dbl-sha2-256": DBL_SHA2_256,
"murmur3-128": MURMUR3_128,
"murmur3-x64-64": MURMUR3X64_64,
"keccak-224": KECCAK_224,
"keccak-256": KECCAK_256,
"keccak-384": KECCAK_384,
Expand All @@ -129,7 +129,7 @@ var Codes = map[uint64]string{
SHA3_384: "sha3-384",
SHA3_512: "sha3-512",
DBL_SHA2_256: "dbl-sha2-256",
MURMUR3_128: "murmur3-128",
MURMUR3X64_64: "murmur3-x64-64",
KECCAK_224: "keccak-224",
KECCAK_256: "keccak-256",
KECCAK_384: "keccak-384",
Expand Down
4 changes: 2 additions & 2 deletions multihash_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ var tCodes = map[uint64]string{
0x16: "sha3-256",
0x17: "sha3-224",
0x56: "dbl-sha2-256",
0x22: "murmur3-128",
0x22: "murmur3-x64-64",
0x1A: "keccak-224",
0x1B: "keccak-256",
0x1C: "keccak-384",
Expand Down Expand Up @@ -51,7 +51,7 @@ var testCases = []TestCase{
{"2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae", 0x12, "sha2-256"},
{"2c26b46b", 0x12, "sha2-256"},
{"2c26b46b68ffc68ff99b453c1d30413413", 0xb240, "blake2b-512"},
{"243ddb9e", 0x22, "murmur3-128"},
{"243ddb9e", 0x22, "murmur3-x64-64"},
{"f00ba4", 0x1b, "keccak-256"},
{"f84e95cb5fbd2038863ab27d3cdeac295ad2d4ab96ad1f4b070c0bf36078ef08", 0x18, "shake-128"},
{"1af97f7818a28edfdfce5ec66dbdc7e871813816d7d585fe1f12475ded5b6502b7723b74e2ee36f2651a10a8eaca72aa9148c3c761aaceac8f6d6cc64381ed39", 0x19, "shake-256"},
Expand Down
1 change: 1 addition & 0 deletions register/all/multihash_all.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,6 @@ package all
import (
_ "github.com/multiformats/go-multihash/register/blake2"
_ "github.com/multiformats/go-multihash/register/blake3"
_ "github.com/multiformats/go-multihash/register/murmur3"
_ "github.com/multiformats/go-multihash/register/sha3"
)
40 changes: 40 additions & 0 deletions register/murmur3/multihash_murmur3.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
This package has no purpose except to perform registration of multihashes.

It is meant to be used as a side-effecting import, e.g.

import (
_ "github.com/multiformats/go-multihash/register/murmur3"
)

This package registers multihashes for murmur3
*/
package murmur3

import (
"hash"

multihash "github.com/multiformats/go-multihash/core"
"github.com/spaolacci/murmur3"
)

func init() {
multihash.Register(multihash.MURMUR3X64_64, func() hash.Hash { return murmur64{murmur3.New64()} })
}

// A wrapper is needed to export the correct size, because murmur3 incorrectly advertises Hash64 as a 128bit hash.
type murmur64 struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe this type name should be as verbose as the constants became, for consistency?

hash.Hash64
}

func (murmur64) BlockSize() int {
return 1
Comment on lines +30 to +31
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks slightly funny. I can't really fathom any harm to it, though.

... but I went on a splunk anyway, and I'd say 16 is probably an accurate number. citation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm trying to stay with what it would report natively: https://github.com/spaolacci/murmur3/blob/539464a789e9b9f01bc857458ffe2c5c1a2ed382/murmur.go#L31

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huh. Remains weird to me that they said it that way, but okay.

}

func (x murmur64) Size() int {
return 8
}

func (x murmur64) Sum(digest []byte) []byte {
return x.Hash64.Sum(digest)
}
2 changes: 1 addition & 1 deletion spec/multicodec
Open in desktop
Submodule multicodec updated 5 files
+11 −0 .github/workflows/validate.yml
+0 −9 .travis.yml
+31 −3 README.md
+485 −442 table.csv
+41 −8 validate.py
2 changes: 1 addition & 1 deletion spec_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func TestSpec(t *testing.T) {

reader := csv.NewReader(file)
reader.LazyQuotes = false
reader.FieldsPerRecord = 4
reader.FieldsPerRecord = 5
reader.TrimLeadingSpace = true

values, err := reader.ReadAll()
Expand Down