-
Notifications
You must be signed in to change notification settings - Fork 21.6k
crypto/kzg4844: pull in the C and Go libs for KZG cryptography #27155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+8,890
−39
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
89a3aba
cryto/kzg4844: pull in the C and Go libs for KZG cryptography
karalabe 2b0c35c
go.mod: pull in the KZG libraries
karalabe 97de0cb
crypto/kzg4844: add basic becnhmarks for ballpark numbers
karalabe 8ac7f3f
cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
karalabe 5e1c523
cmd/utils, crypto/kzg4844: run library init on startup
karalabe 560d702
crypto/kzg4844: make linter happy
karalabe 9446750
crypto/kzg4844: push missing file
karalabe 8bf9845
crypto/kzg4844: fully disable CKZG but leave in the sources
karalabe d5fdc86
build, crypto/kzg4844, internal: link CKZG by default and with portab…
karalabe 8170d10
crypto/kzg4844: drop verifying the trusted setup in gokzg
karalabe 4005f78
internal/build: yolo until it works?
karalabe c46ef8b
cmd/utils: make flag description friendlier
karalabe a79f1f2
crypto/ckzg: no need for double availability check
karalabe 6de8078
build: tiny flag cleanup nitpick
karalabe File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| // Copyright 2023 The go-ethereum Authors | ||
| // This file is part of the go-ethereum library. | ||
| // | ||
| // The go-ethereum library is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU Lesser General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // The go-ethereum library is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU Lesser General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU Lesser General Public License | ||
| // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| // Package kzg4844 implements the KZG crypto for EIP-4844. | ||
| package kzg4844 | ||
|
|
||
| import ( | ||
| "embed" | ||
| "errors" | ||
| "sync/atomic" | ||
| ) | ||
|
|
||
| //go:embed trusted_setup.json | ||
| var content embed.FS | ||
|
|
||
| // Blob represents a 4844 data blob. | ||
| type Blob [131072]byte | ||
|
|
||
| // Commitment is a serialized commitment to a polynomial. | ||
| type Commitment [48]byte | ||
|
|
||
| // Proof is a serialized commitment to the quotient polynomial. | ||
| type Proof [48]byte | ||
|
|
||
| // Point is a BLS field element. | ||
| type Point [32]byte | ||
|
|
||
| // Claim is a claimed evaluation value in a specific point. | ||
| type Claim [32]byte | ||
|
|
||
| // useCKZG controls whether the cryptography should use the Go or C backend. | ||
| var useCKZG atomic.Bool | ||
|
|
||
| // UseCKZG can be called to switch the default Go implementation of KZG to the C | ||
| // library if fo some reason the user wishes to do so (e.g. consensus bug in one | ||
| // or the other). | ||
| func UseCKZG(use bool) error { | ||
| if use && !ckzgAvailable { | ||
| return errors.New("CKZG unavailable on your platform") | ||
| } | ||
| useCKZG.Store(use) | ||
|
|
||
| // Initializing the library can take 2-4 seconds - and can potentially crash | ||
| // on CKZG and non-ADX CPUs - so might as well so it now and don't wait until | ||
| // a crpyto operation is actually needed live. | ||
| if use { | ||
| ckzgIniter.Do(ckzgInit) | ||
| } else { | ||
| gokzgIniter.Do(gokzgInit) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // BlobToCommitment creates a small commitment out of a data blob. | ||
| func BlobToCommitment(blob Blob) (Commitment, error) { | ||
| if useCKZG.Load() { | ||
| return ckzgBlobToCommitment(blob) | ||
| } | ||
| return gokzgBlobToCommitment(blob) | ||
| } | ||
|
|
||
| // ComputeProof computes the KZG proof at the given point for the polynomial | ||
| // represented by the blob. | ||
| func ComputeProof(blob Blob, point Point) (Proof, Claim, error) { | ||
| if useCKZG.Load() { | ||
| return ckzgComputeProof(blob, point) | ||
| } | ||
| return gokzgComputeProof(blob, point) | ||
| } | ||
|
|
||
| // VerifyProof verifies the KZG proof that the polynomial represented by the blob | ||
| // evaluated at the given point is the claimed value. | ||
| func VerifyProof(commitment Commitment, point Point, claim Claim, proof Proof) error { | ||
| if useCKZG.Load() { | ||
| return ckzgVerifyProof(commitment, point, claim, proof) | ||
| } | ||
| return gokzgVerifyProof(commitment, point, claim, proof) | ||
| } | ||
|
|
||
| // ComputeBlobProof returns the KZG proof that is used to verify the blob against | ||
| // the commitment. | ||
| // | ||
| // This method does not verify that the commitment is correct with respect to blob. | ||
| func ComputeBlobProof(blob Blob, commitment Commitment) (Proof, error) { | ||
| if useCKZG.Load() { | ||
| return ckzgComputeBlobProof(blob, commitment) | ||
| } | ||
| return gokzgComputeBlobProof(blob, commitment) | ||
| } | ||
|
|
||
| // VerifyBlobProof verifies that the blob data corresponds to the provided commitment. | ||
| func VerifyBlobProof(blob Blob, commitment Commitment, proof Proof) error { | ||
| if useCKZG.Load() { | ||
| return ckzgVerifyBlobProof(blob, commitment, proof) | ||
| } | ||
| return gokzgVerifyBlobProof(blob, commitment, proof) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| // Copyright 2023 The go-ethereum Authors | ||
| // This file is part of the go-ethereum library. | ||
| // | ||
| // The go-ethereum library is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU Lesser General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // The go-ethereum library is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU Lesser General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU Lesser General Public License | ||
| // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| //go:build ckzg && !nacl && !js && cgo && !gofuzz | ||
|
|
||
| package kzg4844 | ||
|
|
||
| import ( | ||
| "encoding/json" | ||
| "errors" | ||
| "sync" | ||
|
|
||
| gokzg4844 "github.com/crate-crypto/go-kzg-4844" | ||
| ckzg4844 "github.com/ethereum/c-kzg-4844/bindings/go" | ||
| "github.com/ethereum/go-ethereum/common/hexutil" | ||
| ) | ||
|
|
||
| // ckzgAvailable signals whether the library was compiled into Geth. | ||
| const ckzgAvailable = true | ||
|
|
||
| // ckzgIniter ensures that we initialize the KZG library once before using it. | ||
| var ckzgIniter sync.Once | ||
|
|
||
| // ckzgInit initializes the KZG library with the provided trusted setup. | ||
| func ckzgInit() { | ||
| config, err := content.ReadFile("trusted_setup.json") | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
| params := new(gokzg4844.JSONTrustedSetup) | ||
| if err = json.Unmarshal(config, params); err != nil { | ||
| panic(err) | ||
| } | ||
| if err = gokzg4844.CheckTrustedSetupIsWellFormed(params); err != nil { | ||
| panic(err) | ||
| } | ||
| g1s := make([]byte, len(params.SetupG1)*(len(params.SetupG1[0])-2)/2) | ||
| for i, g1 := range params.SetupG1 { | ||
| copy(g1s[i*(len(g1)-2)/2:], hexutil.MustDecode(g1)) | ||
| } | ||
| g2s := make([]byte, len(params.SetupG2)*(len(params.SetupG2[0])-2)/2) | ||
| for i, g2 := range params.SetupG2 { | ||
| copy(g2s[i*(len(g2)-2)/2:], hexutil.MustDecode(g2)) | ||
| } | ||
| if err = ckzg4844.LoadTrustedSetup(g1s, g2s); err != nil { | ||
| panic(err) | ||
| } | ||
| } | ||
|
|
||
| // ckzgBlobToCommitment creates a small commitment out of a data blob. | ||
| func ckzgBlobToCommitment(blob Blob) (Commitment, error) { | ||
| ckzgIniter.Do(ckzgInit) | ||
|
|
||
| commitment, err := ckzg4844.BlobToKZGCommitment((ckzg4844.Blob)(blob)) | ||
| if err != nil { | ||
| return Commitment{}, err | ||
| } | ||
| return (Commitment)(commitment), nil | ||
| } | ||
|
|
||
| // ckzgComputeProof computes the KZG proof at the given point for the polynomial | ||
| // represented by the blob. | ||
| func ckzgComputeProof(blob Blob, point Point) (Proof, Claim, error) { | ||
| proof, claim, err := ckzg4844.ComputeKZGProof((ckzg4844.Blob)(blob), (ckzg4844.Bytes32)(point)) | ||
| if err != nil { | ||
| return Proof{}, Claim{}, err | ||
| } | ||
| return (Proof)(proof), (Claim)(claim), nil | ||
| } | ||
|
|
||
| // ckzgVerifyProof verifies the KZG proof that the polynomial represented by the blob | ||
| // evaluated at the given point is the claimed value. | ||
| func ckzgVerifyProof(commitment Commitment, point Point, claim Claim, proof Proof) error { | ||
| valid, err := ckzg4844.VerifyKZGProof((ckzg4844.Bytes48)(commitment), (ckzg4844.Bytes32)(point), (ckzg4844.Bytes32)(claim), (ckzg4844.Bytes48)(proof)) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if !valid { | ||
| return errors.New("invalid proof") | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // ckzgComputeBlobProof returns the KZG proof that is used to verify the blob against | ||
| // the commitment. | ||
| // | ||
| // This method does not verify that the commitment is correct with respect to blob. | ||
| func ckzgComputeBlobProof(blob Blob, commitment Commitment) (Proof, error) { | ||
| proof, err := ckzg4844.ComputeBlobKZGProof((ckzg4844.Blob)(blob), (ckzg4844.Bytes48)(commitment)) | ||
| if err != nil { | ||
| return Proof{}, err | ||
| } | ||
| return (Proof)(proof), nil | ||
| } | ||
|
|
||
| // ckzgVerifyBlobProof verifies that the blob data corresponds to the provided commitment. | ||
| func ckzgVerifyBlobProof(blob Blob, commitment Commitment, proof Proof) error { | ||
| valid, err := ckzg4844.VerifyBlobKZGProof((ckzg4844.Blob)(blob), (ckzg4844.Bytes48)(commitment), (ckzg4844.Bytes48)(proof)) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if !valid { | ||
| return errors.New("invalid proof") | ||
| } | ||
| return nil | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| // Copyright 2023 The go-ethereum Authors | ||
| // This file is part of the go-ethereum library. | ||
| // | ||
| // The go-ethereum library is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU Lesser General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // The go-ethereum library is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU Lesser General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU Lesser General Public License | ||
| // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| //go:build !ckzg || nacl || js || !cgo || gofuzz | ||
|
|
||
| package kzg4844 | ||
|
|
||
| import "sync" | ||
|
|
||
| // ckzgAvailable signals whether the library was compiled into Geth. | ||
| const ckzgAvailable = false | ||
|
|
||
| // ckzgIniter ensures that we initialize the KZG library once before using it. | ||
| var ckzgIniter sync.Once | ||
|
|
||
| // ckzgInit initializes the KZG library with the provided trusted setup. | ||
| func ckzgInit() { | ||
| panic("unsupported platform") | ||
| } | ||
|
|
||
| // ckzgBlobToCommitment creates a small commitment out of a data blob. | ||
| func ckzgBlobToCommitment(blob Blob) (Commitment, error) { | ||
| panic("unsupported platform") | ||
| } | ||
|
|
||
| // ckzgComputeProof computes the KZG proof at the given point for the polynomial | ||
| // represented by the blob. | ||
| func ckzgComputeProof(blob Blob, point Point) (Proof, Claim, error) { | ||
| panic("unsupported platform") | ||
| } | ||
|
|
||
| // ckzgVerifyProof verifies the KZG proof that the polynomial represented by the blob | ||
| // evaluated at the given point is the claimed value. | ||
| func ckzgVerifyProof(commitment Commitment, point Point, claim Claim, proof Proof) error { | ||
| panic("unsupported platform") | ||
| } | ||
|
|
||
| // ckzgComputeBlobProof returns the KZG proof that is used to verify the blob against | ||
| // the commitment. | ||
| // | ||
| // This method does not verify that the commitment is correct with respect to blob. | ||
| func ckzgComputeBlobProof(blob Blob, commitment Commitment) (Proof, error) { | ||
| panic("unsupported platform") | ||
| } | ||
|
|
||
| // ckzgVerifyBlobProof verifies that the blob data corresponds to the provided commitment. | ||
| func ckzgVerifyBlobProof(blob Blob, commitment Commitment, proof Proof) error { | ||
| panic("unsupported platform") | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will still SIGILL when run on CPUs without ADX and using go-kzg (
--crypto.kzg=gokzg) because blst checks this when the package is imported, not when a method is called. I suspect this is not how you thought it worked.For example, when blst isn't configured to be portable, this will cause a SIGILL on older hardware:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Damn, thanks for the heads up
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have a new attempt at CKGZ integration:
--tags=ckzgbuild tag. This way the library is still in our codebase but does not build by default, so anyone running go build or go install blindly will not get surprises with that pooplib (blst)ci.gobuilder to add both theckzgbuild tag, as well as theCGO_FLAGS="-O -D__BLST_PORTABLE__"CGO env var. This will make sure that anything built with our build scripts (docker, debian, ppt, brew, make, etc) will have CKZG available to allow failsafe switching, but in portable mode so no surprises.The only people losing out of CKZG is people who use their own build pipeline via go build/go install and people linking against Geth as a dependency. For these two sets of users CKGZ will only be available if they specify the build tags on their own end and with or without the damn cgo flag
All our pre-built binaries will have the portable CKZG included so we should be fairly safe with what we distribute but also fairly flexible. We'd need to update the README with the potential memo that anyone using their own build system needs to adapt. ¯_(ツ)_/¯
Best I could come up with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. Hopefully blst adds runtime detection soon (or allows the community to). By the way, I reached out to someone on the team privately and they have seen yours & Martin's comments. They said: