Enable blake3 for Bazel builds #565
Conversation
There was a problem hiding this comment.
Reviewable status: 0 of 2 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / ubuntu-22.04, Local / ubuntu-22.04, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), pre-commit-checks, publish-image, ubuntu-20.04, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable (waiting on @adam-singer and @allada)
There was a problem hiding this comment.
Based on bazelbuild/bazel@0a8380b && bazelbuild/remote-apis#235 should we be considering SHA256TREE? Seems like some existing research has been done, at a high level Blake3 might be better for x86/x64 and slower than SHA256 on Arm, SHA256TREE is mentioned in the thread better on both platforms.. Not sure if its worth evaluating which digest to use based on platform, or just keep it simple, pick one for now and revisit later.
Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: 1 of 2 LGTMs obtained, and pending CI: docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04) (waiting on @allada)
There was a problem hiding this comment.
@adam-singer Interesting. Is that used in the same way as the other ones though? If something other than blake3 is faster on arm we could set the digest function to different values for different configurations to get the most out of all CI runners.
cross-ref #566
Reviewable status: 1 of 2 LGTMs obtained, and pending CI: docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04) (waiting on @allada)
There was a problem hiding this comment.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04)
There was a problem hiding this comment.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04)
a discussion (no related file):
verify store does not yet support blake3:
aaronmondal
force-pushed
the
enable-blake3
branch
from
c39ce7f
to
a8465e1
Compare
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
aaronmondal
force-pushed
the
enable-blake3
branch
from
a8465e1
to
1f1a3f0
Compare
There was a problem hiding this comment.
Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04)
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04)
There was a problem hiding this comment.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04)
deployment-examples/docker-compose/local-storage-cas.json line 28 at r2 (raw file):
}, "verify_size": true, "hash_verification_function": "blake3"
Hmmm, lets remove verification store here all together. We use it in testing and by default bazel uses sha256, so it'll make this not work "out-of-the-box" unless the user adds cli flags (and has clients that support blake3).
If we remove verification store it should just work.
aaronmondal
force-pushed
the
enable-blake3
branch
from
1f1a3f0
to
297d85e
Compare
There was a problem hiding this comment.
Reviewable status: 2 of 2 LGTMs obtained, and pending CI: Bazel Dev / ubuntu-22.04, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), publish-image, windows-2022 / stable
deployment-examples/docker-compose/local-storage-cas.json line 28 at r2 (raw file):
Previously, allada (Nathan (Blaise) Bruer) wrote…
Hmmm, lets remove verification store here all together. We use it in testing and by default bazel uses sha256, so it'll make this not work "out-of-the-box" unless the user adds cli flags (and has clients that support blake3).
If we remove verification store it should just work.
Done.
There was a problem hiding this comment.
Reviewed 1 of 1 files at r3.
Reviewable status:complete! 2 of 2 LGTMs obtained
There was a problem hiding this comment.
Reviewed all commit messages.
Reviewable status:
This change is