-
Notifications
You must be signed in to change notification settings - Fork 37.4k
Add minisketch subtree and integrate in build/test #21859
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@sipsorcery Could you help with adding MSVC build config here? |
@sipa will do. |
From ci:
|
@MarcoFalke I believe it is simply msan not being able to reason through the intrinsics (see src/minisketch/src/field/clmul_common_impl.h). |
Attached diff adds a |
Do minisketch's own tests get run when running |
CI would need to run them, though. As this is the only place where sanitizers are run regularly. |
e436b49
to
7142c44
Compare
cb7d61c
to
12fdba1
Compare
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Failure on arm: You might have to modify the |
@theuni If it's an issue, I think non-verify is sufficient. The verify mode mostly adds algorithmic property checks, not things that I'd expect to be very architecture/build dependent. |
It's not much of an issue, it's just kinda hacky to build everything twice, especially if it's unlikely to be as useful as it is running in the upstream tests. I can add it back if there's a need. See here for a current integration branch: https://github.com/theuni/bitcoin/commits/minisketch-split . Not sure if it's easier for you to integrate that into this PR or for me to pull your other changes in on top of mine? I'm happy to do the latter if you'd like. |
writer.Flush(); | ||
} | ||
|
||
void Deserialize(const unsigned char* ptr) override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May be helpful to add a fuzz harness for sketch deserialization. A large number of security bugs come from deserializing untrusted data.
I've put together a branch for this. It's based off current master and includes all the latest changes from upstream minisketch, as well as integrating the changes here with @theuni's build changes. It's available here: https://github.com/fanquake/bitcoin/tree/minisketch_integration. Note that I've included some of the review suggestions, as well as squashed one commit (theuni@2fff4ce) from Corys branch. I've also made some additions to I've opened an issue upstream, bitcoin-core/minisketch#46, as now that we've got minisketch/src/minisketch.cpp:104:13: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
case 2:
^
minisketch/src/minisketch.cpp:104:13: note: insert '[[fallthrough]];' to silence this warning
case 2:
^
[[fallthrough]];
minisketch/src/minisketch.cpp:104:13: note: insert 'break;' to avoid fall-through
case 2:
^
break; |
git-subtree-dir: src/minisketch git-subtree-split: ea986a869eaff3120aa9c0caaf01143ce51471df
AC_DEFINE'd values won't be passed down to minisketch because it does not use bitcoin-config.h. Thus we need a way to know if we should manually add defines for minisketch files.
I've updated my branch (https://github.com/fanquake/bitcoin/tree/minisketch_integration) to be rebased on master, include the upstream minisktech changes, namely bitcoin-core/minisketch#45 & bitcoin-core/minisketch#47 and update our integration accordingly. Had a conflict in the MSVC config files, not 100% sure if how it's been resolved is correct. |
cf4897c
to
238bf6e
Compare
Reviewed @fanquake's branch, and switched to it. |
Guix builds
|
@@ -297,7 +297,7 @@ mkdir -p "$DISTSRC" | |||
${HOST_CXXFLAGS:+CXXFLAGS="${HOST_CXXFLAGS}"} \ | |||
${HOST_LDFLAGS:+LDFLAGS="${HOST_LDFLAGS}"} | |||
|
|||
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool | |||
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool src/minisketch/config.status src/minisketch/libtool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Options used to compile and link:
external signer = yes
multiprocess = no
with libs = yes
with wallet = yes
with sqlite = yes
with bdb = yes
with gui / qt = yes
with qr = yes
with zmq = yes
with test = yes
with bench = no
with upnp = yes
with natpmp = yes
use asm = yes
ebpf tracing = no
sanitizers =
debug enabled = no
gprof enabled = no
werror = no
target os = linux
build os = linux-gnu
CC = x86_64-linux-gnu-gcc
CFLAGS = -pthread -pipe -O2 -O2 -g -ffile-prefix-map=/bitcoin=.
CPPFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -I/bitcoin/depends/x86_64-linux-gnu/include/ -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS -DPROVIDE_FUZZ_MAIN_FUNCTION
CXX = x86_64-linux-gnu-g++ -std=c++17
CXXFLAGS = -fstack-reuse=none -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -pipe -O2 -O2 -g -ffile-prefix-map=/bitcoin=. -fno-extended-identifiers -fvisibility=hidden
LDFLAGS = -lpthread -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -pie -L/bitcoin/depends/x86_64-linux-gnu/lib -Wl,--as-needed -Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2 -static-libstdc++ -Wl,-O2
ARFLAGS = cr
+ sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool src/minisketch/config.status src/minisketch/libtool
sed: can't read src/minisketch/config.status: No such file or directory
sed: can't read src/minisketch/libtool: No such file or directory
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can remove this now that we've integrated the minisketch build. Have dropped it from my branch.
@@ -6,7 +6,7 @@ | |||
print-%: FORCE | |||
@echo '$*'='$($*)' | |||
|
|||
DIST_SUBDIRS = secp256k1 univalue | |||
DIST_SUBDIRS = secp256k1 minisketch univalue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pretty sure there's no need for this since we're building it ourselves. IIRC including it here was problematic while I was testing, but I've forgotten why.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea this isn't needed, and was breaking make dist
. Have dropped it from my branch.
I've pushed some fixups to https://github.com/fanquake/bitcoin/tree/minisketch_integration (sorry @sipa). Guix build: bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
6f2f140e623e86a13c3c6723528c9241d7f195ee9b440c2b70cb48b7f897efa4 guix-build-1977a5eac1ac/output/aarch64-linux-gnu/SHA256SUMS.part
f143030f742955210bbaa93f72b88aed7e64168869712ae2d06d4920b4797feb guix-build-1977a5eac1ac/output/aarch64-linux-gnu/bitcoin-1977a5eac1ac-aarch64-linux-gnu-debug.tar.gz
e7873892728dec8e46766a8094b8d32dd91d8b8e3cba7380750b9f0af046274f guix-build-1977a5eac1ac/output/aarch64-linux-gnu/bitcoin-1977a5eac1ac-aarch64-linux-gnu.tar.gz
f15a670b0bc218ec29f8d55db2c6beb08707ffdcacc5bd950ec300ef8f1d4c3d guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/SHA256SUMS.part
ecc3de5d0e229782fabc4d367a2868256f34041a67af3b3ce585e2b6e0425501 guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/bitcoin-1977a5eac1ac-arm-linux-gnueabihf-debug.tar.gz
1c53d7facb75bef4541fe4076dd8f8b1b053739fbe62fc5430c8cb2fcef19160 guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/bitcoin-1977a5eac1ac-arm-linux-gnueabihf.tar.gz
d8a9ae46890e515ec1f463745a6468dd861397d34b7ed7154f3ac67cb7b19a89 guix-build-1977a5eac1ac/output/dist-archive/bitcoin-1977a5eac1ac.tar.gz
ed1d14528d00211c1369067b0700534b0dd63a28c1a6d29d838d2efa4dc135ac guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/SHA256SUMS.part
80422fc2edb2cdf44b79962697b7e465cef87bdf984e6bb95d8a54db6c1bde2c guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/bitcoin-1977a5eac1ac-powerpc64-linux-gnu-debug.tar.gz
e0d15a9e6a13d40d3ac8cba00edb8cc445da96414c236a9adcbcb850e3bd909b guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/bitcoin-1977a5eac1ac-powerpc64-linux-gnu.tar.gz
b7d2395d35c2891eed5473a085460e02842522d4aadfba2af2bcd62fe2b58a11 guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/SHA256SUMS.part
195239762f7eb08f1c821c90b342d18dd83bc0ab66a987afe11ce531614b37ef guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/bitcoin-1977a5eac1ac-powerpc64le-linux-gnu-debug.tar.gz
5b6fe49560e675a6a660c1d01c1bd7fc07650584879a93578ddabaa4e14b1a46 guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/bitcoin-1977a5eac1ac-powerpc64le-linux-gnu.tar.gz
edef8e58acd9f9328b8ab77631c72b727d3efe246801c219f583ab1eeff00bcb guix-build-1977a5eac1ac/output/riscv64-linux-gnu/SHA256SUMS.part
1d8e37a871448ed9ec6febf825521a70e0e7b26502df7022159c6e9ffbc307b3 guix-build-1977a5eac1ac/output/riscv64-linux-gnu/bitcoin-1977a5eac1ac-riscv64-linux-gnu-debug.tar.gz
27a616dede243cc01b1b1687fb66d627c5472c8419cc493510350f9ecaea1dd7 guix-build-1977a5eac1ac/output/riscv64-linux-gnu/bitcoin-1977a5eac1ac-riscv64-linux-gnu.tar.gz
f811ea4e8ae6a0e6aac962ef6445f16c4cf316e969d97cd25a9f6d98024299ff guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/SHA256SUMS.part
3ab7a4c0c4b936033b4dc48b652478db69de3864cdb33123dd197c453baafec5 guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx-unsigned.dmg
4ed8e72a15af32c4d44e86b025735689fd8190f3b70ba126ab64f2049e112fe1 guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx-unsigned.tar.gz
b32fca59c82d9502aaacd9521fcd8ce60c3ab177056433eb55fe6fee51fac6fd guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx64.tar.gz
8e2d602f8577eb96112b3859f9b31ccdce44a202615b3d2a4f212b2406e9e2fd guix-build-1977a5eac1ac/output/x86_64-linux-gnu/SHA256SUMS.part
cebad9c8b47ad0ff2a15cc6c4899b8461dd143a51127d1f538e722d373b06bd3 guix-build-1977a5eac1ac/output/x86_64-linux-gnu/bitcoin-1977a5eac1ac-x86_64-linux-gnu-debug.tar.gz
27c050901873acb873d6ba999d9cee3e82c7ea5b17625627f0b86bd55b0ba813 guix-build-1977a5eac1ac/output/x86_64-linux-gnu/bitcoin-1977a5eac1ac-x86_64-linux-gnu.tar.gz
8627e3ed867fb065ac7c9080764f7fb0b1ad7359564a0948b4820dc39764b2a0 guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/SHA256SUMS.part
461232a659284bdea174acc0f8f4462b02133a2c0f21a81d0b2af7533b12220b guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win-unsigned.tar.gz
9c2c1b9c94016fbbd6606a0dee55a9d7d78e9725468e0b79aa413a339fb97119 guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64-debug.zip
aa344b4f6861f090477e2ba515b20bdb686735ebcfe87c32e75d1f41db8b5e42 guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64-setup-unsigned.exe
9a413697d21318a0930419b51cab86defff7a0a13eca6d59a3288dee00f427a0 guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64.zip |
Closing in favour of #23114. |
…o build/test 29173d6 ubsan: add minisketch exceptions (Cory Fields) 54b5e1a Add thin Minisketch wrapper to pick best implementation (Pieter Wuille) ee9dc71 Add basic minisketch tests (Pieter Wuille) 0659f12 Add minisketch dependency (Gleb Naumenko) 0eb7928 Add MSVC build configuration for libminisketch (Pieter Wuille) 8bc166d build: add minisketch build file and include it (Cory Fields) b2904ce build: add configure checks for minisketch (Cory Fields) b6487dc Squashed 'src/minisketch/' content from commit 89629eb2c7 (fanquake) Pull request description: This takes over #21859, which has [recently switched](bitcoin/bitcoin#21859 (comment)) to my integration branch. A few more build issues came up (and have been fixed) since, and after discussing with sipa it was decided I would open a PR to shepherd any final changes through. > This adds a `src/minisketch` subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added: > * A very basic unit test (just to make sure compilation & running works; actual correctness checking is done through minisketch's own tests). > * A wrapper in `minisketchwrapper.{cpp,h}` that runs a benchmark to determine which field implementation to use. Only changes since my last update to the branch in the previous PR have been rebasing on master and fixing an issue with a header in an introduced file. ACKs for top commit: naumenkogs: ACK 29173d6 Tree-SHA512: 1217d3228db1dd0de12c2919314e1c3626c18a416cf6291fec99d37e34fb6eec8e28d9e9fb935f8590273b8836cbadac313a15f05b4fd9f9d3024c8ce2c80d02
This adds a
src/minisketch
subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added:minisketchwrapper.{cpp,h}
that runs a benchmark to determine which field implementation to use.