Skip to content

Add minisketch subtree and integrate in build/test #21859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from

Conversation

sipa
Copy link
Member

@sipa sipa commented May 5, 2021

This adds a src/minisketch subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added:

  • A very basic unit test (just to make sure compilation & running works; actual correctness checking is done through minisketch's own tests).
  • A wrapper in minisketchwrapper.{cpp,h} that runs a benchmark to determine which field implementation to use.

@sipa sipa force-pushed the 202105_minisketch branch from c71c4c9 to 3ee2281 Compare May 5, 2021 01:37
@sipa
Copy link
Member Author

sipa commented May 5, 2021

@sipsorcery Could you help with adding MSVC build config here?

@sipa sipa force-pushed the 202105_minisketch branch from 3ee2281 to 7faf093 Compare May 5, 2021 02:20
@sipsorcery
Copy link
Contributor

@sipa will do.

@maflcko
Copy link
Member

maflcko commented May 5, 2021

From ci:


SUMMARY: MemorySanitizer: use-of-uninitialized-value /tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/minisketch/src/fields/clmul_common_impl.h:23:23 in unsigned int (anonymous namespace)::MulWithClMulReduce<unsigned int, 32, 141u>(unsigned int, unsigned int)
  ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
    This could still be a bug in your code, too!

@sipa
Copy link
Member Author

sipa commented May 5, 2021

@MarcoFalke I believe it is simply msan not being able to reason through the intrinsics (see src/minisketch/src/field/clmul_common_impl.h).

@sipsorcery
Copy link
Contributor

Attached diff adds a minisketch project to the msvc build config. I did make a few small code changes to minisketch.h and minisketch.cpp to remove a warning and change the location for a header.

minisketch.diff.gz

@jnewbery
Copy link
Contributor

jnewbery commented May 6, 2021

Do minisketch's own tests get run when running make check from bitcoin's base directory? I think it'd be good to modify make check to only run the base bitcoin tests, and then have another build target (make check_all?) that also runs the subtree unit tests. Those subtrees only get updated every few months. I expect many developers run make check many times a day and it seems wasteful to rerun the same unmodified tests on the same unmodified subtrees every time.

@maflcko
Copy link
Member

maflcko commented May 6, 2021

CI would need to run them, though. As this is the only place where sanitizers are run regularly.

@sipa sipa force-pushed the 202105_minisketch branch 8 times, most recently from e436b49 to 7142c44 Compare May 7, 2021 05:17
@sipa sipa force-pushed the 202105_minisketch branch 2 times, most recently from cb7d61c to 12fdba1 Compare May 7, 2021 06:39
@DrahtBot
Copy link
Contributor

DrahtBot commented May 7, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #22646 ([POC] Tighter Univalue integration, remove --with-system-univalue by fanquake)
  • #21515 (Erlay: bandwidth-efficient transaction relay protocol by naumenkogs)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@maflcko
Copy link
Member

maflcko commented May 7, 2021

Failure on arm: ./build-aux/test-driver: line 107: ./test-verify: cannot execute binary file: Exec format error

You might have to modify the ./ci/test/wrap-qemu.sh script.

@sipa
Copy link
Member Author

sipa commented Jul 20, 2021

@theuni If it's an issue, I think non-verify is sufficient. The verify mode mostly adds algorithmic property checks, not things that I'd expect to be very architecture/build dependent.

@theuni
Copy link
Member

theuni commented Jul 20, 2021

It's not much of an issue, it's just kinda hacky to build everything twice, especially if it's unlikely to be as useful as it is running in the upstream tests. I can add it back if there's a need.

See here for a current integration branch: https://github.com/theuni/bitcoin/commits/minisketch-split . Not sure if it's easier for you to integrate that into this PR or for me to pull your other changes in on top of mine? I'm happy to do the latter if you'd like.

writer.Flush();
}

void Deserialize(const unsigned char* ptr) override

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be helpful to add a fuzz harness for sketch deserialization. A large number of security bugs come from deserializing untrusted data.

@fanquake
Copy link
Member

I've put together a branch for this. It's based off current master and includes all the latest changes from upstream minisketch, as well as integrating the changes here with @theuni's build changes. It's available here: https://github.com/fanquake/bitcoin/tree/minisketch_integration.

Note that I've included some of the review suggestions, as well as squashed one commit (theuni@2fff4ce) from Corys branch. I've also made some additions to Makefile.minisketch.include, and pruned some unneeded changes from 8311cea. Also dropped the use of BasicTestingSetup from the added minisketch test.

I've opened an issue upstream, bitcoin-core/minisketch#46, as now that we've got -Wimplicit-fallthrough turned on, we either need to annotate fallthroughs, or suppress the warnings. i.e:

minisketch/src/minisketch.cpp:104:13: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
            case 2:
            ^
minisketch/src/minisketch.cpp:104:13: note: insert '[[fallthrough]];' to silence this warning
            case 2:
            ^
            [[fallthrough]]; 
minisketch/src/minisketch.cpp:104:13: note: insert 'break;' to avoid fall-through
            case 2:
            ^
            break; 

@fanquake
Copy link
Member

fanquake commented Sep 16, 2021

I've updated my branch (https://github.com/fanquake/bitcoin/tree/minisketch_integration) to be rebased on master, include the upstream minisktech changes, namely bitcoin-core/minisketch#45 & bitcoin-core/minisketch#47 and update our integration accordingly. Had a conflict in the MSVC config files, not 100% sure if how it's been resolved is correct.

@sipa
Copy link
Member Author

sipa commented Sep 17, 2021

Reviewed @fanquake's branch, and switched to it.

@DrahtBot
Copy link
Contributor

Guix builds

File commit e69cbac
(master)
commit d40c25a
(master and this pull)
SHA256SUMS.part 3a878af99efa151a...
*-aarch64-linux-gnu-debug.tar.gz 504514ca43584dc4...
*-aarch64-linux-gnu.tar.gz a7f1043c5be2bad7...
*-arm-linux-gnueabihf-debug.tar.gz 01025cf9adb91f2f...
*-arm-linux-gnueabihf.tar.gz fe378f5c927d3c18...
*-osx-unsigned.dmg 58d3a322c88960b2...
*-osx-unsigned.tar.gz 5f841485eaff6daf...
*-osx64.tar.gz d55e205495c5d767...
*-powerpc64-linux-gnu-debug.tar.gz 9a6c6812019b74bf...
*-powerpc64-linux-gnu.tar.gz 0ee4928babe8a06f...
*-powerpc64le-linux-gnu-debug.tar.gz 1d342c95661b5b24...
*-powerpc64le-linux-gnu.tar.gz bd8ea614f91c2853...
*-riscv64-linux-gnu-debug.tar.gz 5469fc63a961d097...
*-riscv64-linux-gnu.tar.gz 1e316726edf1fe21...
*-win-unsigned.tar.gz 1c32a6162f0f5538...
*-win64-debug.zip e7640bc4e2145270...
*-win64-setup-unsigned.exe 8e1ff1a0052908f7...
*-win64.zip 4ce4c3029f41f5af...
*-x86_64-linux-gnu-debug.tar.gz 93faa3c2bd0f0d33...
*-x86_64-linux-gnu.tar.gz 40e4f747759f74c3...
*.tar.gz ec27e3d27919ffdc... 76ccd2524fc79166...
guix_build.log 31ace9d1e85bed5c... 2b355a35d6fe8e3d...
guix_build.log.diff 440a46f08e4ab3e9...

@@ -297,7 +297,7 @@ mkdir -p "$DISTSRC"
${HOST_CXXFLAGS:+CXXFLAGS="${HOST_CXXFLAGS}"} \
${HOST_LDFLAGS:+LDFLAGS="${HOST_LDFLAGS}"}

sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool src/minisketch/config.status src/minisketch/libtool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Options used to compile and link:
  external signer = yes
  multiprocess    = no
  with libs       = yes
  with wallet     = yes
    with sqlite   = yes
    with bdb      = yes
  with gui / qt   = yes
    with qr       = yes
  with zmq        = yes
  with test       = yes
  with bench      = no
  with upnp       = yes
  with natpmp     = yes
  use asm         = yes
  ebpf tracing    = no
  sanitizers      = 
  debug enabled   = no
  gprof enabled   = no
  werror          = no

  target os       = linux
  build os        = linux-gnu

  CC              = x86_64-linux-gnu-gcc
  CFLAGS          = -pthread -pipe -O2 -O2 -g -ffile-prefix-map=/bitcoin=.
  CPPFLAGS        =   -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -I/bitcoin/depends/x86_64-linux-gnu/include/  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS -DPROVIDE_FUZZ_MAIN_FUNCTION
  CXX             = x86_64-linux-gnu-g++ -std=c++17
  CXXFLAGS        =   -fstack-reuse=none -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection     -pipe -O2 -O2 -g -ffile-prefix-map=/bitcoin=. -fno-extended-identifiers -fvisibility=hidden
  LDFLAGS         = -lpthread  -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -pie  -L/bitcoin/depends/x86_64-linux-gnu/lib -Wl,--as-needed -Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2 -static-libstdc++ -Wl,-O2
  ARFLAGS         = cr

+ sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool src/minisketch/config.status src/minisketch/libtool
sed: can't read src/minisketch/config.status: No such file or directory
sed: can't read src/minisketch/libtool: No such file or directory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this now that we've integrated the minisketch build. Have dropped it from my branch.

@@ -6,7 +6,7 @@
print-%: FORCE
@echo '$*'='$($*)'

DIST_SUBDIRS = secp256k1 univalue
DIST_SUBDIRS = secp256k1 minisketch univalue
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure there's no need for this since we're building it ourselves. IIRC including it here was problematic while I was testing, but I've forgotten why.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea this isn't needed, and was breaking make dist. Have dropped it from my branch.

@fanquake
Copy link
Member

I've pushed some fixups to https://github.com/fanquake/bitcoin/tree/minisketch_integration (sorry @sipa). make dist(dir) and the Guix build have both been fixed. I also added a missing exclude for minisketch to the copyright header update script.

Guix build:

bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
6f2f140e623e86a13c3c6723528c9241d7f195ee9b440c2b70cb48b7f897efa4  guix-build-1977a5eac1ac/output/aarch64-linux-gnu/SHA256SUMS.part
f143030f742955210bbaa93f72b88aed7e64168869712ae2d06d4920b4797feb  guix-build-1977a5eac1ac/output/aarch64-linux-gnu/bitcoin-1977a5eac1ac-aarch64-linux-gnu-debug.tar.gz
e7873892728dec8e46766a8094b8d32dd91d8b8e3cba7380750b9f0af046274f  guix-build-1977a5eac1ac/output/aarch64-linux-gnu/bitcoin-1977a5eac1ac-aarch64-linux-gnu.tar.gz
f15a670b0bc218ec29f8d55db2c6beb08707ffdcacc5bd950ec300ef8f1d4c3d  guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/SHA256SUMS.part
ecc3de5d0e229782fabc4d367a2868256f34041a67af3b3ce585e2b6e0425501  guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/bitcoin-1977a5eac1ac-arm-linux-gnueabihf-debug.tar.gz
1c53d7facb75bef4541fe4076dd8f8b1b053739fbe62fc5430c8cb2fcef19160  guix-build-1977a5eac1ac/output/arm-linux-gnueabihf/bitcoin-1977a5eac1ac-arm-linux-gnueabihf.tar.gz
d8a9ae46890e515ec1f463745a6468dd861397d34b7ed7154f3ac67cb7b19a89  guix-build-1977a5eac1ac/output/dist-archive/bitcoin-1977a5eac1ac.tar.gz
ed1d14528d00211c1369067b0700534b0dd63a28c1a6d29d838d2efa4dc135ac  guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/SHA256SUMS.part
80422fc2edb2cdf44b79962697b7e465cef87bdf984e6bb95d8a54db6c1bde2c  guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/bitcoin-1977a5eac1ac-powerpc64-linux-gnu-debug.tar.gz
e0d15a9e6a13d40d3ac8cba00edb8cc445da96414c236a9adcbcb850e3bd909b  guix-build-1977a5eac1ac/output/powerpc64-linux-gnu/bitcoin-1977a5eac1ac-powerpc64-linux-gnu.tar.gz
b7d2395d35c2891eed5473a085460e02842522d4aadfba2af2bcd62fe2b58a11  guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/SHA256SUMS.part
195239762f7eb08f1c821c90b342d18dd83bc0ab66a987afe11ce531614b37ef  guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/bitcoin-1977a5eac1ac-powerpc64le-linux-gnu-debug.tar.gz
5b6fe49560e675a6a660c1d01c1bd7fc07650584879a93578ddabaa4e14b1a46  guix-build-1977a5eac1ac/output/powerpc64le-linux-gnu/bitcoin-1977a5eac1ac-powerpc64le-linux-gnu.tar.gz
edef8e58acd9f9328b8ab77631c72b727d3efe246801c219f583ab1eeff00bcb  guix-build-1977a5eac1ac/output/riscv64-linux-gnu/SHA256SUMS.part
1d8e37a871448ed9ec6febf825521a70e0e7b26502df7022159c6e9ffbc307b3  guix-build-1977a5eac1ac/output/riscv64-linux-gnu/bitcoin-1977a5eac1ac-riscv64-linux-gnu-debug.tar.gz
27a616dede243cc01b1b1687fb66d627c5472c8419cc493510350f9ecaea1dd7  guix-build-1977a5eac1ac/output/riscv64-linux-gnu/bitcoin-1977a5eac1ac-riscv64-linux-gnu.tar.gz
f811ea4e8ae6a0e6aac962ef6445f16c4cf316e969d97cd25a9f6d98024299ff  guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/SHA256SUMS.part
3ab7a4c0c4b936033b4dc48b652478db69de3864cdb33123dd197c453baafec5  guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx-unsigned.dmg
4ed8e72a15af32c4d44e86b025735689fd8190f3b70ba126ab64f2049e112fe1  guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx-unsigned.tar.gz
b32fca59c82d9502aaacd9521fcd8ce60c3ab177056433eb55fe6fee51fac6fd  guix-build-1977a5eac1ac/output/x86_64-apple-darwin18/bitcoin-1977a5eac1ac-osx64.tar.gz
8e2d602f8577eb96112b3859f9b31ccdce44a202615b3d2a4f212b2406e9e2fd  guix-build-1977a5eac1ac/output/x86_64-linux-gnu/SHA256SUMS.part
cebad9c8b47ad0ff2a15cc6c4899b8461dd143a51127d1f538e722d373b06bd3  guix-build-1977a5eac1ac/output/x86_64-linux-gnu/bitcoin-1977a5eac1ac-x86_64-linux-gnu-debug.tar.gz
27c050901873acb873d6ba999d9cee3e82c7ea5b17625627f0b86bd55b0ba813  guix-build-1977a5eac1ac/output/x86_64-linux-gnu/bitcoin-1977a5eac1ac-x86_64-linux-gnu.tar.gz
8627e3ed867fb065ac7c9080764f7fb0b1ad7359564a0948b4820dc39764b2a0  guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/SHA256SUMS.part
461232a659284bdea174acc0f8f4462b02133a2c0f21a81d0b2af7533b12220b  guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win-unsigned.tar.gz
9c2c1b9c94016fbbd6606a0dee55a9d7d78e9725468e0b79aa413a339fb97119  guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64-debug.zip
aa344b4f6861f090477e2ba515b20bdb686735ebcfe87c32e75d1f41db8b5e42  guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64-setup-unsigned.exe
9a413697d21318a0930419b51cab86defff7a0a13eca6d59a3288dee00f427a0  guix-build-1977a5eac1ac/output/x86_64-w64-mingw32/bitcoin-1977a5eac1ac-win64.zip

@fanquake
Copy link
Member

Closing in favour of #23114.

@fanquake fanquake closed this Sep 28, 2021
fanquake added a commit to bitcoin-core/gui that referenced this pull request Nov 12, 2021
…o build/test

29173d6 ubsan: add minisketch exceptions (Cory Fields)
54b5e1a Add thin Minisketch wrapper to pick best implementation (Pieter Wuille)
ee9dc71 Add basic minisketch tests (Pieter Wuille)
0659f12 Add minisketch dependency (Gleb Naumenko)
0eb7928 Add MSVC build configuration for libminisketch (Pieter Wuille)
8bc166d build: add minisketch build file and include it (Cory Fields)
b2904ce build: add configure checks for minisketch (Cory Fields)
b6487dc Squashed 'src/minisketch/' content from commit 89629eb2c7 (fanquake)

Pull request description:

  This takes over #21859, which has [recently switched](bitcoin/bitcoin#21859 (comment)) to my integration branch. A few more build issues came up (and have been fixed) since, and after discussing with sipa it was decided I would open a PR to shepherd any final changes through.

  > This adds a `src/minisketch` subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added:
  > * A very basic unit test (just to make sure compilation & running works; actual correctness checking is done through minisketch's own tests).
  > * A wrapper in `minisketchwrapper.{cpp,h}` that runs a benchmark to determine which field implementation to use.

  Only changes since my last update to the branch in the previous PR have been rebasing on master and fixing an issue with a header in an introduced file.

ACKs for top commit:
  naumenkogs:
    ACK 29173d6

Tree-SHA512: 1217d3228db1dd0de12c2919314e1c3626c18a416cf6291fec99d37e34fb6eec8e28d9e9fb935f8590273b8836cbadac313a15f05b4fd9f9d3024c8ce2c80d02
@bitcoin bitcoin locked and limited conversation to collaborators Oct 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.